Civilization / Humanities & Culture

Distributional Semantics (Context)

In the 1950s linguists such as Harris and Firth proposed that words used in similar contexts have similar meanings; turned into word vectors, this idea is the root of word2vec and language models.

Martin Joos (1950) defined a word's meaning through the probabilities of the words around it, Zellig Harris (1954) argued that similar words occur in similar surroundings, and J. R. Firth (1957) summed it up as knowing a word by the company it keeps. The psychologist Charles Osgood (1957) placed word meanings as points in a space, and information-retrieval researchers (Salton, 1971) described words and documents as vectors. Combined, these ideas became the word vectors of word2vec (2013) and the embeddings at the heart of large language models.