Researched
Word Embeddings (word2vec)
Mikolov's word2vec (2013) turns words into vectors in which meaning shows up as direction, like king minus man plus woman.
Open in the interactive tree →Learning vectors from huge amounts of text made words usable as input for neural networks. Embeddings of this kind are the first layer of modern language models.
Prerequisites
- Deep Learning2012
Unlocks
- Attention Mechanism2014
- Transformer Architecture2017Word embeddings are the input layer of transformers