Static embeddings are pre-trained, fixed representations of words or tokens in a continuous vector space. Each word is assigned a specific vector, and this vector does not change during the training of downstream tasks. These embeddings capture the semantic meaning of words based on their usage in large corpora.
Key Characteristics of Static Embeddings:
- Fixed Representations: Once trained, the embeddings remain constant. The vector for a word like “king” will always be the same.
- Context-Independent: The same word will have the same vector regardless of the context in which it appears. For example, “bank” will have one vector, even though it could mean a financial institution or the side of a river.
- Efficient and Lightweight: Since embeddings are pre-trained and fixed, they require less computational power during downstream model training.
- Semantic Relationships: Static embeddings capture relationships between words, such as analogies (e.g., king — man + woman ≈ queen).
Examples of Static Embeddings:
- Word2Vec: Trained to predict words from surrounding context (CBOW) or to predict surrounding words from a center word (Skip-gram).
- GloVe (Global Vectors): Embeddings trained based on word co-occurrence matrices.
- FastText: Enhances Word2Vec by incorporating subword information, useful for out-of-vocabulary words.
Limitations:
- Lack of Contextual Understanding: They don’t adapt to different contexts. For example, “bat” will have the same embedding whether referring to the animal or sports equipment.
- Out-of-Vocabulary Words: Words not seen during training are not assigned embeddings (except for FastText, which can generate embeddings for new words using subword information).
Comparison with Dynamic Embeddings:
- Dynamic Embeddings (e.g., BERT, GPT): These embeddings adjust based on the context in which a word appears. For example, “bat” will have different embeddings depending on the sentence.
- Static Embeddings: Faster and simpler but less accurate in context-sensitive tasks.
Would you like to dive deeper into how static embeddings are generated or how they compare with contextual embeddings in performance?
