GloVe Algorithm

A GloVe Algorithm is a continuous dense distributional word model training algorithm proposed in (Pennington et al., 2014).

• Context:
  • It can be implemented by a GloVe-based System.
  • It precomputes a Word-Word Co-Occurrence Matrix.
  • It trains on Global Word-Word Co-occurrence Counts.
  • It identifies Global Vectors.
  • It uses Weighted Least Squares.
• Example(s):
  • glove-python - python implementation of GloVe.
  • Glove v.2.0,
  • GloVe v.1.2,
  • GloVe v.1.0,
  • …
• Counter-Example(s):
  • word2vec Algorithm with Skip-Gram with Negative Sampling.
  • Continuous-BoW.
• See: SGD Algorithm, Neural Sequence Learning Task, Natural Language Model, Sentiment Analysis, Word Sense Disambiguation, ULMFiT, Sequence-to-Sequence Learning.

---

---

References

2019

• (Pennington et al., 2019) ⇒ Jeffrey Pennington, Richard Socher, and Christopher D. Manning. "GloVe: Global Vectors for Word Representation". Retrieved: 2019-02-24
  • QUOTE: GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space.

2015

• (Rothe & Schütze, 2015) ⇒ Sascha Rothe, and Hinrich Schütze. (2015). “AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes.” In: arXiv preprint arXiv:1507.01127.
  • QUOTE: … Unsupervised methods for word embeddings (also called “distributed word representations”) have become popular in natural language processing (NLP). These methods only need very large corpora as input to create sparse representations (e.g., based on local collocations) and project them into a lower dimensional dense vector space. Examples for word embeddings are SENNA (Collobert and Weston, 2008), the hierarchical log-bilinear model (Mnih and Hinton, 2009), word2vec (Mikolov et al., 2013c) and GloVe (Pennington et al., 2014).

2014a

• (Rehurek, 2014) ⇒ Radim Rehurek. (2014) "Making sense of word2vec" Published Online: 2014-12-23
  • Their method GloVe (Global Vectors) identified a matrix which, when factorized using the particular SGD algorithm of word2vec, yields out exactly these two matrices. So where word2vec was a bit hazy about what’s going on underneath, GloVe explicitly names the “objective” matrix, identifies the factorization, and provides some intuitive justification as to why this should give us working similarities. …

    … Basically, where GloVe precomputes the large word x word co-occurrence matrix in memory and then quickly factorizes it, word2vec sweeps through the sentences in an online fashion, handling each co-occurrence separately. So, there is a tradeoff between taking more memory (GloVe) vs. taking longer to train (word2vec). Also, once computed, GloVe can re-use the co-occurrence matrix to quickly factorize with any dimensionality, whereas word2vec has to be trained from scratch after changing its embedding dimensionality.

2014b

• (Pennington et al., 2014) ⇒ Jeffrey Pennington, Richard Socher, and Christopher D. Manning. (2014). “GloVe: Global Vectors for Word Representation.” In: Proceedings of EMNLP 2014.
  • QUOTE: Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic, but the origin of these regularities has remained opaque. We analyze and make explicit the model properties needed for such regularities to emerge in word vectors. The result is a new global log-bilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods. Our model efficiently leverages statistical information by training only on the nonzero elements in a word-word co-occurrence matrix, rather than on the entire sparse matrix or on individual context windows in a large corpus. The model produces a vector space with meaningful substructure, as evidenced by its performance of 75% on a recent word analogy task. It also outperforms related models on similarity tasks and named entity recognition.

---