fastText System
A fastText System is a Word Embedding System that be used to learn vectorized word representation and solve text classification tasks.
- Context:
- It was first introduced by Bojanowski et al. (2017) and Joulin et al. (2017).
- It can learn and detect Subword Units and OOV Words.
- Example(s):
- fastText Library,
- ...
- …
- Counter-Example(s):
- See: NLP System, Subword Embedding System, Language Model, Bag-Of-Words.
References
2021a
- (Wikipedia, 2021) ⇒ https://en.wikipedia.org/wiki/fastText Retrieved:2021-5-8.
- fastText is a library for learning of word embeddings and text classification created by Facebook's AI Research (FAIR) lab. The model allows one to create an unsupervised learning or supervised learning algorithm for obtaining vector representations for words. Facebook makes available pretrained models for 294 languages. fastText uses a neural network for word embedding. The fasttext algorithm is based on these two papers:
- Enriching Word Vectors with Subword Information , Piotr Bojanowski, Edouard Grave, Armand Joulin and Tomas Mikolov, 2016
- Bag of Tricks for Efficient Text Classification, Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov, 2016
- fastText is a library for learning of word embeddings and text classification created by Facebook's AI Research (FAIR) lab. The model allows one to create an unsupervised learning or supervised learning algorithm for obtaining vector representations for words. Facebook makes available pretrained models for 294 languages. fastText uses a neural network for word embedding. The fasttext algorithm is based on these two papers:
2021b
- (FB Research, 2021) ⇒ https://research.fb.com/fasttext/ Retrieved: 2021-05-07.
- QUOTE: FastText combines some of the most successful concepts introduced by the natural language processing and machine learning communities in the last few decades. These include representing sentences with bag of words and bag of n-grams, as well as using subword information, and sharing information across classes through a hidden representation. We also employ a hierarchical softmax that takes advantage of the unbalanced distribution of the classes to speed up computation. These different concepts are being used for two different tasks: efficient text classification and learning word vector representations.
(...)
FastText also represents a text by a low dimensional vector, which is obtained by summing vectors corresponding to the words appearing in the text. In fastText, a low dimensional vector is associated to each word of the vocabulary. This hidden representation is shared across all classifiers for different categories, allowing information about words learned for one category to be used by other categories. These kind of representations, called bag of words, ignore word order. In fastText we also use vectors to represent word ngrams to take into account local word order, which is important for many text classification problems.
Our experiments show that fastText is often on par with deep learning classifiers in terms of accuracy, and many orders of magnitude faster for training and evaluation. With fastText, we were often able to cut training times from several days to just a few seconds, and achieve state-of-the-art performance on many standard problems, such as sentiment analysis or tag prediction.
- QUOTE: FastText combines some of the most successful concepts introduced by the natural language processing and machine learning communities in the last few decades. These include representing sentences with bag of words and bag of n-grams, as well as using subword information, and sharing information across classes through a hidden representation. We also employ a hierarchical softmax that takes advantage of the unbalanced distribution of the classes to speed up computation. These different concepts are being used for two different tasks: efficient text classification and learning word vector representations.
2017a
- (Bojanowski et al., 2017) ⇒ Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. (2017). “Enriching Word Vectors with Subword Information.” In: Transactions of the Association for Computational Linguistics, 5.
2017b
- (Joulin et al., 2017) ⇒ Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. (2017). “Bag of Tricks for Efficient Text Classification.” In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL2017) Volume 2: Short Papers.