Bidirectional Encoder Representations from Transformers (BERT) Language Model Training System

From GM-RKB
Jump to navigation Jump to search

A Bidirectional Encoder Representations from Transformers (BERT) Language Model Training System is an Transformer-based LM training system that implements a BERT algorithm to solve a BERT task (to produce BERT LMs based on a BERT meta-model).



References

2020

2019a

2019b

2019b

2018a

  • https://ai.googleblog.com/search/label/Natural%20Language%20Understanding
    • QUOTE: This week, we open sourced a new technique for NLP pre-training called Bidirectional Encoder Representations from Transformers, or BERT. With this release, anyone in the world can train their own state-of-the-art question answering system (for a variety of other models) in about 30 minutes on a single Cloud TPU, or in a few hours using a single GPU.

2018b

  • https://github.com/google-research/bert/blob/master/README.md
    • QUOTE: ... BERT is a method of pre-training language representations, meaning that we train a general-purpose "language understanding" model on a large text corpus (like Wikipedia), and then use that model for downstream NLP tasks that we care about (like question answering). BERT outperforms previous methods because it is the first unsupervised, deeply bidirectional system for pre-training NLP.

      Unsupervised means that BERT was trained using only a plain text corpus, which is important because an enormous amount of plain text data is publicly available on the web in many languages. …