Autoregressive Language Model

An Autoregressive Language Model is a language model that uses sequential prediction to generate text tokens (for performing natural language tasks through context-based generation).

• AKA: Sequential Language Model, One-Step-At-A-Time Model.
• Context:
  • It can typically process Input Text through sequential analysis.
  • It can typically generate Next Tokens through probability distributions.
  • It can typically maintain Context Window through previous token tracking.
  • It can typically perform Token Selection through learned patterns.
  • It can typically support Text Generation through iterative predictions.
  • ...
  • It can often optimize Generation Quality through context understanding.
  • It can often enhance Model Performance through training data scale.
  • It can often improve Prediction Accuracy through pattern recognition.
  • It can often handle Task Adaptation through fine-tuning processes.
  • ...
  • It can range from being a Small-Scale Model to being a Large-Scale Model, depending on its training data volume.
  • It can range from being a Basic Predictor to being an Advanced Generator, depending on its architectural complexity.
  • It can range from being a Task-Specific Model to being a General-Purpose Model, depending on its application scope.
  • ...
  • It can have Training Datasets of text content for pattern learning.
  • It can perform Content Generation for specific tasks.
  • It can support Multiple Applications through versatile architecture.
  • ...
• Examples:
  • Model Architecture Types, such as:
    • Commercial Implementations, such as:
      • GPT-4 Model for advanced text generation.
      • Gemini Model for multimodal processing.
    • Open Source Implementations, such as:
      • BLOOM Model for multilingual generation.
      • LLaMA Model for research purposes.
  • Specialized Functions, such as:
    • Language Processing Types, such as:
      • Translation Model for language conversion.
      • Summarization Model for text condensation.
  • ...
• Counter-Examples:
  • Bidirectional Models, which process text input in both directions simultaneously.
  • Masked Language Models, which predict random tokens rather than sequential tokens.
  • Statistical Models, which use linear regression instead of neural networks.
• See: Language Model, Neural Network, Text Generation, Machine Learning, Natural Language Processing.

---

---

References

2019

• (Yang, Dai et al., 2019) ⇒ Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. (2019). “Xlnet: Generalized Autoregressive Pretraining for Language Understanding.” Advances in Neural Information Processing Systems, 32.
  • ABSTRACT: With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment setting, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.

---