Autoregressive Language Model

An Autoregressive Language Model is a language model that uses sequential prediction to generate text tokens (for performing natural language tasks through context-based generation).

• AKA: Sequential Language Model, One-Step-At-A-Time Model, Left-to-Right Language Model, Causal Language Model, Decoder-Only Language Model, Unidirectional Language Model.
• Context:
  • Autoregressive Language Model Input: autoregressive language model prompt tokens formatted as autoregressive language model token sequences.
  • Autoregressive Language Model Output: autoregressive language model generated tokens forming autoregressive language model text sequences.
  • Autoregressive Language Model Performance Measure: autoregressive language model quality metrics such as autoregressive language model perplexity, autoregressive language model token accuracy, and autoregressive language model human evaluation score.
  • ...
  • It can typically process Autoregressive Language Model Input Text through autoregressive sequential analysis.
  • It can typically generate Autoregressive Language Model Next Tokens through autoregressive probability distributions over autoregressive language model vocabulary.
  • It can typically maintain Autoregressive Language Model Context Window through autoregressive previous token tracking.
  • It can typically perform Autoregressive Language Model Token Selection through autoregressive learned patterns.
  • It can typically support Autoregressive Language Model Text Generation through autoregressive iterative predictions.
  • It can typically analyze autoregressive language model input sequences to produce autoregressive language model probability estimations for possible autoregressive language model next tokens.
  • It can typically encode autoregressive language model contextual meaning through autoregressive language model hidden state representations.
  • It can typically apply autoregressive language model attention mechanisms to focus on autoregressive language model relevant context.
  • It can typically leverage autoregressive language model parameter knowledge acquired during autoregressive language model training phase.
  • It can typically build autoregressive language model responses one autoregressive language model token at a time.
  • It can typically model the joint probability of a autoregressive language model sequence as a product of autoregressive language model conditional probabilities for each autoregressive language model token given all previous autoregressive language model tokens.
  • It can typically enforce autoregressive language model causal constraints to prevent autoregressive language model future information leakage during autoregressive language model training and autoregressive language model inference.
  • ...
  • It can often optimize Autoregressive Language Model Generation Quality through autoregressive context understanding.
  • It can often enhance Autoregressive Language Model Performance through autoregressive training data scale.
  • It can often improve Autoregressive Language Model Prediction Accuracy through autoregressive pattern recognition.
  • It can often handle Autoregressive Language Model Task Adaptation through autoregressive fine-tuning processes.
  • It can often implement autoregressive language model sampling techniques like autoregressive language model temperature adjustment, autoregressive language model nucleus sampling, and autoregressive language model top-k filtering.
  • It can often utilize autoregressive language model parallel computation during autoregressive language model training phase but requires autoregressive language model sequential generation during autoregressive language model inference.
  • It can often incorporate autoregressive language model layer normalization, autoregressive language model residual connections, and autoregressive language model positional encodings.
  • It can often mitigate autoregressive language model repetition issues through autoregressive language model penalty mechanisms.
  • It can often demonstrate autoregressive language model in-context learning capabilities through autoregressive language model few-shot examples.
  • It can often handle autoregressive language model information retention across autoregressive language model long context windows.
  • It can often experience autoregressive language model exposure bias due to autoregressive language model train-test discrepancy between using autoregressive language model ground truth context (training) and autoregressive language model generated context (inference).
  • It can often accumulate autoregressive language model generation errors over autoregressive language model long sequences through autoregressive language model error compounding.
  • It can often apply autoregressive language model teacher forcing during autoregressive language model training to stabilize autoregressive language model learning process.
  • It can often benefit from autoregressive language model prompt engineering to guide autoregressive language model output behavior.
  • ...
  • It can range from being a Small-Scale Autoregressive Language Model to being a Large-Scale Autoregressive Language Model, depending on its autoregressive language model parameter count.
  • It can range from being a Basic Autoregressive Language Model Predictor to being an Advanced Autoregressive Language Model Generator, depending on its autoregressive language model architectural complexity.
  • It can range from being a Task-Specific Autoregressive Language Model to being a General-Purpose Autoregressive Language Model, depending on its autoregressive language model application scope.
  • It can range from being a Character-Level Autoregressive Language Model to being a Token-Level Autoregressive Language Model to being a Word-Level Autoregressive Language Model, depending on its autoregressive language model prediction granularity.
  • It can range from being a Shallow Autoregressive Language Model to being a Deep Autoregressive Language Model, depending on its autoregressive language model layer count.
  • It can range from being a Narrow-Context Autoregressive Language Model to being a Wide-Context Autoregressive Language Model, depending on its autoregressive language model context window size.
  • It can range from being a Traditional Autoregressive Language Model to being a Transformer-Based Autoregressive Language Model, depending on its autoregressive language model architectural paradigm.
  • It can range from being a Recurrent Autoregressive Language Model to being a Feed-Forward Autoregressive Language Model, depending on its autoregressive language model processing mechanism.
  • It can range from being a Domain-Specific Autoregressive Language Model to being a Multi-Domain Autoregressive Language Model, depending on its autoregressive language model training data diversity.
  • It can range from being a Base Autoregressive Language Model to being an Instruction-Tuned Autoregressive Language Model, depending on its autoregressive language model training objective.
  • It can range from being a Deterministic Autoregressive Language Model to being a Stochastic Autoregressive Language Model, depending on its autoregressive language model decoding strategy.
  • ...
  • It can have Autoregressive Language Model Training Datasets of autoregressive language model text content for autoregressive language model pattern learning.
  • It can perform Autoregressive Language Model Content Generation for autoregressive language model specific tasks.
  • It can support Autoregressive Language Model Multiple Applications through autoregressive language model versatile architecture.
  • It can utilize autoregressive language model causal attention masks to ensure autoregressive language model unidirectional information flow.
  • It can implement autoregressive language model next-token prediction objective during autoregressive language model training.
  • It can provide autoregressive language model instruction following when autoregressive language model fine-tuned on autoregressive language model instruction datasets.
  • It can serve as a autoregressive language model foundation model for autoregressive language model downstream tasks.
  • It can enable autoregressive language model conditional generation through autoregressive language model prompt engineering.
  • It can benefit from autoregressive language model reinforcement learning from human feedback for autoregressive language model alignment.
  • It can exhibit autoregressive language model emergent capabilities at sufficient autoregressive language model scale.
  • It can produce autoregressive language model chain-of-thought reasoning when properly autoregressive language model prompted.
  • It can apply autoregressive language model scaling laws that relate autoregressive language model performance to autoregressive language model parameter count, autoregressive language model training data size, and autoregressive language model compute resources.
  • It can follow autoregressive language model probability factorization using the chain rule: P(w₁,...,wₙ) = ∏ᵢ P(wᵢ|w₁,...,wᵢ₋₁).
  • It can manage autoregressive language model decoding trade-offs between autoregressive language model diversity and autoregressive language model coherence.
  • It can deploy autoregressive language model self-correction strategies to recover from autoregressive language model generation errors in autoregressive language model long-form content.
  • ...
• Examples:
  • Architecture Types, such as:
    • Transformer-based Models, such as:
      • GPT-4, an OpenAI LLM Model.
      • Gemini Ultra, a Google LLM.
      • Claude 3 Opus, an Anthropic LLM.
      • LLaMA 3, a Meta LLM.
      • Mistral 7B, an Open-Source LLM.
    • Recurrent Neural Network Models, such as:
      • GPT-1, an RNN-transformer hybrid.
      • ELMo, a bidirectional LSTM implementation.
      • AWD-LSTM, a regularized recurrent network.
    • Convolutional Models, such as:
      • Gated CNN, for hierarchical convolution.
      • ByteNet, for dilated convolution.
  • Processing Mechanisms, such as:
    • Recurrent Models, such as:
      • LSTM-based Models for sequential state tracking.
      • GRU-based Models for efficient recurrence.
    • Feed-Forward Models, such as:
      • Transformer Decoder Models for parallel context processing.
      • Linear Attention Models for efficient attention computation.
  • Prediction Granularitys, such as:
    • Character-Level Models, such as:
      • Char-RNN for character-by-character generation.
    • Subword-Level Models, such as:
      • BPE-based Model for subword tokenization.
      • WordPiece Model for efficient vocabulary coverage.
    • Word-Level Models, such as:
      • N-gram Model for statistical word prediction.
  • Size Categorys, such as:
    • Small Models (<1B parameters), such as:
      • GPT-2 Small (124M) for lightweight applications.
      • Phi-1.5 (1.3B) for efficient deployment.
    • Medium Models (1B-10B parameters), such as:
      • GPT-2 XL (1.5B) for balanced performance.
      • Mistral 7B for efficient scaling.
    • Large-Scale Language Model (LLM)s (>10B parameters), such as:
      • OpenAI GPT-3 Large Language Model (LLM) (175B) for emergent capabilities.
      • LLaMA 2 70B for open research.
      • Claude 3 Opus for advanced understanding.
  • Application Domains, such as:
    • General-Purpose Models, such as:
      • PaLM 2 for multi-task performance.
      • GPT-4o for conversational interface.
    • Specialized Models, such as:
      • Code Models, such as:
        • CodeLlama for programming assistance.
        • Salesforce CodeGen Model for code generation.
        • StarCoder for multi-language development.
      • Scientific Models, such as:
        • Galactica for scientific knowledge.
        • Med-PaLM for medical domain.
  • Training Paradigms, such as:
    • Base Pretrained Models, such as:
      • Chinchilla for optimal scaling law application.
      • BLOOM for multilingual capability.
    • Instruction-Tuned Models, such as:
      • ChatGPT for conversational ability.
      • Flan-T5 for instruction following.
    • RLHF-Enhanced Models, such as:
      • Claude 3 for human alignment.
      • GPT-4 for preference optimization.
  • Decoding Strategys, such as:
    • Deterministic Models, such as:
      • Greedy Decoding for highest-probability selection.
      • Beam Search for multiple-path exploration.
    • Stochastic Models, such as:
      • Temperature Sampling for controlled randomness.
      • Nucleus Sampling for dynamic probability threshold.
  • Organizational Origins, such as:
    • Commercial Developer Models, such as:
      • OpenAI Models, such as GPT-4, GPT-3.5, and ChatGPT.
      • Anthropic Models, such as Claude 3 Opus, Claude 3 Sonnet, and Claude 3 Haiku.
      • Google Models, such as Gemini Ultra, Gemini Pro, and PaLM 2.
    • Research Institute Models, such as:
      • EleutherAI Institute models, such as GPT-Neo, GPT-J, and Pythia.
      • Hugging Face models, such as BLOOM and OPT.
  • Unidirectional One-Token-at-a-Time Large Language Model (LLM)s, such as:
    • GPT-3 for few-shot natural language processing (NLP) tasks.
    • Llama 2 for open-source research.
    • PaLM for massive scale generalization.
  • ...
  • Autoregressive Language Model Processing Mechanisms, such as:
    • Recurrent Autoregressive Language Models, such as:
      • LSTM-based Autoregressive Language Models for autoregressive language model sequential state tracking.
      • GRU-based Autoregressive Language Models for autoregressive language model efficient recurrence.
      • Vanilla RNN Autoregressive Language Models for autoregressive language model simple recurrent processing.
    • Feed-Forward Autoregressive Language Models, such as:
      • Transformer Decoder Autoregressive Language Models for autoregressive language model parallel context processing.
      • Convolutional Autoregressive Language Models for autoregressive language model local pattern recognition.
      • Linear Attention Autoregressive Language Models for autoregressive language model efficient attention computation.
  • Autoregressive Language Model Prediction Granularitys, such as:
    • Character-Level Autoregressive Language Models, such as:
      • Char-RNN Autoregressive Language Model for autoregressive language model character-by-character generation.
      • Character Transformer Autoregressive Language Model for autoregressive language model fine-grained text modeling.
    • Subword-Level Autoregressive Language Models, such as:
      • BPE-based Autoregressive Language Model for autoregressive language model subword tokenization.
      • WordPiece Autoregressive Language Model for autoregressive language model efficient vocabulary coverage.
      • SentencePiece Autoregressive Language Model for autoregressive language model language-agnostic tokenization.
    • Word-Level Autoregressive Language Models, such as:
      • Traditional N-gram Autoregressive Language Model for autoregressive language model statistical word prediction.
      • Neural Word-Level Autoregressive Language Model for autoregressive language model whole-word embedding.
  • Autoregressive Language Model Size Categorys, such as:
    • Small Autoregressive Language Models (<1B parameters), such as:
      • GPT-2 Language Model (124M) for autoregressive language model lightweight applications.
      • Phi-1.5 Language Model (1.3B) for autoregressive language model efficient deployment.
      • OPT-125M Language Model for autoregressive language model research purposes.
    • Medium Autoregressive Language Models (1B-10B parameters), such as:
      • GPT-2 XL Language Model (1.5B) for autoregressive language model balanced performance.
      • Mistral 7B x Model for autoregressive language model efficient scaling.
      • Phi-2 Language Model (2.7B) for autoregressive language model surprisingly strong performance.
    • Large Autoregressive Language Models (>10B parameters), such as:
      • GPT-3 Language Model (175B) for autoregressive language model emergent capabilities.
      • LLaMA 2 70B Language Model for autoregressive language model open research.
      • Claude 3 Opus Language Model for autoregressive language model advanced understanding.
  • Autoregressive Language Model Application Domains, such as:
    • General-Purpose Autoregressive Language Models, such as:
      • PaLM 2 Language Model for autoregressive language model multi-task performance.
      • GPT-4o Language Model for autoregressive language model conversational interface.
      • Claude 3 Sonnet Language Model for autoregressive language model balanced capabilities.
    • Specialized Autoregressive Language Models, such as:
      • Code Autoregressive Language Models, such as:
        • CodeLlama Language Model for autoregressive language model programming assistance.
        • StarCoder Language Model for autoregressive language model multi-language development.
        • DeepSeek Coder Language Model for autoregressive language model code completion.
      • Scientific Autoregressive Language Models, such as:
        • Galactica Autoregressive Language Model for autoregressive language model scientific knowledge.
        • Med-PaLM Autoregressive Language Model for autoregressive language model medical domain.
        • BioGPT Autoregressive Language Model for autoregressive language model biomedical text.
  • Autoregressive Language Model Training Paradigms, such as:
    • Base Pretrained Autoregressive Language Models, such as:
      • Chinchilla Autoregressive Language Model for autoregressive language model optimal scaling law application.
      • OPT Autoregressive Language Model for autoregressive language model research foundation.
      • BLOOM Autoregressive Language Model for autoregressive language model multilingual capability.
    • Instruction-Tuned Autoregressive Language Models, such as:
      • ChatGPT Autoregressive Language Model for autoregressive language model conversational ability.
      • Flan-T5 Autoregressive Language Model for autoregressive language model instruction following.
      • Alpaca Autoregressive Language Model for autoregressive language model academic instruction tuning.
    • RLHF-Enhanced Autoregressive Language Models, such as:
      • Claude 3 Autoregressive Language Model for autoregressive language model human alignment.
      • GPT-4 Autoregressive Language Model for autoregressive language model preference optimization.
      • Gemini Pro Autoregressive Language Model for autoregressive language model aligned generation.
    • Teacher-Forced Autoregressive Language Models, such as:
      • Standard RNN Autoregressive Language Model for autoregressive language model stable training.
      • Early Transformer Autoregressive Language Model for autoregressive language model sequential training.
  • Autoregressive Language Model Decoding Strategys, such as:
    • Deterministic Autoregressive Language Models, such as:
      • Greedy Decoding Autoregressive Language Model for autoregressive language model highest-probability selection.
      • Beam Search Autoregressive Language Model for autoregressive language model multiple-path exploration.
    • Stochastic Autoregressive Language Models, such as:
      • Temperature Sampling Autoregressive Language Model for autoregressive language model controlled randomness.
      • Top-k Sampling Autoregressive Language Model for autoregressive language model restricted vocabulary selection.
      • Nucleus Sampling Autoregressive Language Model for autoregressive language model dynamic probability threshold.
  • ...
• Counter-Examples:
  • Bidirectional Language Models, which process text input in both directions simultaneously without autoregressive constraints, allowing each token representation to incorporate context from both preceding and following tokens as in BERT architecture.
  • Masked Language Models, which predict random masked tokens rather than sequential tokens, enabling parallel training and bidirectional context modeling but requiring a different inference approach than autoregressive generation as demonstrated by BERT and RoBERTa.
  • Non-Autoregressive Language Models, which generate multiple output tokens in parallel rather than sequentially, sacrificing some generation quality for increased inference speed as seen in NART architectures.
  • Diffusion Language Models, which generate text through iterative denoising processes rather than left-to-right token prediction, allowing for non-sequential refinement of the entire output as in Diffusion-LM approaches.
  • Statistical Language Models, which use n-gram probability and linear regression instead of neural network architectures, lacking the representational capacity and long-range dependency modeling of autoregressive neural language models.
  • Energy-Based Language Models, which learn unnormalized probability distributions rather than explicit conditional probabilities for next token prediction, focusing on scoring configurations instead of sequential generation.
  • Flow-Based Language Models, which use invertible transformations to model complex probability distributions rather than sequential conditional probabilities, enabling bidirectional inference and exact likelihood computation.
  • Encoder-Only Language Models, which focus on text representation rather than text generation, producing contextualized embeddings without autoregressive decoding capability.
• See: Language Model, Neural Network, Text Generation, Machine Learning, Natural Language Processing, Transformer Architecture, Decoder-Only Model, GPT Architecture, Next-Token Prediction, Neural Language Model, Large Language Model, Causal Attention, Self-Attention Mechanism, Unidirectional Context Processing, Left-to-Right Generation, Instruction Tuning, Reinforcement Learning from Human Feedback, Chain-of-Thought Reasoning, In-Context Learning, Prompt Engineering, Token-by-Token Generation, Exposure Bias in Sequence Modeling, Generative Pre-Training, Teacher Forcing, Nucleus Sampling, Beam Search Algorithm, Perplexity Metric, Scaling Laws for Neural Language Models.

---

---

References

2025-05-03

• Perplexity.ai
  • Question: What are Autoregressive Language Models? Please explain their mechanisms, capabilities, common variants, architectural paradigms, and practical implications.
  • Answer: An Autoregressive Language Model is a language model that generates text by predicting each next token based on the sequence of previous tokens. Formally, an ARLM models the joint probability of a sequence $(w_1, w_2, \ldots, w_n)$ as a product of conditional probabilities: $P(w_1, \dots, w_n) = \prod_{i=1}^{n} P(w_i \mid w_1,\ldots,w_{i-1})$.
    • Definition and Terminology:
      • ARLM models generate text by iteratively predicting words left-to-right from earlier words, using the prior context as input for each prediction.
      • This approach is often called causal language modeling or a sequential language model, emphasizing that each token is generated in sequence conditioned on its predecessors.
      • Such models are also described as decoder-only or unidirectional language models in the Transformer framework, since they use only past context when predicting each word.
      • A classic example is OpenAI's GPT family, where the Transformer's decoder architecture with masked self-attention is used to ensure the model only attends to earlier tokens.
    • Input-Output Mechanism:
      • Context Window and Sequential Processing:
        • ARLMs process input and output text sequentially.
        • They maintain an autoregressive context window, which is the span of tokens the model can utilize as context for prediction.
        • Modern Transformer-based ARLMs can leverage very long context windows (thousands of tokens).
        • As the generation progresses, previously generated tokens are appended to the context (up to the limit of the context window).
        • In earlier RNN-based ARLMs (e.g. LSTM networks), context is maintained through the recurrent hidden state.
      • Token Prediction Process:
        • Autoregressive generation is a step-by-step process of predicting and appending tokens.
        • At each step, the model computes $P(w_{\text{next}} \mid \text{context})$ and selects a next token.
        • The typical generation loop involves: starting with an initial prompt, predicting the next token distribution, selecting one token, appending it to the context, and repeating.
        • This iterative procedure continues until a stopping condition is met.
        • The model generates outputs causally in one direction, enforced in Transformers by causal attention masking.
    • Key Model Behaviors and Characteristics:
      • Context Maintenance:
        • ARLMs excel at maintaining an internal representation of the prior context.
        • In recurrent architectures, this is achieved via the recurrent hidden state.
        • In Transformer architectures, the model explicitly attends to all previous tokens at each layer.
        • This enables capture of long-range dependencies and reference to earlier content in the text.
        • The limitation is the size of the context window.
      • Next-Token Prediction and Selection:
        • At each generation step, an ARLM outputs a probability distribution over the vocabulary.
        • Token selection strategies include:
          • Greedy decoding: selecting the highest-probability token.
          • Beam search: exploring multiple candidate sequences.
          • Stochastic sampling: randomly sampling according to predicted probabilities.
          • Controlled sampling: techniques like top-k sampling or top-p (nucleus) sampling.
      • Text Generation Dynamics:
        • ARLMs generate text one token at a time with each decision conditioning on prior outputs.
        • The model is essentially always "reading" what it has written so far and then extending it.
        • This autoregressive feedback loop means the model can adapt its style and content.
        • However, errors can compound: a poor word choice becomes part of the context.
        • In training, ARLMs learn via a process known as teacher forcing.
        • The downside is a train–test mismatch called exposure bias.
      • Causal Attention Masking:
        • In modern ARLMs based on the Transformer architecture, causal self-attention ensures the model attends only to past tokens.
        • A binary mask is applied in each self-attention layer.
        • This enforces the autoregressive property during training and generation.
        • Without it, a Transformer would attend bidirectionally and "cheat" by looking ahead.
        • Causal masking makes the self-attention behave like a shifting context window.
      • Training with Teacher Forcing and Its Implications:
        • ARLMs are typically trained with a teacher forcing setup.
        • During inference, the model must use its own previously generated outputs as context.
        • This discrepancy can cause error accumulation.
        • Over a long sequence, such compounding can lead to degeneration.
        • Modern large ARLMs do exhibit a degree of self-recovery.
        • Techniques like scheduled sampling have been proposed to mitigate exposure bias.
        • In practice, careful prompt engineering and constraints on generation can help guide the model.
    • Model Performance Factors:
      • Model Scale and Training Data:
        • The size of the model (number of parameters) and the scale of training data profoundly affect an ARLM's performance.
        • Larger models trained on very large corpora achieve lower perplexity and exhibit more sophisticated capabilities.
        • In recent years, large language models (LLMs) with hundreds of billions of parameters have demonstrated strong performance.
        • These improvements follow certain scaling laws.
        • Small-scale ARLMs tend to have lower fidelity and may struggle with long-range coherence.
      • Prompt Engineering:
        • Since ARLMs are conditioned on their input context, how the input (prompt) is crafted can significantly alter quality and relevance.
        • Prompt engineering is the practice of designing input prompts to elicit desired behaviors.
        • A well-constructed prompt provides clear context or instructions.
        • This has become an important technique especially for large general-purpose models.
        • Good prompts can reduce ambiguity and prevent the model from going off-topic.
      • Decoding (Sampling) Strategies:
        • The strategy used to convert the model's next-token probabilities into actual token outputs impacts the generated text.
        • Different applications call for different decoding strategies.
        • There is often a trade-off between determinism and creativity.
        • Stochastic sampling methods (especially top-p/nucleus or top-k sampling) usually yield more interesting text.
        • Adjusting the temperature of the sampling is another lever.
        • Beam search is often used in structured generation but less common in creative generation.
        • The choice of decoding strategy and hyperparameters is critical for balancing coherence, originality, and correctness.
      • Error Accumulation and Mitigation:
        • Due to their autoregressive nature, ARLMs are susceptible to error accumulation.
        • The longer the generated passage, the more opportunities for accumulated error.
        • Mitigation tactics include:
          • Careful prompt design
          • Intermediate checks (e.g., using chain-of-thought prompts)
          • Truncation or regeneration
          • Training strategies like scheduled sampling
        • There is also reinforcement learning from human feedback (RLHF) used in fine-tuning.
        • Even the largest ARLMs cannot completely eliminate error accumulation.
        • Research continues into making ARLM generation more reliable over long horizons.
    • Classification of Autoregressive Language Models:
      • By Scale (Size of Model):
        • Small-scale ARLMs have relatively few parameters or limited training data.
        • Large-scale ARLMs are the massive Large Language Models (LLMs) with billions of parameters.
        • Large models often exhibit emergent abilities.
        • As one moves from small to large scale, the model transitions from a limited predictor to a more general generator.
      • By Model Complexity:
        • A basic predictor might be a unigram or bigram model or a single-layer neural network.
        • An advanced generator is a deep, multi-layer network that can model highly nonlinear relationships.
        • Advanced ARLMs often incorporate additional techniques enabling contextually rich text generation.
        • Complexity can also refer to training regimen, with models undergoing unsupervised pre-training then supervised instruction tuning.
      • By Application Scope:
        • Task-specific ARLMs are trained or fine-tuned to excel in one particular task or domain.
        • General-purpose ARLMs are trained on very diverse corpora spanning many domains.
        • GPT-3 is a prime example of a general-purpose model.
        • Many ARLMs start general-purpose and then are fine-tuned to become more task-specific.
        • Some models are designed for interactive use while others for generating standalone text.
      • By Prediction Granularity:
        • Character-level models predict one character at a time.
        • Word-level models treat each word as a token and predict words one by one.
        • Subword/token-level models typically use subword tokenization (such as Byte Pair Encoding or WordPiece tokens).
        • Some ARLMs can also operate at byte-level granularity.
      • By Architectural Paradigm:
        • "Traditional" ARLMs include the statistical n-gram models and the early neural network models.
        • Transformer-based ARLMs are the backbone of nearly all cutting-edge LMs today.
        • These models forego recurrence entirely and instead process the input context in parallel.
        • Examples include GPT-1, GPT-2, GPT-3, GPT-Neo/GPT-J, and Meta's LLaMA.
        • This paradigm currently defines the state-of-the-art in language modeling.
      • By Processing Mechanism:
        • Recurrent processing models process one token after another, maintaining an internal state.
        • Feed-forward (non-recurrent) processing models process the context in a single shot or in parallel layers.
        • Transformer ARLMs fall in the feed-forward category.
        • Feed-forward ARLMs are typically faster to train on parallel hardware.
        • Recent advancements try to get the best of both worlds by allowing very long effective contexts in a feed-forward manner.
      • By Domain and Data Diversity:
        • Domain-specific ARLMs are trained on text from a particular domain or style.
        • Multi-domain or general-domain ARLMs are trained on a mix of data from diverse sources.
        • Models like GPT-2 and GPT-3 purposely use very diverse training data.
        • There are also multilingual ARLMs versus monolingual models.
        • Some ARLMs are adapted to multi-modal domains.
    • Conclusion:
      • Autoregressive language models form the foundation of most modern text generation systems.
      • By always conditioning on the past and predicting the future token, they mirror the sequential nature of language.
      • Mastery of ARLMs involves understanding aspects from theoretical underpinnings to practical considerations.
      • As research and engineering progress, ARLMs continue to grow in capability.
      • The autoregressive principle remains at the heart of how these models operate.

2019

• (Yang, Dai et al., 2019) ⇒ Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. (2019). “Xlnet: Generalized Autoregressive Pretraining for Language Understanding.” Advances in Neural Information Processing Systems, 32.
  • ABSTRACT: With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment setting, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.

---