Foundational Pre-Trained Neural Model

A Foundational Pre-Trained Neural Model is a pre-trained large neural model that serves as a starting point for various downstream artificial intelligence (AI) tasks.

• Context:
  • It can (typically) be trained on large Training Datasets.
  • It can be the basis for a Fine-Tuned Neural Model.
  • …
• Example(s):
  • Foundational Encoder-Only Text Model], such as:
    • BERT, RoBERTa, ALBERT, DistilBERT.
    • ...
  • a Foundation LLMs, such as:
    • Foundational Base LLM, such as GPT-3
    • Foundational Instruct LLM, such as LLaMA-Instruct model.
    • ...
  • Foundational Computer Vision Models, such as:
    • Foundational ___ CV Model, such as: ResNet, a convolutional neural network model for image recognition.
    • Foundational Object Detection CV Model, such as: YOLO, a real-time object detection system.
• Counter-Example(s):
  • a Domain-Specific Neural Model.
• See: Large Language Models, BERT (Language Model), Generative pre-Trained Transformer.

---

---

References

2023

• chat
  • A foundation model is a large pre-trained neural network model that serves as a starting point for various downstream artificial intelligence (AI) tasks. It is typically trained on massive amounts of diverse data (often from the Web), enabling it to learn and capture a wide range of language features and nuances. However, a foundation model is not typically fine-tuned for any specific task or application. If needed however, it is amenable to fine-tuning on a smaller dataset for some specific tasks. Examples of foundation models include BERT, GPT, and T5.

2023

• (Wikipedia, 2023) ⇒ https://en.wikipedia.org/wiki/Foundation_models Retrieved:2023-4-22.
  • A foundation model (also called base model) is a large artificial intelligence (AI) model trained on a vast quantity of data at scale (often by self-supervised learning or semi-supervised learning) resulting in a model that can be adapted to a wide range of downstream tasks. Foundation models have helped bring about a major transformation in how AI systems are built, such as by powering prominent chatbots and other user-facing AI. The Stanford Institute for Human-Centered Artificial Intelligence's (HAI) Center for Research on Foundation Models (CRFM) popularized the term.

    Early examples of foundation models were pre-trained large language models (LLMs) including Google's BERT and OpenAI's GPT-n series. Such broad models can in turn be used for task and/or domain specific models using sequences of other kinds of tokens, such as medical codes.

    Beyond text, several visual and multimodal foundation models have been produced— including DALL-E, Flamingo, Florence and NOOR. Visual foundation models (VFMs) have been combined with text-based LLMs to develop sophisticated task-specific models.

---