DSPy

A DSPy is a Programming Framework that enables the structured programming of Foundation Models with a focus on optimizing Language Model prompts and configurations through declarative, self-improving language programs.

• Context:
  • It can (typically) provide a structured approach to optimize Language Model prompts and configurations, addressing the limitations of traditional prompt engineering.
  • It can (typically) utilize "Signatures" to define task-specific behavior of Language Models through a declarative specification of input and output behavior, facilitating more systematic and high-level optimization.
  • It can (often) compile declarative Language Model calls into self-improving pipelines, leveraging a combination of signatures, modules, and teleprompters to optimize the usage of Language Models in arbitrary computational graphs.
  • It can (often) require modest labeling efforts for its optimization process, as it can extrapolate intermediate labels needed to support the entire pipeline from a few instances containing questions and their human-annotated answers.
  • It can (often) be used to address complex task solving using Language Models and retrieval models, offering a promising direction for future developments in Artificial Intelligence and Machine Learning.
  • ...
• Example(s):
  • A program that utilizes DSPy to optimize a Chain Of Thought process for complex question answering by generating and refining prompts based on a structured signature of input and output fields.
  • A retrieval-augmented generation (RAG) pipeline developed with DSPy, where labels are only needed for the initial query and the final response, with DSPy extrapolating necessary intermediate labels.
  • ...
• Counter-Example(s):
  • A simple Script that uses hardcoded prompts for a Language Model without any optimization.
  • A Machine Learning Pipeline that relies solely on manual prompt engineering and does not employ a structured programming approach for Language Model optimization.
• See: Programming Framework, Foundation Model, Language Model, Machine Learning Pipeline, Artificial Intelligence.

---

---

References

2024

• https://github.com/stanfordnlp/dspy
  • NOTES:
    • It represents a novel paradigm in leveraging language models (LMs) for complex tasks, emphasizing algorithmic optimization of LM prompts and weights.
    • It facilitates the systematic assembly and refinement of LM-driven pipelines, moving away from labor-intensive manual fine-tuning processes.
    • It introduces LM-driven algorithms or optimizers that autonomously adjust prompts and weights to enhance task-specific performance and reliability.
    • It is versatile enough to improve the performance of both cutting-edge models like GPT-3.5 and GPT-4 and localized models such as T5-base or Llama2-13b.
    • It analogizes the simplification of neural network development, where declarative layers and parameter learning replace manual coding and tuning efforts.
    • It outlines an iterative development cycle for task optimization, from defining the task and desired outcomes to constructing and refining pipelines with specialized modules and optimizers.
    • It is positioned as an evolving framework aimed at systematizing prompt engineering and pipeline optimization with structured modules and algorithmic improvements.

2024

• (Oosterlinck et al., 2024) ⇒ D'Oosterlinck, Karel, Omar Khattab, François Remy, Thomas Demeester, Chris Develder, and Christopher Potts. (2024). “In-Context Learning for Extreme Multi-Label Classification.” arXiv preprint arXiv:2401.12178
  • ABSTRACT: Multi-label classification problems with thousands of classes are hard to solve with in-context learning alone, as language models (LMs) might lack prior knowledge about the precise classes or how to assign them, and it is generally infeasible to demonstrate every class in a prompt. We propose a general program, 𝙸𝚗𝚏𝚎𝚛--𝚁𝚎𝚝𝚛𝚒𝚎𝚟𝚎--𝚁𝚊𝚗𝚔, that defines multi-step interactions between LMs and retrievers to efficiently tackle such problems. We implement this program using the 𝙳𝚂𝙿𝚢 programming model, which specifies in-context systems in a declarative manner, and use 𝙳𝚂𝙿𝚢 optimizers to tune it towards specific datasets by bootstrapping only tens of few-shot examples. Our primary extreme classification program, optimized separately for each task, attains state-of-the-art results across three benchmarks (HOUSE, TECH, TECHWOLF). We apply the same program to a benchmark with vastly different characteristics and attain competitive performance as well (BioDEX). Unlike prior work, our proposed solution requires no finetuning, is easily applicable to new tasks, alleviates prompt engineering, and requires only tens of labeled examples. Our code is public at this https URL.

2023

• (Khattab et al., 2023) ⇒ Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq et al. (2023). “Dspy: Compiling Declarative Language Model Calls Into Self-improving Pipelines.” arXiv preprint arXiv:2310.03714

2023

• (Singhvi et al., 2023) ⇒ Arnav Singhvi, Manish Shetty, Shangyin Tan, Christopher Potts, Koushik Sen, Matei Zaharia, and Omar Khattab. (2023). “DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines.” arXiv preprint arXiv:2312.13382

---