AI Commonsense Benchmark Task

An AI Commonsense Benchmark Task is a commonsense task that is a reasoning benchmark task (that involves common sense knowledge).

• Context:
  • …
• Example(s):
  • …
• Counter-Example(s):
  • Mathematical Reasoning Benchmark, like geometric reasoning benchmark.
  • Statistical Reasoning Benchmark.
  • Deductive Reasoning Benchmark.
• See: Naive Physics Benchmark, Perception Task, Planning Task.

---

---

References

2023

• (Richardson & Heck, 2023) ⇒ Christopher Richardson, and Larry Heck. (2023). “Commonsense Reasoning for Conversational AI: A Survey of the State of the Art.” arXiv preprint arXiv:2302.07926
  • ABSTRACT: Large, transformer-based pretrained language models like BERT, GPT, and T5 have demonstrated a deep understanding of contextual semantics and language syntax. Their success has enabled significant advances in conversational AI, including the development of open-dialogue systems capable of coherent, salient conversations which can answer questions, chat casually, and complete tasks. However, state-of-the-art models still struggle with tasks that involve higher levels of reasoning - including commonsense reasoning that humans find trivial. This paper presents a survey of recent conversational AI research focused on commonsense reasoning. The paper lists relevant training datasets and describes the primary approaches to include commonsense in conversational AI. The paper also discusses benchmarks used for evaluating commonsense in conversational AI problems. Finally, the paper presents preliminary observations of the limited commonsense capabilities of two state-of-the-art open dialogue models, BlenderBot3 and LaMDA, and its negative effect on natural interactions. These observations further motivate research on commonsense reasoning in conversational AI.

2023

• (Ernest Davis, 2023) ⇒ Ernest Davis. (2023). “Benchmarks for Automated Commonsense Reasoning: A Survey."
  • ABSTRACT: More than one hundred benchmarks have been developed to test the commonsense knowledge and commonsense reasoning abilities of artificial intelligence (AI) systems. However, these benchmarks are often flawed and many aspects of common sense remain untested. Consequently, we do not currently have any reliable way of measuring to what extent existing AI systems have achieved these abilities. This paper surveys the development and uses of AI commonsense benchmarks. We discuss the nature of common sense; the role of common sense in AI; the goals served by constructing commonsense benchmarks; and desirable features of commonsense benchmarks. We analyze the common flaws in benchmarks, and we argue that it is worthwhile to invest the work needed ensure that benchmark examples are consistently high quality. We survey the various methods of constructing commonsense benchmarks. We enumerate 139 commonsense benchmarks that have been developed: 102 text-based, 18 image-based, 12 video based, and 7 simulated physical environments. We discuss the gaps in the existing benchmarks and aspects of commonsense reasoning that are not addressed in any existing benchmark. We conclude with a number of recommendations for future development of commonsense AI benchmarks.

---