Chatbot System Evaluation Task

A Chatbot System Evaluation Task is an AI system evaluation task for chatbot systems, which involves assessing various aspects of a chatbot features (including conversational abilities, accuracy of responses, user satisfaction, and the ability to handle different types of queries effectively).

• Context:
  • input: an Chatbot System.
    • input (optional): a Chatbot System Benchmark (with a chatbot evaluation dataset), a Standardized Chatbot Testing Protocol.
  • outout: a Chatbot Evaluation Result.
  • It can be supported by an Chatbot Evaluation System.
  • It can range from being a Chatbot Response Evaluation Task to being a Chatbot Performance Evaluation Task.
  • It can range from being a User Feedback-Based Chatbot System Evaluation, and Expert Feedback-Based Chatbot System Evaluation.
  • It can range from being a General Chatbot Evaluation Task to being a Chatbot Skill-Specific Evaluation Task (of a chatbot skill).
  • It can range from being a Quantitative Chatbot System Evaluation Task to being a Qualitative Chatbot System Evaluation Task.
  • It can range from being an Offline Chatbot System Evaluation Task to being a Production Chatbot System Evaluation Task.
  • It can range from being a Manual Chatbot System Evaluation Task to an Automated Chatbot System Evaluation Task.
  • ...
• Example(s):
  • Customer Service Chatbot Evaluation Task for customer service chatbots behaviors, such as: customer service chatbot customer effort score (CES).
  • An Informational Chatbot User Satisfaction Survey Task for informational chatbot interactions, focusing on measures like: informational chatbot user satisfaction.
  • A Healthcare Chatbot Accuracy Testing Task for healthcare chatbot diagnostics, evaluating aspects like: Healthcare Chatbot Diagnostic Accuracy.
  • A Retail Chatbot Efficiency Comparison Task for retail chatbot purchasing guidance, measuring metrics such as: Retail Chatbot Transaction Efficiency.
  • A Conversational AI Linguistic Quality Analysis Task for conversational chatbot, to assess: chatbot conversational linguistic coherence.
  • A Pre-Signature Contract Chatbot Evaluation Task for a pre-signature contract chatbot, focusing on capabilities like: pre-signature contract chatbot user satisfaction.
  • …
• Counter-Example(s):
  • An Image Recognition System Evaluation Task for image recognition systemss.
  • A Recommendation System Evaluation Task for recommender systems
  • Static FAQ System Evaluation that does not involve conversational interaction.
  • …
• See: Chatbot, Natural Language Processing, Human-Computer Interaction, User Experience Design, Conversational AI.

---

---

References

2020

• (Celikyilmaz et al., 2020) ⇒ Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. (2020). “Evaluation of Text Generation: A Survey.” In: arXiv preprint arXiv:2006.14799.

2020

• (Almansor & Hussain, 2020) ⇒ Eman H. Almansor, and Farookh Khadeer Hussain. (2020). “Survey on Intelligent Chatbots: State-of-the-Art and Future Research Directions.” In: CISIS 2019. Advances in Intelligent Systems and Computing, vol 993. Springer, Cham. https://doi.org/10.1007/978-3-030-22354-0_47

2019

• (Maroengsit et al., 2019) ⇒ Waranpong Maroengsit, Thunyathon Piyakulpinyo, Kanokphan Phonyiam, and others. (2019). “A Survey on Evaluation Methods for Chatbots.” In: Processings 7th International Conference on Information and Education Technology. https://doi.org/10.1145/3323771.3323824

2018

• (Peras, 2018) ⇒ Damir Peras. (2018). “Chatbot Evaluation Metrics.” In: Economic and Social Development: Book of Proceedings (Dec 14/Dec 15, 2018).

---