Human-AI Collaborative Taxonomy Development

A Human-AI Collaborative Taxonomy Development is a taxonomy development task where human experts and artificial intelligence systems to jointly create structured classification systems (taxonomies), enhancing the organization and retrieval of knowledge across various domains through iterative feedback loops and complementary.

• AKA: AI-Assisted Taxonomy Creation Task, Human-AI Collaborative Taxonomy Construction, Hybrid Intelligence Taxonomy Design, Hybrid Taxonomy Engineering.
• Context:
  • It can involve AI models generating initial taxonomy structures based on large datasets, which are then refined through human expert feedback.
  • It can facilitate the creation of domain-specific taxonomies, particularly in fields where nuanced understanding is essential, such as profession-specific writing.
  • It can improve the adaptability and relevance of taxonomies by incorporating iterative human-AI interactions.
  • It can leverage methodologies like the three-step approach: taxonomy generation by AI, validation through human-AI dialogue, and merging/testing to finalize the taxonomy.
  • It combines human contextual understanding with AI pattern recognition to balance domain expertise and computational scalability in taxonomy creation.
  • It of ten implementats the use LLMs for candidate term generation while humans perform semantic validation and hierarchy optimization.
  • It can include key challenges for maintaining conceptual consistency across iterative versions and preventing AI hallucination in category suggestions.
  • It can require ethical considerations addressing algorithmic bias through diverse expert panels and transparency mechanisms in decision pathways.
  • ...
• Example(s):
  • Human-AI Collaborative Taxonomy Construction: A Case Study in Profession-Specific Writing Assistants, which demonstrates the development of taxonomies for business writing contexts through iterative human-AI collaboration.
  • To Classify is to Interpret: Building Taxonomies from Heterogeneous Data through Human-AI Collaboration, which explores taxonomy construction in data-rich environments using human-AI partnerships.
  • Developing Medical Diagnosis Taxonomy through clinician-AI collaboration, where GPT-4 proposes symptom relationships validated by physicians.
  • Creating AI Ethics Harm Classification with crowdsourced incident reports analyzed by NLP models and categorized by ethicists.
  • Building Software Engineering Interaction Taxonomy through analysis of 10,000 developer-GitHub Copilot exchanges.
  • Designing Educational Competency Framework where instructors refine AI-generated skill maps using Bloom's taxonomy principles.
• Counter-Example(s):
  • Fully automated taxonomy generation systems that do not incorporate human feedback, potentially lacking contextual accuracy.
  • Manual taxonomy development processes that do not utilize AI capabilities, which may be time-consuming and less scalable.
  • Algorithmic Taxonomy Generation: Fully automated clustering approaches lacking human validation, like unsupervised topic modeling outputs.
  • Expert-Driven Classification: Traditional manual taxonomy development without AI augmentation, as seen in early ICD disease codes.
  • Static Taxonomies: Fixed classification systems without continuous AI-assisted updates, such as outdated library categorization schemes.
  • Black Box Categorization: Opaque AI systems producing taxonomies without explainable decision pathways or human audit trails.
  • ..
• See: Human-AI Interaction Patterns, Knowledge Graph Curation, Ontology Engineering, Co-Design Methodology, Explainable AI Governance, NIST AI Use Taxonomy, AI Harms Classification, Automatic Taxonomy Construction, Human-AI Collaboration, Knowledge Organization Systems, Hybrid Intelligence Systems.

---

---

References

2024

• (Lee et al., 2024) ⇒ Minhwa Lee, Zae Myung Kim, Vivek Khetan, & Dongyeop Kang (2024). "Human-AI Collaborative Taxonomy Construction: A Case Study in Profession-Specific Writing Assistants". In: Proceedings of CHI In2Writing Workshop 2024, arXiv:2406.18675.
  • QUOTE: Human-AI collaborative taxonomy construction serves as foundational step for developing AI writing assistants that precisely interpret and generate text aligned with profession-specific requirements.

    This three-stage methodology integrates iterative expert feedback and LLM-mediated validation processes to address domain-specific nuances in business writing contexts.

    Taxonomy merging phase employs AI consensus-building to mitigate individual biases while maintaining domain coherence.

2023

• (Meier et al., 2023) ⇒ S. Meier, L. Gygli, & N. Henze. (2023). "To Classify is to Interpret: Building Taxonomies from Heterogeneous Data through Human-AI Collaboration". In: Proceedings of Mensch und Computer 2023, arXiv: 2307.16481
  • QUOTE: Taxonomy building is a task that requires interpreting and classifying data within a given frame of reference, which comes to play in many areas of application that deal with knowledge and information organization.

    We propose an approach that allows the user to iteratively take into account multiple model's outputs as part of their sensemaking process.

2021

• (Dellermann et al., 2021) ⇒ D. Dellermann, M. Durward, S. Szopinski, C. J. vom Brocke, & O. Thomas. (2021). "The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems". In: arXiv Preprint: arXiv:2105.03354.
  • QUOTE: Such systems possess the ability to accomplish complex goals by combining human and artificial intelligence to collectively achieve superior results and continuously improve by learning from each other.

    Following a taxonomy development method, this article provides three main contributions: First, we present a structured overview of interdisciplinary research on the role of humans in the machine learning pipeline. Second, we envision hybrid intelligence systems and conceptualize the relevant dimensions for system design for the first time.

---