2004 ProjectHaloTowardsaDigiAristotle
- (Friedland et al., 2004) ⇒ Noah S. Friedland, Paul G. Allen, Gavin Matthews, Michael Witbrock, David Baxter, Jon Curtis, Blake Shepard, Pierluigi Miraglia, Jürgen Angele, Steffen Staab, Eddie Moench, Henrik Oppermann, Dirk Wenke, David Israel, Vinay Chaudhri, Bruce Porter, Ken Barker, James Fan, Shaw Yi Chaw, Peter Yeh, Dan Tecuci, and Peter Clark. (2004) "Project Halo: Towards a Digital Aristotle.” In: The AI magazine. (25:4)
Subject Headings: Project Halo.
Notes
Cited By
Quotes
Abstract
Project Halo is a multistaged effort, sponsored by Vulcan Inc, aimed at creating Digital Aristotle, an application that will encompass much of the world's scientific knowledge and be capable of applying sophisticated problem solving to answer novel questions. Vulcan envisions two primary roles for Digital Aristotle: as a tutor to instruct students in the sciences and as an interdisciplinary research assistant to help scientists in their work.
As a first step towards this goal, we have just completed a six-month pilot phase designed to assess the state of the art in applied knowledge representation and reasoning (KR&R). Vulcan selected three teams, each of which was to formally represent 70 pages from the advanced placement (AP) chemistry syllabus and deliver knowledge-based systems capable of answering questions on that syllabus. The evaluation quantified each system's coverage of the syllabus in terms of its ability to answer novel, previously unseen questions and to provide human-readable answer justifications. These justifications will play a critical role in building user trust in the question-answering capabilities of Digital Aristotle.
Prior to the final evaluation, a "failure taxonomy" was collaboratively developed in an attempt to standardize failure analysis and to facilitate cross-platform comparisons. Despite differences in approach, all three systems did very well on the challenge, achieving performance comparable to the human median. The analysis also provided key insights into how the approaches might he scaled, while at the same time suggesting how the cost of producing such systems might be reduced. This outcome leaves us highly optimistic that the technical challenges facing this effort in the years to come can be identified and overcome.
This article presents the motivation and longterm goals of Project Halo, describes in detail the six-month first phase of the project - the Halo Pilot-its KR&R challenge, empirical evaluation, results, and failure analysis. The pilot's outcome is used to define challenges for the next phase of the project and beyond.
References
- Voorheese, E.M., The Twelfth Text Retrieval Conference (TREC2003), . 2003, Department of Commerce, National Institute of Standards and Technology. See http://trec.nist.gov/pubs/trec12/t12_proceedings.html.
- Brown, T.L., et al., Chemistry: The Central Science. 2003, New Jersey: Prentice Hall.
- Barker, K., et al. A Question-answering System for AP Chemistry: Assessing KR&R Technologies. in 9th International Conference on Principles of Knowledge Representation and Reasoning. 2004. Whistler, Canada.
- Angele, J., et al. Ontology-based Query and Answering in Chemistry: Ontonova @ Project Halo. in Proceedings of the Second International Semantic Web Conference (ISWC2003). 2003: Springer Verlag.
- Witbrock, M. and G. Matthews, Cycorp Project Halo Final Report. 2003.
- Barker, K., B. Porter, and P. Clark, A Library of Generic Concepts for Composing Knowledge Bases, in Proceedings of 1st Int Conf on Knowledge Capture (K-Cap'01). 2001. p. 14--21
- Novak, G., Conversion of Units of Measurement. IEEE Transactions on Software Engineering, 1995. 21(8): p. 651-661.
- Kifer, M., G. Lausen, and J. Wu, Logical Foundations of Object Oriented and Frame Based Languages. Journal of the ACM, 1995. 42: p. 741--843.
- Clark, P. and B. Porter, KM -- The Knowledge Machine: Users Manual. 1999. The system code and documentation are available at http://www.cs.utexas.edu/users/mfkb/km.html.
- Friedland, N., et al. Towards a Quantitative Platform-independent Quantitative Analysis of Knowledge Systems. in Proceedings of the 9th International Conference of Knowledge Representation and Reasoning. 2004. Whistler, Canada: AAAI Press.
- Clancey, W.J., The epistemology of a rule-based expert system: A framework for explanation. Artificial Intelligence, 1983. 20: p. 215-251.
- Chaudhri, V. and R. Fikes, eds. AAAI Fall Symposium on Question Answering Systems. . 1999, AAAI.
- Neches, R., et al., Enabling Technology for Knowledge Sharing. AI Magazine, 1991. 12(3): p. 36--56.
- Cohen, P., et al., The DARPA High Performance Knowledge Bases Project. AI Magazine, 1998. 19(4): p. 25--49.
- Brachman, R.J., et al., Reducing CLASSIC to “Practice: Knowledge Representation Theory Meets Reality. Artificial Intelligence Journal, 1999. 114: p. 203-237.
- Keyes, J., Why Expert Systems Fail? IEEE Expert, 1989. 4: p. 50-53.
- Batanov, D. and P. Brezillon, eds. First International Conference on Successes and Failures of Knowledge-based Systems in Real World Applications. . 1996, Asian Institute of Technology: Bangkok, Thailand.
- Pool, M., J.F. K. Murray, M. Mehrotra, R. Schrag, J. Blythe, and H.C. J. Kim, P. Miraglia, T. Russ, and D. Schneider. Evaluation of Expert Knowledge Elicited for Critiquing Military Courses of Action. in Proceedings of the Second International Conference on Knowledge Capture (KCAP-03). 2003.,