2004 EvaluatingCollaborativeFilterin

From GM-RKB
Jump to navigation Jump to search

Subject Headings:

Notes

Cited By

Quotes

Abstract

Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole. In addition to reviewing the evaluation strategies used by prior researchers, we present empirical results from the analysis of various accuracy metrics on one content domain where all the tested metrics collapsed roughly into three equivalence classes. Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated.

References

  • 1. Charu C. Aggarwal, Joel L. Wolf, Kun-Lung Wu, Philip S. Yu, Horting Hatches An Egg: A New Graph-theoretic Approach to Collaborative Filtering, Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p.201-212, August 15-18, 1999, San Diego, California, USA doi:10.1145/312129.312230
  • 2. Brian Amento, Loren Terveen, Will Hill, Deborah Hix, Robert Schulman, Experiments in Social Data Mining: The TopicShop System, ACM Transactions on Computer-Human Interaction (TOCHI), v.10 n.1, p.54-85, March 2003 doi:10.1145/606658.606661
  • 3. Brian Amento, Will Hill, Loren Terveen, Deborah Hix, Peter Ju, An Empirical Evaluation of User Interfaces for Topic Management of Web Sites, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, p.552-559, May 15-20, 1999, Pittsburgh, Pennsylvania, USA doi:10.1145/302979.303156
  • 4. Ricardo A. Baeza-Yates, Berthier Ribeiro-Neto, Modern Information Retrieval, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
  • 5. Bailey, B. P., Gurak, L. J., and Konstan, J. A. 2001. An Examination of Trust Production in Computer-mediated Exchange. In: Proceedings of the 7th Conference on Human Factors and the Web (July).]].
  • 6. Marko Balabanović, Yoav Shoham, Fab: Content-based, Collaborative Recommendation, Communications of the ACM, v.40 n.3, p.66-72, March 1997 doi:10.1145/245108.245124
  • 7. Chumki Basu, Haym Hirsh, William Cohen, Recommendation As Classification: Using Social and Content-based Information in Recommendation, Proceedings of the Fifteenth National/tenth Conference on Artificial Intelligence/Innovative Applications of Artificial Intelligence, p.714-720, July 1998, Madison, Wisconsin, USA
  • 8. Daniel Billsus, Michael J. Pazzani, Learning Collaborative Information Filters, Proceedings of the Fifteenth International Conference on Machine Learning, p.46-54, July 24-27, 1998
  • 9. Breese, J. S., Heckerman, D., and Kadie, C. 1998. Empirical Analysis of Predictive Algorithms for Collaborative Filtering. In: Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence (UAI-98). G. F. Cooper, and S. Moral, Eds. Morgan-Kaufmann, San Francisco, Calif., 43--52.]].
  • 10. John Canny, Collaborative Filtering with Privacy via Factor Analysis, Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 11-15, 2002, Tampere, Finland doi:10.1145/564376.564419
  • 11. Mark Claypool, David Brown, Phong Le, Makoto Waseda, Inferring User Interest, IEEE Internet Computing, v.5 n.6, p.32-39, November 2001 doi:10.1109/4236.968829
  • 12. Cleverdon, C. and Kean, M. 1968. Factors Determining the Performance of Indexing Systems. Aslib Cranfield Research Project, Cranfield, England.]].
  • 13. Dan Cosley, Shyong K. Lam, Istvan Albert, Joseph A. Konstan, John Riedl, Is Seeing Believing?: How Recommender System Interfaces Affect Users' Opinions, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA doi:10.1145/642611.642713
  • 14. Dahlen, B. J., Konstan, J. A., Herlocker, J. L., Good, N., Borchers, A., and Riedl, J. 1998. Jump-starting Movielens: User Benefits of Starting a Collaborative Filtering System with "dead Data". TR 98-017. University of Minnesota.]].
  • 15. Pedro Domingos, Matt Richardson, Mining the Network Value of Customers, Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p.57-66, August 26-29, 2001, San Francisco, California doi:10.1145/502512.502525
  • 16. David Goldberg, David Nichols, Brian M. Oki, Douglas Terry, Using Collaborative Filtering to Weave An Information Tapestry, Communications of the ACM, v.35 n.12, p.61-70, Dec. 1992 doi:10.1145/138859.138867
  • 17. Ken Goldberg, Theresa Roeder, Dhruv Gupta, Chris Perkins, Eigentaste: A Constant Time Collaborative Filtering Algorithm, Information Retrieval, v.4 n.2, p.133-151, July 2001 doi:10.1023/A:1011419012209
  • 18. Nathaniel Good, J. Ben Schafer, Joseph A. Konstan, Al Borchers, Badrul Sarwar, Jon Herlocker, John Riedl, Combining Collaborative Filtering with Personal Agents for Better Recommendations, Proceedings of the Sixteenth National Conference on Artificial Intelligence and the Eleventh Innovative Applications of Artificial Intelligence Conference Innovative Applications of Artificial Intelligence, p.439-446, July 18-22, 1999, Orlando, Florida, USA
  • 19. Hanley, J. A. and Mcneil, B. J. 1982. The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve. Radiology 143, 29--36.]].
  • 20. Harman, D. 1995. The TREC Conferences. Hypertext---Information Retrieval---Multimedia: Synergieeffekte Elektronisher Informationssysteme. In: Proceedings of HIM '95.]].
  • 21. Stephen P. Harter, Variations in Relevance Assessments and the Measurement of Retrieval Effectiveness, Journal of the American Society for Information Science, v.47 n.1, p.37-49, Jan. 1996 <37::AID-ASI4>3.3.CO;2-I doi:10.1002/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.3.CO;2-I
  • 22. David Heckerman, David Maxwell Chickering, Christopher Meek, Robert Rounthwaite, Carl Kadie, Dependency Networks for Inference, Collaborative Filtering, and Data Visualization, The Journal of Machine Learning Research, 1, p.49-75, 9/1/2001 doi:10.1162/153244301753344614
  • 23. Martin G. Helander, Thomas K. Landauer, Prasad V. Prabhu, Handbook of Human-Computer Interaction, Elsevier Science Inc., New York, NY, 1997
  • 24. Jonathan L. Herlocker, Joseph A. Konstan, Al Borchers, John Riedl, An Algorithmic Framework for Performing Collaborative Filtering, Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, p.230-237, August 15-19, 1999, Berkeley, California, USA doi:10.1145/312624.312682
  • 25. Jonathan L. Herlocker, Joseph A. Konstan, John Riedl, Explaining Collaborative Filtering Recommendations, Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work, p.241-250, December 2000, Philadelphia, Pennsylvania, USA doi:10.1145/358916.358995
  • 26. Jon Herlocker, Joseph A. Konstan, John Riedl, An Empirical Analysis of Design Choices in Neighborhood-Based Collaborative Filtering Algorithms, Information Retrieval, v.5 n.4, p.287-310, October 2002 doi:10.1023/A:1020443909834
  • 27. Will Hill, Larry Stead, Mark Rosenstein, George Furnas, Recommending and Evaluating Choices in a Virtual Community of Use, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, p.194-201, May 07-11, 1995, Denver, Colorado, USA doi:10.1145/223904.223929
  • 28. Joseph A. Konstan, Bradley N. Miller, David Maltz, Jonathan L. Herlocker, Lee R. Gordon, John Riedl, GroupLens: Applying Collaborative Filtering to Usenet News, Communications of the ACM, v.40 n.3, p.77-87, March 1997 doi:10.1145/245108.245126
  • 29. Le, C. T., Lindren, B. R. 1995. Construction and Comparison of Two Receiver Operating Characteristics Curves Derived from the Same Samples. Biom. J. 37, 869--877.]].
  • 30. Linton, F., Charron, A., and Joy, D. 1998. OWL: A Recommender System for Organization-wide Learning. In: Proceedings of the 1998 Workshop on Recommender Systems 65--69.]].
  • 31. David W. McDonald, Evaluating Expertise Recommendations, Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, September 30-October 03, 2001, Boulder, Colorado, USA doi:10.1145/500286.500319
  • 32. Sean M. McNee, Istvan Albert, Dan Cosley, Prateep Gopalkrishnan, Shyong K. Lam, Al Mamunur Rashid, Joseph A. Konstan, John Riedl, On the Recommending of Citations for Research Papers, Proceedings of the 2002 ACM Conference on Computer Supported Cooperative Work, November 16-20, 2002, New Orleans, Louisiana, USA doi:10.1145/587078.587096
  • 33. Bradley N. Miller, Istvan Albert, Shyong K. Lam, Joseph A. Konstan, John Riedl, MovieLens Unplugged: Experiences with An Occasionally Connected Recommender System, Proceedings of the 8th International Conference on Intelligent User Interfaces, January 12-15, 2003, Miami, Florida, USA doi:10.1145/604045.604094
  • 34. Bradley N. Miller, John T. Riedl, Joseph A. Konstan, Experiences with GroupLens: Marking Usenet Useful Again, Proceedings of the Annual Conference on USENIX Annual Technical Conference, p.17-17, January 06-10, 1997, Anaheim, California
  • 35. Bamshad Mobasher, Honghua Dai, Tao Luo, Miki Nakagawa, Effective Personalization based on Association Rule Discovery from Web Usage Data, Proceedings of the 3rd International Workshop on Web Information and Data Management, November 09-01, 2001, Atlanta, Georgia, USA doi:10.1145/502932.502935
  • 36. Masahiro Morita, Yoichi Shinoda, Information Filtering based on User Behavior Analysis and Best Match Text Retrieval, Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, p.272-281, July 03-06, 1994, Dublin, Ireland
  • 37. Mui, L., Ang, C., and Mohtashemi, M. 2001. A Probabilistic Model for Collaborative Sanctioning. Technical Memorandum 617. MIT LCS.]].
  • 38. William M. Newman, Better Or Just Different? On the Benefits of Designing Interactive Systems in Terms of Critical Parameters, Proceedings of the 2nd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, p.239-245, August 18-20, 1997, Amsterdam, The Netherlands doi:10.1145/263552.263615
  • 39. Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1995
  • 40. David M. Pennock, Eric Horvitz, Steve Lawrence, C. Lee Giles, Collaborative Filtering by Personality Diagnosis: A Hybrid Memory and Model-Based Approach, Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, p.473-480, June 30-July 03, 2000
  • 41. Al Mamunur Rashid, Istvan Albert, Dan Cosley, Shyong K. Lam, Sean M. McNee, Joseph A. Konstan, John Riedl, Getting to Know You: Learning New User Preferences in Recommender Systems, Proceedings of the 7th International Conference on Intelligent User Interfaces, January 13-16, 2002, San Francisco, California, USA doi:10.1145/502716.502737
  • 42. P. Krishna Reddy, Masaru Kitsuregawa, P. Sreekanth, S. Srinivasa Rao, A Graph Based Approach to Extract a Neighborhood Customer Community for Collaborative Filtering, Proceedings of the Second International Workshop on Databases in Networked Information Systems, p.188-200, December 16-18, 2002
  • 43. Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom, John Riedl, GroupLens: An Open Architecture for Collaborative Filtering of Netnews, Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work, p.175-186, October 22-26, 1994, Chapel Hill, North Carolina, USA doi:10.1145/192844.192905
  • 44. Paul Resnick, Hal R. Varian, Recommender Systems, Communications of the ACM, v.40 n.3, p.56-58, March 1997 doi:10.1145/245108.245121
  • 45. Rogers, S. C. 2001. Marketing Strategies, Tactics, and Techniques : A Handbook for Practitioners. Quorum Books, Westport, Conn.]].
  • 46. Badrul Sarwar, George Karypis, Joseph Konstan, John Riedl, Analysis of Recommendation Algorithms for E-commerce, Proceedings of the 2nd ACM Conference on Electronic Commerce, p.158-167, October 17-20, 2000, Minneapolis, Minnesota, USA doi:10.1145/352871.352887
  • 47. Sarwar, B. M., Karypis, G., Konstan, J. A., and Riedl, J. 2000b. Application of Dimensionality Reduction in Recommender System--A Case Study. In: Proceedings of the ACM WebKDD 2000 Web Mining for E-Commerce Workshop.]].
  • 48. Badrul Sarwar, George Karypis, Joseph Konstan, John Riedl, Item-based Collaborative Filtering Recommendation Algorithms, Proceedings of the 10th International Conference on World Wide Web, p.285-295, May 01-05, 2001, Hong Kong, Hong Kong doi:10.1145/371920.372071
  • 49. Badrul M. Sarwar, Joseph A. Konstan, Al Borchers, Jon Herlocker, Brad Miller, John Riedl, Using Filtering Agents to Improve Prediction Quality in the GroupLens Research Collaborative Filtering System, Proceedings of the 1998 ACM Conference on Computer Supported Cooperative Work, p.345-354, November 14-18, 1998, Seattle, Washington, USA doi:10.1145/289444.289509
  • 50. J. Ben Schafer, Joseph A. Konstan, John Riedl, Meta-recommendation Systems: User-controlled Integration of Diverse Recommendations, Proceedings of the Eleventh International Conference on Information and Knowledge Management, November 04-09, 2002, McLean, Virginia, USA doi:10.1145/584792.584803
  • 51. Andrew I. Schein, Alexandrin Popescul, Lyle H. Ungar, David M. Pennock, Methods and Metrics for Cold-start Recommendations, Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 11-15, 2002, Tampere, Finland doi:10.1145/564376.564421
  • 52. Andrew I. Schein, Alexandrin Popescul, Lyle H. Ungar, David M. Pennock, Methods and Metrics for Cold-start Recommendations, Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 11-15, 2002, Tampere, Finland doi:10.1145/564376.564421
  • 53. Upendra Shardanand, Pattie Maes, Social Information Filtering: Algorithms for Automating “word of Mouth”, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, p.210-217, May 07-11, 1995, Denver, Colorado, USA doi:10.1145/223904.223931
  • 54. Rashmi Sinha, Kirsten Swearingen, The Role of Transparency in Recommender Systems, CHI '02 Extended Abstracts on Human Factors in Computing Systems, April 20-25, 2002, Minneapolis, Minnesota, USA doi:10.1145/506443.506619
  • 55. Swearingen, K. and Sinha, R. 2001. Beyond Algorithms: An HCI Perspective on Recommender Systems. In: Proceedings of the SIGIR 2001 Workshop on Recommender Systems.]].
  • 56. Swets, J. A. 1963. Information Retrieval Systems. Science 141, 245--250.]].
  • 57. Swets, J. A. 1969. Effectiveness of Information Retrieval Methods. Amer. Doc. 20, 72--89.]].
  • 58. Andrew H. Turpin, William Hersh, Why Batch and User Evaluations Do Not Give the Same Results, Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, p.225-231, September 2001, New Orleans, Louisiana, USA doi:10.1145/383952.383992
  • 59. Voorhees, E. M. and Harman, D. K. 1999. Overview of the Seventh Text REtrieval Conference (TREC-7). In NIST Special Publication 500-242 (July), E. M. Voorhees, and D. K. Harman, Eds. NIST, 1--24.]].
  • 60. Alan Wexelblat, Pattie Maes, Footprints: History-rich Tools for Information Foraging, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, p.270-277, May 15-20, 1999, Pittsburgh, Pennsylvania, USA doi:10.1145/302979.303060
  • 61. Whittaker, S., Terveen, L. G., and Nardi, B. 2000. Let's Stop Pushing the Envelope and Start Addressing It: A Reference Task Agenda for HCI. Human-Computer Interact. 15, 2-3 (Sept.), 75--106.]].
  • 62. Y. Y. Yao, Measuring Retrieval Effectiveness based on User Preference of Documents, Journal of the American Society for Information Science, v.46 n.2, p.133-145, March 1995 <133::AID-ASI6>3.0.CO;2-Z doi:10.1002/(SICI)1097-4571(199503)46:2<133::AID-ASI6>3.0.CO;2-Z

}};


 AuthorvolumeDate ValuetitletypejournaltitleUrldoinoteyear
2004 EvaluatingCollaborativeFilterinJoseph A. Konstan
Jonathan L. Herlocker
Loren G. Terveen
John T. Riedl		Evaluating Collaborative Filtering Recommender Systems10.1145/963770.9637722004
Retrieved from "http://www.gabormelli.com/RKB/index.php?title=2004_EvaluatingCollaborativeFilterin&oldid=844658"