2004 DataMininginMetricSpaceAnEmpiri

(Caruana & Niculescu-Mizil, 2004) ⇒ Rich Caruana, and Alexandru Niculescu-Mizil. (2004). “Data Mining in Metric Space: An Empirical Analysis of Supervised Learning Performance Criteria.” In: Proceedings of the tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ISBN:1-58113-888-1 doi:10.1145/1014052.1014063

Subject Headings:

Notes

Cited By

Quotes

Abstract

Many criteria can be used to evaluate the performance of supervised learning. Different criteria are appropriate in different settings, and it is not always clear which criteria to use. A further complication is that learning methods that perform well on one criterion may not perform well on other criteria. For example, SVMs and boosting are designed to optimize accuracy, whereas neural nets typically optimize squared error or cross entropy. We conducted an empirical study using a variety of learning methods (SVMs, neural nets, k-nearest neighbor, bagged and boosted trees, and boosted stumps) to compare nine boolean classification performance metrics: Accuracy, Lift, F-Score, Area under the ROC Curve, Average Precision, Precision/Recall Break-Even Point, Squared Error, Cross Entropy, and Probability Calibration. Multidimensional scaling (MDS) shows that these metrics span a low dimensional manifold. The three metrics that are appropriate when predictions are interpreted as probabilities: squared error, cross entropy, and calibration, lay in one part of metric space far away from metrics that depend on the relative order of the predicted values: ROC area, average precision, break-even point, and lift. In between them fall two metrics that depend on comparing predictions to a threshold: accuracy and F-score. As expected, maximum margin methods such as SVMs and boosted trees have excellent performance on metrics like accuracy, but perform poorly on probability metrics such as squared error. What was not expected was that the margin methods have excellent performance on ordering metrics such as ROC area and average precision. We introduce a new metric, SAR, that combines squared error, accuracy, and ROC area into one metric. MDS and correlation analysis shows that SAR is centrally located and correlates well with other metrics, suggesting that it is a good general purpose metric to use when more specific criteria are not known.

References

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2004 DataMininginMetricSpaceAnEmpiri	Rich Caruana Alexandru Niculescu-Mizil			Data Mining in Metric Space: An Empirical Analysis of Supervised Learning Performance Criteria				10.1145/1014052.1014063		2004