2011 LeakageinDataMiningFormulationD

• (Kaufman et al., 2011) ⇒ Shachar Kaufman, Saharon Rosset, and Claudia Perlich. (2011). “Leakage in Data Mining: Formulation, Detection, and Avoidance.” In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2011) Journal. ISBN:978-1-4503-0813-7 doi:10.1145/2020408.2020496

Subject Headings: Information Leakage.

Cited By

• http://scholar.google.com/scholar?q=%222011%22+Leakage+in+Data+Mining%3A+Formulation%2C+Detection%2C+and+Avoidance
• http://dl.acm.org/citation.cfm?id=2020408.2020496&preflayout=flat#citedby

Quotes

Author Keywords

• Algorithms; classifier design and evaluation; data mining; data mining; leakage; predictive modeling; statistical inference; theory

Abstract

Deemed "one of the top ten data mining mistakes ", leakage is essentially the introduction of information about the data mining target, which should not be legitimately available to mine from. In addition to our own industry experience with real-life projects, controversies around several major public data mining competitions held recently such as the INFORMS 2010 Data Mining Challenge and the IJCNN 2011 Social Network Challenge are evidence that this issue is as relevant today as it has ever been. While acknowledging the importance and prevalence of leakage in both synthetic competitions and real-life data mining projects, existing literature has largely left this idea unexplored. What little has been said turns out not to be broad enough to cover more complex cases of leakage, such as those where the classical i.i.d. assumption is violated, that have been recently documented. In our new approach, these cases and others are explained by explicitly defining modeling goals and analyzing the broader framework of the data mining problem. The resulting definition enables us to derive general methodology for dealing with the issue. We show that it is possible to avoid leakage with a simple specific approach to data management followed by what we call a learn-predict separation, and present several ways of detecting leakage when the modeler has no control over how the data have been collected.

References

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2011 LeakageinDataMiningFormulationD	Saharon Rosset Claudia Perlich Shachar Kaufman			Leakage in Data Mining: Formulation, Detection, and Avoidance				10.1145/2020408.2020496		2011