2013 AGeneralBootstrapPerformanceDia

Subject Headings:

Notes

As datasets become larger, more complex, and more available to diverse groups of analysts, it would be quite useful to be able to automatically and generically assess the quality of estimates, much as we are able to automatically train and evaluate predictive models such as classifiers. However, despite the fundamental importance of estimator quality assessment in data analysis, this task has eluded highly automatic solutions. While the bootstrap provides perhaps the most promising step in this direction, its level of automation is limited by the difficulty of evaluating its finite sample performance and even its asymptotic consistency. Thus, we present here a general diagnostic procedure which directly and automatically evaluates the accuracy of the bootstrap's outputs, determining whether or not the bootstrap is performing satisfactorily when applied to a given dataset and estimator. We show that our proposed diagnostic is effective via an extensive empirical evaluation on a variety of estimators and simulated and real datasets, including a real-world query workload from Conviva, Inc. involving 1.7 TB of data (i.e., approximately 0.5 billion data points).

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2013 AGeneralBootstrapPerformanceDia	Ariel Kleiner Ameet Talwalkar Sameer Agarwal Ion Stoica Michael I. Jordan			A General Bootstrap Performance Diagnostic				10.1145/2487575.2487650		2013