Counterfactual Evaluation Task
A Counterfactual Evaluation Task is an quasi-experiment/out-of-sample evaluation task that ...
- …
- Counter-Example(s):
- See: Quasi-Experiment, Counterfactual Learning, Causal Learning, Prospective Evaluation, Causal Equilibria.
References
2018
- (Wikipedia, 2018) ⇒ https://en.wikipedia.org/wiki/Impact_evaluation#Counterfactual_evaluation_designs Retrieved:2018-6-10.
- Counterfactual analysis enables evaluators to attribute cause and effect between interventions and outcomes. The 'counterfactual' measures what would have happened to beneficiaries in the absence of the intervention, and impact is estimated by comparing counterfactual outcomes to those observed under the intervention. The key challenge in impact evaluation is that the counterfactual cannot be directly observed and must be approximated with reference to a comparison group. There are a range of accepted approaches to determining an appropriate comparison group for counterfactual analysis, using either prospective (ex ante) or retrospective (ex post) evaluation design. Prospective evaluations begin during the design phase of the intervention, involving collection of baseline and end-line data from intervention beneficiaries (the 'treatment group') and non-beneficiaries (the 'comparison group'); they may involve selection of individuals or communities into treatment and comparison groups. Retrospective evaluations are usually conducted after the implementation phase and may exploit existing survey data, although the best evaluations will collect data as close to baseline as possible, to ensure comparability of intervention and comparison groups.
2015
- (Swaminathan & Joachims, 2015) ⇒ Adith Swaminathan, and Thorsten Joachims. (2015). “Counterfactual Risk Minimization: Learning from Logged Bandit Feedback.” In: Proceedings of the International Conference on Machine Learning, pp. 814-823.
- QUOTE: … … 4.3. When is counterfactual learning possible? The bounds in Theorem 1 are with respect to the randomness in h0. Known impossibility results for counterfactual evaluation using h0 (Langford et al., 2008) also apply to counterfactual learning … …
2013
- (Bottou et al., 2013) ⇒ Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. (2013). “Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising.” In: The Journal of Machine Learning Research, 14(1).
- QUOTE: Readers with an interest in real-life applications will find (iv) a selection of practical counterfactual analysis technique s applicable to many real-life machine learning systems. Readers with an interest in computational advertising will find a principled framework that (v) explains how to soundly use machine learning techniques for ad placement, and (vi) conceptually connects machine learning and auction theory in a compelling manner.
The paper is organized as follow]]s. Section 2 gives an overview of the advertisement placement problem which serves as our main example. In particular, we stress some of the difficulties encountered when one approaches such a problem without a principled perspective. Section 3 provides a condensed review of the essential concepts of causal modeling and inference. Section 4 centers on formulating and answering counterfactual question s such as “how would the system have performed during the data collection period if certain interventions had been carried out on the system?” We describe importance sampling method s for counterfactual analysis, with clear conditions of validity and confidence intervals. Section 5 illustrates how the structure of the causal graph reveals opportunities to exploit prior information and vastly improve the confidence intervals. Section 6 describes how counterfactual analysis provides essential signals that can drive learning algorithms. Assume that we have identified interventions that would have caused the system to perform well during the data collection period. Which guarantee can we obtain on the performance of these same interventions in the future? Section 7 presents counterfactual differential techniques for the study of equilibria. Using data collected when the system is at equilibrium, we can estimate how a small intervention displaces the equilibrium. This provides an elegant and effective way to reason about long-term feedback effects. Various appendices complete the main text with information that we think more relevant to readers with specific backgrounds. …
- QUOTE: Readers with an interest in real-life applications will find (iv) a selection of practical counterfactual analysis technique s applicable to many real-life machine learning systems. Readers with an interest in computational advertising will find a principled framework that (v) explains how to soundly use machine learning techniques for ad placement, and (vi) conceptually connects machine learning and auction theory in a compelling manner.
2008
- (Langford et al., 2008) ⇒ John Langford, Alexander Strehl, and Jennifer Wortman. (2008). “Exploration Scavenging.” In: Proceedings of the 25th International Conference on Machine learning (ICML-2008).