Prediction Error Cost-Benefit Matrix

Context:
- It can be an Performance Measure to a Classification Problem (and allow predictive modelers to describe the costs and the benefits associated with each possible prediction).
- It can be used to Optimize Return on Investment.
- It can be associated with Confusion Matrix.
- …
Example(s):
- a Fraud Detection Cost-Benefit Matrix.
- a E-commerce Product Recommendation System Cost-Benefit Matrix.
- a Bank Loan Approval System Cost-Benefit Matrix.
- a Medical Diagnostic System Cost-Benefit Matrix.
- a Contract Review System Cost-Benefit Matrix.
- an Identity Cost-Benefit Matrix (which optimizes for accuracy).
- ...
Counter-Example(s)
- an Intrinsic Performance Measure.
- a F1 Measure.
See: Outlier Detection.

References

ChatGPT
- A Cost-Benefit Matrix is a structured framework used in classification tasks to evaluate the financial and operational implications of correct and incorrect predictions. By detailing the rewards for accurate classifications and penalties for errors, it helps predictive modelers quantify the real-world impact of their models. In various contexts, from fraud detection to medical diagnostics, this matrix serves as a performance measure, guiding optimization towards desired outcomes, often linked to maximizing return on investment. While closely associated with confusion matrices, it distinctly emphasizes tangible outcomes over mere statistical accuracy, offering a more holistic view of a model's effectiveness in a given application.

(Branco et al., 2016) ⇒ Paula Branco, Luís Torgo, and Rita P. Ribeiro. (2016). “A Survey of Predictive Modeling on Imbalanced Domains.” ACM Computing Surveys (CSUR) 49, no. 2
- QUOTE: ... In order to perform a modification on a selected algorithm, it is essential to understand why it fails when the distribution is skewed. Also, some of the adaptations assume that a cost/cost-benefit matrix is known for different error types, which is frequently not the case. ...

https://github.com/podopie/DAT18NYC/blob/master/classes/13-expected_value_cost_benefit_analysis.ipynb
- QUOTE: Objectives
  - Extend our metrics into business application
  - Identify and understand a cost benefit matrix.
  - Using a confusion matrix with a cost benefit matrix to solve for expected value

The technique we'll use is a cost benefit matrix. This is very similar to our confusion matrix:

\begin{bmatrix}TP & FP\\FN & TN\end{bmatrix}

Which to find probabilities turns into this:

\begin{bmatrix}p(TP) & p(FP)\\p(FN) & p(TN)\end{bmatrix}

and our cost benefit matrix will be somewhat similar:

\begin{bmatrix}b(TP) & c(FP)\\c(FN) & b(TN)\end{bmatrix}

where:

[math]\displaystyle{ b }[/math] represents benefit (the benefits of accurately predicting positives and negatives), while...
[math]\displaystyle{ c }[/math] represents cost (the costs of misclassifying positives and negatives).

To simplify we'll associate benefits to be positive value and costs to be negative value:

\begin{bmatrix}v(TP) & -v(FP)\\-v(FN) & v(TN)\end{bmatrix}