Confidence Score

A Confidence Score is a ordinal value associated to a predicted value that is intended to distinguish between strong predictions over weak predictions.

• AKA: Confidence Level, Prediction Likelihood Value, Prediction Confidence Estimate.
• Context:
  • It can be produced by a Confidence Estimation Function (produced by a Confidence Estimation).
  • It can range from being a Coarse Confidence Score (such as a strong prediction score) to being a Fine-Grained Confidence Score.
  • It can range from being a Classification Confidence Score to being a Rank Prediction Confidence Score being an Estimation Confidence Score.
  • It can range from being an Overconfident Confidence Level to being an Underconfident Confidence Level.
  • …
• Example:
  • low likelihood
  • 0.87 expected precision.
  • In a Nearest Neighbor Algorithm-based Prediction the distance measure can be used as a confidence score, in that this value often results in the desired ranking.
  • …
• Counter-Example(s):
  • an Estimated Probability Value.
  • a Log Likelihood Value.
  • a Confidence Interval.
• See: Statistical Significance Level, Chunking Task, Frequent Event, Infrequent Event.

---

---

References

2008

• (Braun et al., 2008) ⇒ P. Braun, et al. (2008). “An Experimentally Derived Confidence Score for Binary Protein-Protein Interactions.” In: Nat. Methods 6, 91–97.

2004

• (Culotta & McCallum, 2004) ⇒ Aron Culotta, and Andrew McCallum. (2004). “Confidence Estimation for Information Extraction.” In: Proceedings of HLT-NAACL (NAACL 2004).

1997

• (Susuki, 1997) ⇒ Einoshin Suzuki. (1997). “Autonomous Discovery of Reliable Exception Rules.” In: Proceedings of KDD Conference (KDD 1997)
  • QUOTE: we propose a novel approach in which exception rules are discovered according to their confidence level based on the normal approximations of the multinomial distributions. This approach can be called as autonomous, since an exception rule is discovered using neither users’ confidence evaluation nor domain knowledge.

---