Confidence Scoring Task

A Confidence Scoring Task is a point estimation task to produce a confidence estimate for a prediction.

• Context:
  • Input: a Prediction.
  • output: a Prediction Confidence Estimate.
    • optional: a Confidence Model.
  • It can be solved by a Confidence Scoring System (that implements a Confidence Scoring algorithm).
• Example(s):
  • Given predictive model [math]\displaystyle{ P(T|O) }[/math], what is the confidence score of predictions for observations [math]\displaystyle{ o_1 }[/math] and [math]\displaystyle{ o_2 }[/math]?
• Counter-Example(s):
  • Given predictive model [math]\displaystyle{ P(T|O) }[/math], what is the confidence that prediction [math]\displaystyle{ p }[/math] has greater than 92% chance of being true?
  • a Hypothesis Testing Task.
• See: Confidence Interval, Confidence Interval.

---

---

References

2004

• (Culotta & McCallum, 2004) ⇒ Aron Culotta, and Andrew McCallum. (2004). “Confidence Estimation for Information Extraction.” In: Proceedings of HLT-NAACL (NAACL 2004).

---