Information Retrieval (IR) Performance Measure

A Information Retrieval (IR) Performance Measure is a information processing task measure for an IR task.

• Context:
  • It can (often) be based on a Ranking Task Performance Measure, such as MRR and NDCG.
  • It can (often) balance Information Searcher Return to Information Searcher Investment (see ROI).
  • It can range from being an Online IR Performance Measure (such as conversion rate) to being an Offline IR Performance Measure (such as MRR).
• Example(s):
  • Clickthrough Rate (on search results).
  • Mean Reciprocal Rank (MRR), NDCG, ...
  • …
• Counter-Example(s):
  • a Recommendations Task Performance Measure.
• See: Classification Performance Measure, Symmetric Difference, Standard Boolean Model, Ground Truth, Ill-Posed, Relevance (Information Retrieval).

---

---

References

2020

• (Wikipedia, 2020) ⇒ https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval) Retrieved:2020-9-18.
  • Evaluation measures for an information retrieval system are used to assess how well the search results satisfied the user's query intent. Such metrics are often split into kinds: online metrics look at users' interactions with the search system, while offline metrics measure relevance, in other words how likely each result, or search engine results page (SERP) page as a whole, is to meet the information needs of the user.

2020

• (Wikipedia, 2020) ⇒ https://en.wikipedia.org/wiki/Information_retrieval#Performance_and_correctness_measures Retrieved:2020-9-3.
  • The evaluation of an information retrieval system' is the process of assessing how well a system meets the information needs of its users. In general, measurement considers a collection of documents to be searched and a search query. Traditional evaluation metrics, designed for Boolean retrievalor top-k retrieval, include precision and recall. All measures assume a ground truth notion of relevancy: every document is known to be either relevant or non-relevant to a particular query. In practice, queries may be ill-posed and there may be different shades of relevancy.

2017

• (Wikipedia, 2017) ⇒ https://en.wikipedia.org/wiki/Information_retrieval#Performance_and_correctness_measures Retrieved:2017-6-21.
  • … In practice, queries may be ill-posed and there may be different shades of relevancy.

    Virtually all modern evaluation metrics (e.g., mean average precision, discounted cumulative gain) are designed for ranked retrieval without any explicit rank cutoff, taking into account the relative order of the documents retrieved by the search engines and giving more weight to documents returned at higher ranks. The mathematical symbols used in the formulas below mean:

    • [math]\displaystyle{ X \cap Y }[/math] - Intersection - in this case, specifying the documents in both sets X and Y
    • [math]\displaystyle{ | X | }[/math] - Cardinality - in this case, the number of documents in set X
    • [math]\displaystyle{ \int }[/math] - Integral.
    • [math]\displaystyle{ \sum }[/math] - Summation.
    • [math]\displaystyle{ \Delta }[/math] - Symmetric difference

2017

• (Tunkelang, 2017a) ⇒ Daniel Tunkelang. (2017). “Evaluating Search: Measuring Searcher Behavior."
  • QUOTE: … We can think of the act of performing a search as a single unit of investment. It’s a coarse unit, but nonetheless a useful one that allows us to derive two ROI measures: the click-through rate (CTR) as the fraction of searches resulting in a click, and the conversion rate as the fraction of searches resulting in a conversion. …

    … As George Box said, “all models are wrong, but some are useful.” It’s a good idea to track a few measures of searcher return and searcher effort. In general, you want to increase the former and decrease the latter. …

---