Accuracy Measure

An Accuracy Measure is a correctness measure that quantifies the degree of closeness between observed values and true values in quantitative terms.

• AKA: Accuracy Metric, Accuracy Indicator, Prediction Accuracy Measure.
• Context:
  • output: Accuracy Scores, Error Rates, Confusion Matrices, Accuracy Confidence Intervals.
  • It can typically quantify System Performance through accuracy measurement tasks and accuracy assessment processes.
  • It can typically evaluate Predictive Model Performance through classification accuracy, regression accuracy, and ranking accuracy.
  • It can typically support Model Selection Tasks through accuracy comparisons and cross-validation accuracy.
  • It can typically inform System Improvement Decisions through accuracy gap analysis and error pattern identification.
  • It can typically complement Other Performance Metrics in multi-metric evaluation frameworks.
  • ...
  • It can often serve as Primary Evaluation Criterion in classification tasks and prediction tasks.
  • It can often guide Model Training Processes through accuracy-based optimization and accuracy threshold setting.
  • It can often detect Model Degradation through accuracy drift monitoring and temporal accuracy tracking.
  • It can often validate System Requirements through minimum accuracy thresholds and accuracy specification compliance.
  • It can often enable Statistical Significance Testing through accuracy difference tests and confidence interval overlap.
  • ...
  • It can range from being a Simple Accuracy Measure to being a Complex Accuracy Measure, depending on its calculation complexity.
  • It can range from being a Binary Accuracy Measure to being a Multi-Class Accuracy Measure, depending on its classification scheme.
  • It can range from being a Point-Estimate Accuracy Measure to being a Distributional Accuracy Measure, depending on its statistical representation.
  • It can range from being a Domain-Agnostic Accuracy Measure to being a Domain-Specific Accuracy Measure, depending on its application context.
  • It can range from being a Hard Accuracy Measure to being a Soft Accuracy Measure, depending on its probability handling.
  • ...
  • It can integrate with Evaluation Frameworks for comprehensive performance assessment.
  • It can combine with Cost Functions for weighted accuracy evaluation.
  • It can feed into Model Selection Algorithms for automated optimization.
  • It can support Performance Dashboards for real-time accuracy monitoring.
  • It can enable A/B Testing Frameworks through comparative accuracy analysis.
  • ...
• Example(s):
  • Classification Accuracy Measures, such as:
    • Binary Classification Accuracy measuring two-class prediction correctness.
      • Balanced Accuracy for imbalanced dataset evaluation.
      • Matthews Correlation Coefficient for binary classification quality.
    • Multi-Class Classification Accuracy evaluating multiple category assignment.
      • Macro-Averaged Accuracy for class-balanced evaluation.
      • Micro-Averaged Accuracy for instance-weighted evaluation.
      • Top-K Accuracy for ranking-based classification.
  • Regression Accuracy Measures, such as:
    • Mean Absolute Error (MAE) for absolute deviation measurement.
    • Root Mean Square Error (RMSE) for squared error assessment.
    • R-Squared (R²) for variance explanation measurement.
    • Mean Absolute Percentage Error (MAPE) for relative error assessment.
  • Domain-Specific Accuracy Measures, such as:
    • Medical Accuracy Measures including:
      • Diagnostic Accuracy for disease identification correctness.
      • Screening Test Accuracy for medical screening effectiveness.
      • Clinical Prediction Accuracy for patient outcome prediction.
    • Contract Analysis Accuracy Measures including:
      • Accuracy-Based Contract Issue-Spotting Measure for contract issue detection accuracy.
      • Contract Classification Accuracy for contract type identification.
      • Contract Entity Recognition Accuracy for legal entity extraction.
    • NLP Accuracy Measures including:
      • Word-Level Accuracy for tokenization correctness.
      • Named Entity Recognition Accuracy for entity identification.
      • Part-of-Speech Tagging Accuracy for grammatical classification.
  • Temporal Accuracy Measures, such as:
    • Time-Series Forecast Accuracy for sequential prediction evaluation.
    • Event Detection Accuracy for temporal pattern recognition.
    • Trend Prediction Accuracy for directional forecast assessment.
  • Probabilistic Accuracy Measures, such as:
    • Calibration Accuracy measuring probability reliability.
    • Brier Score for probabilistic prediction quality.
    • Log Loss for probability distribution accuracy.
  • Structured Prediction Accuracys, such as:
    • Sequence Labeling Accuracy for sequential tagging tasks.
    • Graph Prediction Accuracy for network structure prediction.
    • Tree Structure Accuracy for hierarchical prediction tasks.
  • ...
• Counter-Example(s):
  • Precision Measure, which evaluates positive predictive value rather than overall correctness.
  • Recall Measure, which measures sensitivity rather than total accuracy.
  • F1 Score, which balances precision and recall rather than measuring overall correctness.
  • Speed Measure, which evaluates processing time rather than prediction correctness.
  • Coverage Measure, which assesses completeness rather than accuracy.
  • Efficiency Measure, which measures resource utilization rather than outcome correctness.
  • Serendipity Measure, which values unexpected discovery rather than prediction accuracy.
• See: Performance Measure, Correctness Measure, Quantitative Measure, Evaluation Metric, Error Measure, Statistical Measure, Precision-Recall Measure, Accuracy-Based Contract Issue-Spotting Measure, Model Assessment Task.

---

---

References

2022

• (Wikipedia, 2022) ⇒ https://en.wikipedia.org/wiki/Accuracy_and_precision Retrieved:2022-1-22.
  • In a set of measurements, accuracy is closeness of the measurements to a specific value, while precision is the closeness of the measurements to each other.

    Accuracy has two definitions:

    1. More commonly, it is a description of systematic errors, a measure of statistical bias; low accuracy causes a difference between a result and a "true" value. ISO calls this trueness.
    2. Alternatively, ISO defines accuracy as describing a combination of both types of observational error above (random and systematic), so high accuracy requires both high precision and high trueness.
  • Precision is a description of random errors, a measure of statistical variability.

    In simpler terms, given a set of data points from repeated measurements of the same quantity, the set can be said to be accurate if their average is close to the true value of the quantity being measured, while the set can be said to be precise if the values are close to each other. In the first, more common definition of "accuracy" above, the two concepts are independent of each other, so a particular set of data can be said to be either accurate, or precise, or both, or neither.

2013

• (Hulley et al., 2013) ⇒ Stephen B. Hulley, Steven R. Cummings, Warren S. Browner, Deborah G. Grady, and Thomas B. Newman. (2013). “Designing Clinical Research: Fourth Edition.” Wolters Kluwer Health. ISBN: 9781469840543
  • QUOTE: Accuracy: The degree to which a measurement corresponds to its true value. For example, self-reported body weight is a less accurate measurement of actual bodyweight than one made with a calibrated electronic scale.

2010

• (Ge et al., 2010) ⇒ Mouzhi Ge, Carla Delgado-Battenfeld, and Dietmar Jannach. (2010). “Beyond Accuracy: Evaluating Recommender Systems by Coverage and Serendipity.” In: Proceedings of the fourth ACM conference on Recommender systems (RecSys-2010).
  • QUOTE: ... Over the last decade, different recommender systems were developed and used in a variety of domains [1]. The primary goal of recommenders is to provide personalized recommendations so as to improve users’ satisfaction. As more and more recommendation techniques are proposed, researchers and practitioners are facing the problem of how to estimate the value of the recommendations. In previous evaluations, most approaches focused only on the accuracy of the generated predictions based, e.g., on the Mean Absolute Error. However, a few recent works argue that accuracy is not the only metric for evaluating recommender systems and that there are other important aspects we need to focus on in future evaluations [4, 8]. The point that the recommender community should move beyond accuracy metrics to evaluate recommenders was for example made in [8]. There, informal arguments were presented supporting that accurate recommendations may sometimes not be the most useful ones to the users, and that evaluation metrics should (1) take into account other factors which impact recommendation quality such as serendipity and (2) be applied to recommendation lists and not on individual items.

---