System Predictability Measure

A System Predictability Measure is a system measure that quantifies how well a system's state can be predicted.

• AKA: Predictability.
• Context:
  • It can involve a variety of techniques, such as statistical models, information theory, or machine learning algorithms.
  • It can be applied to various domains, including weather forecasting, financial markets, or complex systems analysis.
  • It can form the basis for predicting future states of a system using current and historical data.
  • It can inform optimization strategies by providing insights into the expected performance of different approaches.
  • ...
• Example(s):
  • A mean square error of a forecast model.
  • Predictive Power, a predictability measure based on information-theoretical principles.
  • The theil index, a statistic used to measure the accuracy of time series forecasts.
  • The perplexity of a probabilistic model, used in information theory and machine learning to measure its predictive performance.
  • …
• Counter-Example(s):
  • Randomness Measure, which assesses the lack of predictability in a data set.
  • Chaotic System Measure, which quantifies the unpredictability of chaotic systems.
• See: Prediction Model, Predictive Modelling Task, Saturation Value, Surprise Function.

---

---

References

2020

• (Wikipedia, 2020) ⇒ https://en.wikipedia.org/wiki/Predictability Retrieved:2020-11-29.
  • Predictability is the degree to which a correct prediction or forecast of a system's state can be made either qualitatively or quantitatively.

2004a

• (DelSole, 2004) ⇒ Timothy DelSole (2004). ["Predictability and Information Theory"]. In: Part I: Measures of predictability. Journal of the atmospheric sciences, 61(20), 2425-2440. DOI: 10.1175/1520-0469(2004)061%3C2425:PAITPI%3E2.0.CO;2.
  • QUOTE: Predictability is the study of the extent to which events can be predicted. Perhaps the most utilized measure of predictability is the mean square error of a (perfect) forecast model. Typically, mean square error increases with lead time and asymptotically approaches a finite value, called the saturation value. The saturation value is comparable to the mean square difference between two randomly chosen fields from the system. Consequently, all predictability is said to be lost when the errors are comparable to the saturation value since the forecast offers no better a prediction than a randomly chosen field from the system. Despite its widespread use, mean square error turns out to be a limited measure of predictability. First, the most complete description of a forecast is its probability distribution. Mean square error is merely a moment of a joint distribution and hence conveys much less information than a distribution. Second, as Schneider and Griffies (1999) pointed out, mean square error depends on the basis set in which the data is represented. Unfortunately, no agreement exists on how to combine different variables to obtain a single measure of predictability — for example, how many degrees Celsius of temperature is equivalent to one meter per second of wind velocity. Third, the question of how to attribute predictability to specific structures in the initial condition has no natural answer in this framework. Finally, the question of how to quantify predictability uniquely with imperfect models is not addressed in this framework since different models produce different characteristic errors.

2004b

• (Yao et al., 2004) ⇒ Weiguang Yao, Christopher Essex, Pei Yu, and Matt Davison (2004). "Measure of Predictability". Physical Review E, 69(6), 066121. DOI:10.1103/physreve.69.066121.
  • QUOTE: Our predictability problem is: Given a series of data, how difficult it is to predict the next point? Different techniques may be used for prediction. Frequently used techniques include neural networks (...) wavelets (...), return maps (...) and nonlincar dynamical forecasting (...). The performance of these techniques may differ depending upon the given data. The task of studying the predictability problem is to show the general difficulty of predictions.

    The predictability problem is often investigated by measuring the spatial complexity of the data. One begins by calculating the probability of finding a point in a specific neighborhood. Denoting by $p_i$ the probability of finding the point in neighborhood $i$, one may define a surprise function:

    $surprise = ln \;p_i$

    which tells how much information is obtained by receiving $p_i$.

2001

• (Diebold & Kilian, 2001) ⇒ (2001). "Measuring Predictability: Theory and Macroeconomic Applications". In: Journal of Applied Econometrics, 16(6), 657-669.
  • QUOTE: We propose a measure of predictability based on the ratio of the expected loss of a short-run forecast to the expected loss of a long-run forecast. This predictability measure can be tailored to the forecast horizons of interest, and it allows for general loss functions, univariate or multivariate information sets, and covariance stationary or difference stationary processes. We propose a simple estimator, and we suggest resampling methods for inference. We then provide several macroeconomic applications.

1999

• (Schneider & Griffies, 1999) ⇒ Tapio Schneider, and Stephen M. Griffies (1999). "A Conceptual Framework for Predictability Studies". In: Journal of climate, 12(10), 3133-3155. DOI: 10.1175/1520-0442(1999)012%3C3133:ACFFPS%3E2.0.CO;2.
  • QUOTE: A conceptual framework is presented for a unified treatment of issues arising in a variety of predictability studies. The predictive power (PP), a predictability measure based on information–theoretical principles, lies at the center of this framework. The PP is invariant under linear coordinate transformations and applies to multivariate predictions irrespective of assumptions about the probability distribution of prediction errors. For univariate Gaussian predictions, the PP reduces to conventional predictability measures that are based upon the ratio of the rms error of a model prediction over the rms error of the climatological mean prediction.

---