2006 AnotherLookAtMeasuresOfForecastAccuracy

From GM-RKB
(Redirected from Hyndman & Koehler, 2006)
Jump to navigation Jump to search

Subject Headings: Forecast accuracy; Forecast evaluation; Forecast error measures; M-competition; Mean absolute scaled error

Notes

Cited By

Quotes

Author Keywords

Forecast accuracy; Forecast evaluation; Forecast error measures; M-competition; Mean absolute scaled error

Abstract

We discuss and compare measures of accuracy of univariate time series forecasts. The methods used in the M-competition as well as the M3-competition, and many of the measures recommended by previous authors on this topic, are found to be degenerate in commonly occurring situations. Instead, we propose that the mean absolute scaled error become the standard measure for comparing forecast accuracy across multiple time series.

2 A critical survey of accuracy measures

Let [math]\displaystyle{ Y_t }[/math] denote the observation at time [math]\displaystyle{ t }[/math] and [math]\displaystyle{ F_t }[/math] denote the forecast of [math]\displaystyle{ Y_t }[/math]. Then define the forecast error [math]\displaystyle{ e_t = Y_t - F_t }[/math]. The forecasts may be computed from a common base time, and be of varying forecast horizons. Thus, we may compute out-of-sample forecasts [math]\displaystyle{ Fn+1, . . . , Fn+m }[/math] based on data from times [math]\displaystyle{ t = 1,...,n }[/math]. Alternatively, the forecasts may be from varying base times, and be of a consistent forecast horizon. That is, we may compute forecasts [math]\displaystyle{ F_{1+h},...,F_{m+h} }[/math] where each [math]\displaystyle{ F_{j+h} }[/math] is based on data from times [math]\displaystyle{ t=1,...,j }[/math]. The in-sample forecasts in the examples above were based on the second scenario with [math]\displaystyle{ h=1 }[/math]. A third scenario arises when we wish to compare the accuracy of methods across many series at a single forecast horizon. Then we compute a single [math]\displaystyle{ F_{n+h} }[/math] based on data from times [math]\displaystyle{ t=1,...,n }[/math] for each of m different series.

We do not distinguish these scenarios in this paper. Rather, we simply look at ways of summarizing forecast accuracy assuming that we have m forecasts and that we observe the data at each forecast period.

We use the notation mean([math]\displaystyle{ x_t }[/math]) to denote the sample mean of [math]\displaystyle{ {x_t} }[/math] over the period of interest (or over the series of interest). Analogously, we use median([math]\displaystyle{ x_t }[/math]) for the sample median and gmean([math]\displaystyle{ x_t }[/math]) for the geometric mean.

References

  • AHLBURG, D.A., CHATFIELD, C., TAYLOR, S.J., THOMPSON, P.A., WINKLER, R.L., MURPHY, A.H., COLLOPY, F., and FILDES, R. (1992) A commentary on error measures. International J. Forecasting, 8, 99–111.
  • ARMSTRONG, J.S. (2001) “Evaluating forecasting methods”, Chapter 14 in Principles of forecasting: a handbook for researchers and practitioners, ed., J.S. Armstrong. Kluwer Academic Publishers: Norwell, MA.
  • ARMSTRONG, J.S., and COLLOPY, F. (1992) Error measures for generalizing about forecasting methods: empirical comparisons. International J Forecasting, 8, 69–80.
  • ASSIMAKOPOULOS, V. and NIKOLOPOULOS, K. (2000) The theta model: a decomposition approach to forecasting. International Journal of Forecasting 16, 521–530.
  • BILLAH, B., KING, M.L., SNYDER, R.D., and KOEHLER, A.B. (2005) Exponential smoothing model selection for forecasting. Working Paper 6/05, Department of Econometrics and Business Statistics, Monash University, Australia.
  • BOWERMAN, B.L., O’CONNELL, R.T., and KOEHLER, A.B. (2004) Forecasting, time series and regression: an applied approach, Thomson Brooks/Cole: Belmont, CA.
  • CHATFIELD, C. (1988) Apples, oranges and mean square error. International J. Forecasting, 4, 515–518.
  • CLEMENTS, M.P., AND HENDRY, D.F. (1993) On the limitations of comparing mean square forecast errors. J. Forecasting, 12, 617–637.
  • COLEMAN, C.D., and SWANSON, D.A. (2004) On MAPE-R as a measure of estimation and forecast accuracy. Working paper. Center for Population Studies, University of Mississippi. Accessed 18 May 2005. http://www.olemiss.edu/depts/population studies/WorkingPapers.html
  • COLLOPY, F., and ARMSTRONG, J.S. (2000) Another error measure for selection of the best forecasting method: the unbiased absolute percentage error. Updated October 2000. Accessed 18 May 2005. http://hops.wharton.upenn.edu/forecast/paperpdf/armstrong-unbiasedAPE.pdf
  • DIEBOLD, F.X. (2001) Elements of forecasting, 2nd ed., South-Western: Cincinnati, Ohio.
  • FILDES, R. (1992) The evaluation of extrapolative forecasting methods. International J. Forecasting, 8, 81–98.
  • GARDNER, E. (1990) Evaluating forecast performance in an inventory control system. Management Science, 36, 490–499.
  • GOODWIN, P., and LAWTON, R. (1999) On the asymmetry of the symmetric MAPE. International Journal of Forecasting, 4, 405–408.
  • GRANGER, C.W.J., and PESARAN, M.H. (2000) Economic and statistical measures of forecast accuracy. J. Forecasting, 19, 537–560.
  • HANKE, J.E., and REITSCH, A.G. (1995) Business forecasting, 5th ed., Prentice-Hall: Englewood Cliffs, NJ.
  • HYNDMAN, R.J., KOEHLER, A.B., SNYDER, R.D., and GROSE, S. (2002) A state space framework for automatic forecasting using exponential smoothing methods. International J. Forecasting, 18(3), 439–454.
  • KOEHLER, A.B. (2001) The asymmetry of the sAPE measure and other comments on the M3- competition. International J. Forecasting, 17, 537–584.
  • LAWRENCE, M., and O’CONNOR, M. (2005) Judgmental forecasting in the presence of loss functions. International J. Forecasting, 21, 3–14.
  • MAKRIDAKIS, S. (1993) Accuracy measures: theoretical and practical concerns. International J. Forecasting, 9, 527–529.
  • MAKRIDAKIS, S., ANDERSON, A., CARBONE, R., FILDES, R., HIBON, M., LEWANDOWSKI, R., NEWTON, J., PARZEN, P., and WINKLER, R. (1982) The accuracy of extrapolation (time series) methods: results of a forecasting competition. J. Forecasting, 1, 111–153.
  • MAKRIDAKIS, S., and HIBON, M. (2000) The M3-competition: results, conclusions and implications. International J. Forecasting, 16, 451–476.
  • MAKRIDAKIS, S., WHEELWRIGHT, S., and HYNDMAN, R.J. (1998) Forecasting: methods and applications, 3rd ed., JohnWiley & Sons: New York.
  • THEIL, H. (1966) Applied economic forecasting. Rand McNally: Chicago, IL.
  • THOMPSON, P.A. (1990) An MSE statistic for comparing forecast accuracy across series. International J. Forecasting, 6 219–227.
  • TSAY, R.S. (2002) Analysis of financial time series, JohnWiley & Sons: New York.
  • SWANSON, D.A., TAYMAN, J., and BARR, C.F. (2000) A note on the measurement of accuracy for subnational demographic estimates. Demography, 2, 193–201.

,

Retrieved from "http://www.gabormelli.com/RKB/index.php?title=2006_AnotherLookAtMeasuresOfForecastAccuracy&oldid=862340"