Measures the quality of central prediction intervals, scaling the interval (Winkler) score by the in-sample mean absolute error of the naive (or seasonal naive) forecast. The interval score rewards narrow intervals and penalizes observations falling outside them, and the scaling makes the measure comparable across series of different magnitudes. This is the prediction-interval metric used in the M4 competition. Smaller scores indicate better calibrated and narrower intervals.
Details
For a central interval at level 1 - alpha with lower and upper bounds \(l_i\) and \(u_i\)
(the alpha/2 and 1 - alpha/2 quantiles):
$$
\mathrm{MSIS} = \frac{\frac{1}{n} \sum_{i=1}^n (u_i - l_i)
+ \frac{2}{\alpha}(l_i - y_i)\mathbf{1}\{y_i < l_i\}
+ \frac{2}{\alpha}(y_i - u_i)\mathbf{1}\{y_i > u_i\}}
{\frac{1}{T-m} \sum_{t=m+1}^T \lvert z_t - z_{t-m} \rvert}
$$
where \(z\) is the training series, \(m\) is the seasonal period, and \(T\) is the length of
the training series. For keyed tasks the score is computed per series and averaged.
Dictionary
This mlr3::Measure can be instantiated via the dictionary mlr3::mlr_measures or with the associated sugar function mlr3::msr():
Meta Information
Task type: “regr”
Range: \([0, \infty)\)
Minimize: TRUE
Average: macro
Required Prediction: “quantiles”
Required Packages: mlr3, mlr3forecast
References
Gneiting, Tilmann, Raftery, E A (2007). “Strictly Proper Scoring Rules, Prediction, and Estimation.” Journal of the American Statistical Association, 102(477), 359–378.
Makridakis, Spyros, Spiliotis, Evangelos, Assimakopoulos, Vassilios (2020). “The M4 Competition: 100,000 time series and 61 forecasting methods.” International Journal of Forecasting, 36(1), 54–74.
See also
Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html#sec-eval
Package mlr3measures for the scoring functions.
as.data.table(mlr_measures)for a table of available Measures in the running session (depending on the loaded packages).Extension packages for additional task types:
mlr3proba for probabilistic supervised regression and survival analysis.
mlr3cluster for unsupervised clustering.
Other Measure:
mlr_measures_fcst.acf1,
mlr_measures_fcst.coverage,
mlr_measures_fcst.mase,
mlr_measures_fcst.mda,
mlr_measures_fcst.mdpv,
mlr_measures_fcst.mdv,
mlr_measures_fcst.mpe,
mlr_measures_fcst.pinball,
mlr_measures_fcst.rmsse,
mlr_measures_fcst.wape,
mlr_measures_fcst.winkler
Super classes
mlr3::Measure -> mlr3::MeasureRegr -> MeasureMSIS