SUMMARY-Error metrics
Error metrics for the forecast of a time serie. Measuring the accuracy and magnitude of errors of the forecast.
We have an example of data points / observations with the forecast.
Mean Absolute Error (MAE)
MAE measures the average size of the error between predicted and actual values - taking the absolute difference between the predicted and actual values and averaging.
MAE = ( ∑
absolute( predictedyi - actualyi ) )
/ n
- Advantages:
simplicity, easy to interpret and less sensitive to outliers.
- Disadvantages:
does not distinguish between overestimation and underestimation
does not provide information about the direction or magnitude of individual errors
step 1
sum of the absolute difference between the predicted and actual values
= |(20) - (20)| + |(20) - (-5)| + |(24) - (28)| + |(27) - (29)| + |(29) - (28)| + |(28) - (31)| + |(30) - (34)| + |(33) - (35)| + |(35) - (36)| + |(36) - (41)| + |(40) - (45)|
= 0 + 25 + 4 + 2 + 1 + 3 + 4 + 2 + 1 + 5 + 5
= 52
step 2
then averaging with n = 11
MAE = 52 / 11 ~ 4.7
Mean Squared Error (MSE)
The average squared difference between the predicted and actual values. Used metric for evaluating the accuracy of forecasting models.
MSE = ( ∑
(predictedyi - actualyi)2 )
/ n
- Advantages:
Simplicity, capture large and small errors and penalizes large errors.
- Disadvantages:
emphasizes large errors
does not differentiate between overestimation and underestimation
step 1
sum of squared difference between the predicted and actual values
= ((20) - (20))
2 + ((20) - (-5))
2 + ((24) - (28))
2 + ((27) - (29))
2 + ((29) - (28))
2 + ((28) - (31))
2 + ((30) - (34))
2 + ((33) - (35))
2 + ((35) - (36))
2 + ((36) - (41))
2 + ((40) - (45))
2
= (0)
2 + (25)
2 + (-4)
2 + (-2)
2 + (1)
2 + (-3)
2 + (-4)
2 + (-2)
2 + (-1)
2 + (-5)
2 + (-5)
2
= 0 + 625 + 16 + 4 + 1 + 9 + 16 + 4 + 1 + 25 + 25
= 726
step 2
then averaging with n = 11
MSE = 726 / 11 = 66
Root Mean Squared Error (RMSE)
The square root of the MSE.
RMSE = √MSE
RMSE = √(( ∑
(predictedyi - actualyi)2 )
/ n )
- Advantages:
Penalizes large errors.
- Disadvantages:
sensitive to outliers
doesn't provide insights into the direction or pattern of forecast errors
doesn't distinguish between over- and under-predictions
RMSE = √MSE
RMSE = √66 (see above)
RMSE ~ 8.1
Mean Absolute Percentage Error (MAPE)
The average percentage difference between the predicted values and the actual values - the absolute difference between the predicted and actual values, dividing it by the actual value, and then averaging the results.
MAPE = (( ∑
((predictedyi - actualyi) / actualyi) )
/ n ) x 100
- Advantages:
errors in terms of percentage - easier to compare with different series.
- Disadvantages:
sensitive to extreme values
difficulty with zero or near-zero values
is not a symmetric metric- the direction of the error affects MAPE
step 1
sum of the difference between the predicted and actual values divided by actual values
∑
((
predictedy
i -
actualy
i) /
actualy
i)
=
(((20) - (20)) / (20)) +
(((20) - (-5)) / (-5)) +
(((24) - (28)) / (28)) +
(((27) - (29)) / (29)) +
(((29) - (28)) / (28)) +
(((28) - (31)) / (31)) +
(((30) - (34)) / (34)) +
(((33) - (35)) / (35)) +
(((35) - (36)) / (36)) +
(((36) - (41)) / (41)) +
(((40) - (45)) / (45))
= (0.000) + (-5.000) + ~(-0.143) + ~(-0.069) + ~(0.036) + ~(-0.097) + ~(-0.118) + ~(-0.057) + ~(-0.028) + ~(-0.122) + ~(-0.111)
= ~(-5.667)
step 2
Divide this result divided by n = 11, then multiply by 100
MAPE = ((-5.667) / 11) = ~ (-51.5)
Symmetric Mean Absolute Percentage Error (SMAPE)
Measures the percentage difference between the observed and predicted values. The absolute difference between the actual and forecasted values, dividing it by the average of the absolute values of the actual and forecasted values, and then multiplying by 100 to express it as a percentage.
SMAPE = (( ∑
( absolute(predictedyi - actualyi) / ( absolute(predictedyi) + absolute(actualyi))) )
/ n ) x 100
- Advantages:
handle zero values and its symmetry (overestimation and underestimation errors are treated equally)
- Disadvantages:
sensitive to extreme values
difficulty with zero or near-zero values for actual and predicted values
step 1
∑
( absolute(
predictedy
i -
actualy
i) / ( absolute(
predictedy
i) + absolute(
actualy
i)) )
=
( |((20) - (20))| / (|(20)| + |(20)|) ) +
( |((20) - (-5))| / (|(20)| + |(-5)|) ) +
( |((24) - (28))| / (|(24)| + |(28)|) ) +
( |((27) - (29))| / (|(27)| + |(29)|) ) +
( |((29) - (28))| / (|(29)| + |(28)|) ) +
( |((28) - (31))| / (|(28)| + |(31)|) ) +
( |((30) - (34))| / (|(30)| + |(34)|) ) +
( |((33) - (35))| / (|(33)| + |(35)|) ) +
( |((35) - (36))| / (|(35)| + |(36)|) ) +
( |((36) - (41))| / (|(36)| + |(41)|) ) +
( |((40) - (45))| / (|(40)| + |(45)|) )
=
( 0 / 40 ) +
( 25 / 25 )+
( 4 / 52 ) +
( 2 / 56 ) +
( 1 / 57 ) +
( 3 / 59 ) +
( 4 / 64 ) +
( 2 / 65 ) +
( 1 / 71 ) +
( 5 / 77 ) +
( 5 / 85 )
0 + 1 + ~0.077 + ~0.036 + ~0.018 + ~0.051 + ~0.063 + ~0.031 + ~0.014 + ~0.065 + ~0.065
= ~1.42
step 2
Divide this result divided by n = 11, then multiply by 100
SMAPE = (1.42 / 11) x 100 = ~12.9
Mean Absolute Scaled Error (MASE)
measures the relative forecast accuracy compared to the naïve or benchmark model - dividing the mean absolute error (MAE) of the model’s forecasts by the MAE .
MASE = MAE / ((∑absolute(predictedyi - actualyi-1) ) / (n-1))
i start with 2
- Advantages:
is scale-independent
is robust against outliers
- Disadvantages:
sensitive to zero or near-zero values in the denominator
assumes the naïve model is accurate (not always true)
step 1
∑absolute(
predictedy
i -
actualy
i-1)
=
|((20) - (-5))| +
|((20) - (28))| +
|((24) - (29))| +
|((27) - (28))| +
|((29) - (31))| +
|((28) - (34))| +
|((30) - (35))| +
|((33) - (36))| +
|((35) - (41))| +
|((36) - (45))|
= 70
step 2
Divide MAE by (result -of step 1- divided by (n - 1)), where n = 11
MASE = MAE / (result / (n -1)) = ~( 4.7 / ( 70/ 10)) = ~0.67