Measuring a forecasting system looks easy: less error against actuals, done. Until you have to define what an error is and how to measure it, and many ways of doing it show up. Here we sort out the error metrics, why they matter and when to use each one.
“How accurate is your system?”, “Our WAPE is 15%, is that good?”, “I can’t get past 60% accuracy on my forecasts, what am I doing wrong?”
Questions like these come up in almost every first technical conversation we have, and like almost everything in engineering, the honest answer is “it depends”: a number like that, on its own, cannot be judged.
The same 15% error can be an excellent result in a category with irregular demand and a disaster in a stable one, where repeating last week’s sales would have given you 8% for free.
Before talking about the ways to measure a system’s performance, it is worth remembering that this is a game of comparisons: accuracy is always a relative quantity. You have to compare it against a baseline model, something very simple like a rule or an average, and see how much the metric you chose improves on it.
Without these three things beside it, the number describes nothing:
- At what level it was measured: the performance of a forecast by SKU and store is not comparable with that of the total by month. More volume usually brings more signal (more information to predict with), so we should demand better performance from more aggregated levels.
- At what horizon: next week is not three months out. Uncertainty and errors tend to grow as the prediction reaches further into the future.
- Against what baseline: if it cannot beat the naive (the one that says the forecast is last period’s value), there is little point in using the model. There are tougher baselines, and which one fits depends on what the series looks like: we go through them right below.
The shortcut for a meeting
When somebody hands you an accuracy number, ask for these three things before forming an opinion. If they don’t have them, the number does not say much yet.
What to compare against: the usual baselines
A baseline is a forecast so simple that it needs no training: a rule anyone can build in a spreadsheet. To work as a yardstick it has to be the best of the simple methods that suit that series, and which one that is depends on what the demand looks like:
| If the series… | Baseline | What it forecasts |
|---|---|---|
| Is stable, with no seasonality | Naive | The last period’s value |
| Is stable but noisy | Moving average or median | The average (or median) of the last k periods |
| Is seasonal | Seasonal naive | The value at the same point of the previous cycle: the same week last year, or the same weekday last week with daily data |
| Has a trend | Naive with drift | The last value plus the average slope of the history |
| Is intermittent (many zeros) | Croston or SBA | The average size of a sale divided by the average interval between sales |
| Is new, with little history | Average of analogues | What similar products (same category, similar price) sold in their first weeks |
Three things keep the comparison honest. The baseline is evaluated exactly like the model: same level, same horizon, same cut-off dates and same metric. If you don’t know which one suits a series best, run several and use the best as the yardstick: a model that beats the naive but loses to the seasonal naive adds nothing yet. And there is one baseline that is not in the table and is usually the most useful, what the planning team does today, which we come back to further down when we get to forecast value added.
The families of metrics
There are many error metrics, and here we go through the most common ones. It helps to sort them into families by what they measure, because within a family the differences are detail and between families they are substance.
In the formulas, \(y_t\) is actual sales in period \(t\), \(\hat{y}_t\) the forecast for that period and \(n\) the number of periods evaluated.
Scale-dependent
MAE (mean absolute error) and RMSE (root mean squared error) live in the product’s units. If your MAE is 40, you are off by 40 units per period.
The difference between the two matters more than it looks: MAE is minimised by the median of the distribution and RMSE by the mean. Because RMSE squares the error, one badly forecast week weighs far more than five slightly-off weeks, so it pushes the model to cover the peaks. MAE treats every error the same and, when the series has many zeros, the forecast that minimises it can be zero.
That last part matters in practice. If you measure intermittent demand with MAE, the winning model can be the one that says “you will sell nothing”, which is literally the worst possible model to buy with.
Both share a problem: they cannot be compared across products. Forty units is nothing for a SKU selling 5,000 and a catastrophe for one selling 60.
Percentage
MAPE divides each error by actual sales and averages the percentages. It solves comparability and brings two problems that make it dangerous.
First, if actual sales in a period are zero the division blows up, so in practice it is computed only over periods with sales, which leaves out exactly the zeros you care about most. And it does not take a zero to break it: forecasting 10 when 1 sold gives 900% error.
Second, and more importantly, it is asymmetric, and it leans the dangerous way. Forecast double what sold and the error is 100%. Forecast half and it is 50%. Falling short has a ceiling (since a forecast cannot go below zero, the error cannot exceed 100%) and overshooting has none.
A model trained or tuned against MAPE learns that under-forecasting is safer, which in an operation is exactly the error you do not want to reward.
sMAPE tries to fix that by dividing by the average of actual and forecast, which bounds it between 0 and 200%. The problem is that it introduces its own asymmetry, in the opposite direction: because the denominator includes the forecast, a forecast below the actual shrinks the denominator and inflates the error. At the same distance from the actual, sMAPE punishes falling short more than overshooting, and that can push a model to over-forecast. It is also undefined when actual and forecast are both zero, which in zero-heavy series happens all the time.
MAAPE is a lesser-known alternative: it takes the arctangent of the percentage error, so no single period can contribute more than a ceiling and one huge error stops dominating the average. With zero sales and a positive forecast it gives that ceiling; when both are zero you have to decide what it is worth (the usual choice is zero error). It is more stable than MAPE on intermittent demand, though it does not solve all of its problems, and it is barely used outside papers.
Weighted
WAPE sums all absolute errors and divides by total sales. It is a volume-weighted average: every unit counts the same, so a high-volume SKU weighs more than a slow one. That is why it fits an operation best, where being wrong on the big product costs real money, and why it does not break on zeros.
Scaled
MASE divides your error by the error of a naive forecast on the same data. A MASE of 1.0 means “same as repeating the previous period”, 0.7 means “30% better than that”, and anything above 1.0 means the model is getting in the way. RMSSE is the same with squared errors.
The denominator is the naive’s error over the training history, of length \(T\); with \(m\) the seasonal period (52 for weekly data with yearly seasonality) it is the seasonal naive, and with \(m = 1\) the plain one.
They are the ones that best tolerate averaging across different series, because each one already comes divided by the difficulty of its own series. If you want a single number for a whole catalogue, it comes from here. In the M5, the forecasting competition on Walmart’s sales, the headline metric was precisely a revenue-weighted RMSSE (results).
Three caveats worth knowing. MASE inherits its baseline: use a plain naive on a seasonal series and the denominator inflates, making your MASE look artificially good, so seasonal series need the seasonal naive. It is not averaged with a simple mean, because each series brings its own scale; the median or a volume-weighted average is better. And since the numerator is MAE, it inherits the median problem: on intermittent demand, forecasting zero can score a decent MASE.
Directional
BIAS (or mean error, ME) tells you which way you are wrong, rather than by how much. A model can have a spectacular WAPE and a bias that is filling your warehouse every month.
With a trap that costs whole meetings: there are two sign conventions and both are in use. The classic literature defines mean error as actual minus forecast, so a positive ME means you under-forecast. Many tools, ours included, compute it the other way round (forecast minus actual), so positive means you overshot. Before reading a bias number, ask which one it is. The sign on its own tells you nothing, and the two get mixed up even in the documentation of evaluation libraries half the industry uses.
Two extensions worth more than bias on its own:
CFE (cumulative forecast error) is the sum of signed errors over the horizon, without averaging. It is useful in inventory because it maps straight onto the lead time: summed over the periods an order covers, it tells you how many units you overshot or fell short by, which is what ends up as excess stock or a shortage.
Tracking signal is CFE divided by mean absolute deviation. It works as an alarm: while it stays within roughly plus or minus four, the drift is consistent with chance; once it leaves that range there is systematic bias and the model needs a review. It is one of the few metrics you can wire straight into an alert threshold without inventing a criterion of your own.
Point and probabilistic
All of the above compare one number to another number. If your system delivers quantiles or intervals, none of them evaluates those: you can have the best WAPE in the industry and intervals that never hold.
To evaluate a distribution there is one concept that organises everything else: a proper scoring rule is one that is minimised exactly when the distribution you report is the true one. It sounds abstract and it is the most practical part of the subject: under a proper rule, telling the truth about your uncertainty is the winning strategy, and under an improper one it pays to lie. If the metric rewards inflating the intervals, somebody will inflate them, and it will not be out of bad faith.
Quantile (pinball) loss. It measures how good a quantile is at its own game, with an asymmetric penalty: for the 90th quantile, falling short costs nine times what overshooting does. That is what makes a model trained on the 90th quantile learn to cover the tail instead of hitting the centre. At the 50th the two slopes even out and the loss is half the absolute error, so choosing with it or with MAE gives the same winner. With \(\hat{y}^{q}_t\) the forecast quantile \(q\):
CRPS. Quantile loss integrated over every quantile, which makes it the natural generalisation of MAE to distributions. It has a useful property: if the prediction is a number rather than a distribution, CRPS reduces exactly to MAE, so it lets you compare a probabilistic model against a point one on the same scale. Its scaled version, divided by a naive the way MASE is, can be averaged across series.
With \(F\) the forecast distribution:
Coverage and calibration. The easiest to explain and the one almost nobody looks at: if your 90% interval is well calibrated, actual sales should land inside it about 90% of the time. If it happens 99% of the time, your intervals are inflated and you are over-buying while looking prudent. If it happens 70% of the time, you have stockouts the model said would not happen.
For an interval \([L_t, U_t]\):
Winkler. Combines interval width with the penalty for falling outside, because coverage on its own can be gamed: an interval from zero to infinity covers 100%.
For a \((1-\alpha)\) interval:
The practical reason this matters: the replenishment decision is a question about the tail of the distribution, not the centre. We cover it in probabilistic forecasting.
Choosing the metric is choosing what you ask the model for
This is the least intuitive part of the subject, and the one that changes results the most.
If you use a metric to select the model, it decides which model wins, and different models behave differently downstream. Choosing with RMSE rewards the one that gets the centre right; choosing with quantile loss at 90% rewards the one that does not fall short on the peaks, which is a question much closer to the one a buyer asks.
Before you change the selection metric
Look at which models won before and which win after: that is where the real effect of the change shows, more than in the metric’s own number.
When the metrics disagree
So far we have talked as if you had to pick one metric. In practice three or four matter at once, and the model that wins on one loses on another: the one that captures peaks best usually has worse bias, the one that deviates least on average usually flattens the events.
There is an orderly way through it, and it starts with a question that needs no weights at all: is any model worse than another on everything? If model B has worse WAPE, worse bias and worse quantile loss than model A, no criterion saves it. Those go immediately, without argument.
What is left is a set where each model is better than the others at something. Choosing inside it is a business decision, and it is worth making explicitly. There are three ways.
As a constraint. “The best WAPE among those with bias under 5%.” This is the one to use when a metric is a requirement: if 10% bias breaks your warehouse, no WAPE compensates for it.
As a weighted average. Normalise the metrics, give them weights and add them up. It is the most common approach and it has a large trap: the normalisation decides the result. Two metrics in different units, one with a longer tail than the other, and the one with more variance dominates the sum without anyone having chosen that.
As a single loss that already contains the trade-off. If what matters is that falling short costs more than overshooting, quantile loss at whatever service level you use says exactly that. It is the best of the three when the trade-off can be expressed in costs, because the number that grades it comes from the business.
The aggregation level changes the number: what to expect at each level
A forecast always has three coordinates: what is forecast, at what level, and at what distance. Change any of the three and the metric changes, and aggregating further up is the easiest way to report a pretty number.
Same model, same data, same week:
| Level | Typical WAPE |
|---|---|
| SKU by store by day | high |
| SKU by store by week | considerably lower |
| SKU by region by month | lower still |
| Company total by month | very low |
There is no magic here: errors of opposite sign cancel each other out as you aggregate. An 8% WAPE at company level is perfectly compatible with chaos at the SKU-store level, which is the level you buy at.
The same goes for the horizon. Forecasting next week is far easier than forecasting twelve weeks out, and the number that matters is the one at the horizon where you make the decision. If your lead time is eight weeks, measuring at one week tells you nothing about the quality of your purchasing.
There is a third axis people forget: how you average across series. There are two ways and they give different numbers. The micro average pools every error and every unit sold in the catalogue and only then divides, so big products weigh more, which is what you want when the subject is money. The macro average computes each series’ error and averages the results, so a SKU selling three units a month weighs the same as the one carrying the category.
The simple mean of a thousand MAPEs is the second one, and it is dominated by the small products, which are the ones with enormous percentages. If the report does not say which of the two it is, it cannot be read.
When many series have to be summarised into one number, the median of the MASEs or the geometric mean of the error ratios hold up far better against extremes than any arithmetic average.
The error that matters when you buy for several weeks
If you place one order that covers the next four weeks, what you need to get right is how much will sell across those four weeks in total, and week-by-week metrics do not always tell you that.
A made-up example to see it: actual demand is 100 units a week for four weeks. Model A forecasts 80, 120, 80 and 120. Model B forecasts 110 every week.
| Metric | Model A | Model B |
|---|---|---|
| Weekly MAE | 20 | 10 |
| Weekly WAPE | 20% | 10% |
| Error on the four-week total | 0 | 40 (+10%) |
Week by week, B has half of A’s error. On the total, A gets all 400 units right and B makes you buy 40 too many. Which one serves you depends on how you buy: if you place a single order for the whole period, the total is what counts; if you replenish weekly, or a stockout in week 1 is not made up for by the surplus in week 2 (a lost sale stays lost), the split between weeks matters too, and there A leaves you short every other week.
The missing metric is simple: add up actual demand over the horizon, add up the forecast, and take the absolute difference. Compute it per SKU and per forecast origin, and only then aggregate it, with a weighting scheme you state out loud (by volume, by value). Taking the absolute value before aggregating is what stops one SKU’s surplus from hiding another’s shortage, the same problem as the micro average above, now across products. It is a relative of CFE: both look at what accumulates instead of each period’s error.
Means add up, quantiles do not
With probabilistic forecasts it gets more interesting. If you want expected demand over the four weeks, you can add up the weekly means: the expectation of a sum is the sum of the expectations, always, whatever happens between weeks.
Quantiles do not work that way. The P90 of total demand depends on how the weeks move together, and adding up the weekly P90s gives you the P90 of the total only in an extreme case: every week rising and falling together, perfectly. When the weeks are roughly independent, a high week tends to be offset by a low one, and the sum of the P90s lands above the P90 of the total.
To put a number on it, with another made-up example (normal demand, independent across weeks): if each week has mean 100 and standard deviation 20, the weekly P90 is about 126 units and four of them add up to 503. The four-week total has mean 400 and standard deviation 40 (variances add, standard deviations do not), so its P90 is about 451. Adding quantiles would have you buy roughly 50 extra units, at a real service level close to 99.5% when you thought you were aiming for 90%. If demand across weeks is strongly correlated, because of a promotion or a trend, the gap shrinks, and only the joint distribution tells you by how much.
The better route is to forecast cumulative lead-time demand directly, or to simulate coherent week-by-week demand paths and take the quantile of their totals. The tools from above still apply for evaluating it: quantile loss on the total and coverage of the total’s intervals.
Measuring the decision as well as the forecast
In inventory, falling short and overshooting almost never cost the same. A missing unit is a lost sale (and sometimes a lost customer); a surplus unit is holding cost, or a markdown at the end of the season.
The simplest case, the newsvendor (one purchase, no second chance), has an exact answer: the optimal quantity is the demand quantile given by dividing the cost of falling short by the sum of both costs. If a lost sale costs you 30 in margin and a leftover unit costs you 10, the ratio is 30 / (30 + 10) = 0.75, and what you should buy is the P75 of demand, above both the mean and the median. It is the same asymmetry as quantile loss, now with the business’s two prices.
With \(c_u\) the cost of a missing unit and \(c_o\) the cost of a surplus one:
Point metrics are still the best diagnostic of where the model fails, but they cannot tell you whether the purchase got better. For that you have to simulate the whole replenishment policy on the past, with lead times, review cycle, minimum order quantities and the stock that was on hand, and measure what comes out of it: stockouts, service level, average inventory and cost. Two models with similar WAPE can produce very different results in that simulation, and it is the only comparison a buyer understands without translation.
Error and bias are different things
This is the distinction that moves the most money and gets looked at least.
Picture two models on the same SKU. The first is off by 20 units a week, sometimes high and sometimes low. The second is off by 15 units a week, always high. The second has the better WAPE, and it is worse: it is piling up 15 units a week of stock nobody asked for, and that accumulation shows up in no error metric at all.
Bias is the first thing to look at, before anything else. Systematic bias almost always has an identifiable cause: a holiday flagged on the wrong day, a promotion that never reached the model, a series censored by stockouts, a mix shift the model has not seen yet. Fix it, and the error drops on its own.
A game of relatives
The absolute number is almost never the question. What you want to know is whether the model adds anything, and that is always answered against something else.
Baselines are free and they are mandatory: repeat the previous period, repeat the same period last year, a moving average. If your model cannot beat all three, what you have is an invoice. MASE makes that comparison explicit, which is why it is the best metric to report upward: a number that already has the baseline inside it cannot be misread.
There is a version of this idea that goes one step further and is worth gold in supply chain: forecast value added. You divide the model’s error by the error of the process it competes against and express the improvement as a percentage. The interesting part is what it compares against: the baseline does not have to be a statistical naive, it can be what the planning team does today. The three-month average the planner builds in their spreadsheet is a perfectly valid baseline, and the question “how much better is this than what we already do?” is the only one that settles whether changing anything is worth it.
Two warnings come with the idea:
- A weak baseline inflates the value added. Choosing it is part of the result, so it gets stated alongside the number.
- Ratios are not averaged arithmetically. Use the geometric mean of the ratios, which the extremes do not distort.
The second relative is against the floor of the series. Some series nobody can forecast well, and that is a property of the demand. A SKU that sells in unpredictable bursts has a high irreducible error, and measuring it with the same yardstick as a stable one produces two false conclusions: that the intermittent one’s model is bad and the stable one’s is good.
So it pays to classify the catalogue before measuring it. The classic classification (Syntetos, Boylan and Croston) separates series by how often they sell and how variable the quantity is when they do. For the ones in the irregular quadrant, chasing period-by-period point accuracy is wasted time: there is no signal to recover. There the useful question changes from “how far off was I” to “how often did I run out” and “what did the error cost me”, which is service level, fill rate and cost-based metrics that turn error into money using the two prices you already know: what a lost sale is worth and what an idle unit costs. How to measure how much signal a series has, its forecastability, and what to ask of it accordingly, is the subject of an upcoming post.
For the easy series the opposite applies: demand more. If a stable, seasonal series does not comfortably beat the seasonal naive, the problem is the model.
Which metric for which decision
| If you want to… | Look at |
|---|---|
| Report the quality of the plan | WAPE at the decision level, plus bias |
| Compare across SKUs or categories | MASE or RMSSE |
| Know whether the model adds anything | MASE against the seasonal naive |
| Pick the winning model | Quantile loss, or RMSE if you want the centre |
| Decide how much to buy | Quantile loss and interval coverage |
| Evaluate an order covering several weeks | Error on the horizon total, and the total’s quantile |
| Know whether purchasing got better | Policy simulation: stockouts, service, inventory, cost |
| Detect that something broke | Tracking signal, and the error trend over time |
| Justify changing the process | Forecast value added against what is done today |
| Evaluate an intermittent series | Service level, fill rate, cost |
| Choose with several metrics at once | Drop the dominated, then decide |
How to measure: rolling-origin backtests and a holdout
To measure a model you have to make it forecast something it has not seen, and with time series the honest way is to simulate the past: pick a cut-off date, train only on what existed up to that day, forecast the following weeks and compare against what happened. Then move the cut-off forward and repeat. That is a rolling-origin backtest, and each cut is an evaluation window (Nixtla explains it here with code, and Hyndman here in more depth).
You use several cuts because a model that wins on one and loses on the other three got lucky once. Besides the average, look at how the metric moves from one cut to the next.
The holdout is the last stretch of history, and it stays untouched while you choose: not for training, nor for picking the model, the metric or the parameters. The backtests are for choosing; the holdout measures, once and at the end, how well your choice performs. If you look at it to decide anything, it stops being a holdout and the number you report comes out optimistic.
Two more things to watch. Each cut trains only on what was known that day: a holiday entered later or a sales correction that leaks into training makes the backtest promise something production never delivers. And each window scores at the decision’s horizon: if your lead time is eight weeks, the window measures at eight weeks.
An alternative: combining instead of choosing
Everything above assumes that in the end you have to keep a single model. The alternative is not to choose, and to build an ensemble: a forecast that combines those of several models.
The simplest ensemble averages the forecasts, or takes their median, and it usually beats keeping the one that scored best in the backtest. It is not magic: the winner of a run wins partly on merit and partly on luck, and the luck does not repeat next month. When you combine, the errors each model makes on its own tend to cancel out, and the variance the selection itself introduces goes away too. It is one of the oldest and most replicated results in the field: Bates and Granger published it in 1969 and the forecasting competitions keep rediscovering it every decade.
There are more elaborate variants: weighting each model by how it did in the backtest, averaging only the best ones, or combining differently by type of series. Each one opens its own questions: with what weights (equal, performance-based, re-estimated how often), whether to combine per SKU or per behaviour group, and above all what happens to the quantiles, because averaging distributions is not averaging numbers. And bias does not fix itself: if all your models are wrong in the same direction, so is the average.
How to build a good ensemble is material for another post, and it is on the list.
The formulas, one by one
For reference. Notation: \(y_t\) is actual sales in period \(t\), \(\hat{y}_t\) the forecast, \(n\) the number of periods evaluated, \(H\) the horizon, \(T\) the length of the training history and \(m\) the seasonal period (52 for weekly data with yearly seasonality). With several series, the sums also run over the series.
Scale-dependent
MAE, mean absolute error. Minimised by the median.
MSE, mean squared error. Minimised by the mean; its units are squared, so it is better for training than for reporting.
RMSE, the square root of MSE: back in the product’s units, still weighing large errors heavily.
RMSLE, RMSE on the logarithm. It measures relative errors, punishes falling short more than overshooting and only works for non-negative values.
Percentage
MAPE. Undefined when any \(y_t\) is zero.
sMAPE. There are several versions; this is the most common, bounded between 0 and 200%.
You will read both claims about its asymmetry, and both are true depending on what is held fixed. With actual sales fixed, a forecast 20 units low scores a larger error than one 20 units high, which is what we described above. With the forecast fixed it goes the other way, which is why some documentation says it punishes overshooting more.
MAAPE. The arctangent bounds each term between 0 and \(\pi/2\): zero sales with a positive forecast add \(\pi/2\), and the case where both are zero needs a convention.
Weighted
WAPE. Minimised by the median, and only undefined if total sales are zero.
Scaled
MASE. The numerator is measured over the evaluated horizon and the denominator is the seasonal naive’s error over the training history. Undefined if the series is constant in training.
RMSSE. The same with squared errors; minimised by the mean.
Directional
Bias or mean error, with the convention we use (forecast minus actual, so positive means overshooting). The classic literature uses the opposite sign.
CFE, cumulative forecast error, with the same sign convention.
Tracking signal, CFE divided by mean absolute deviation. Outside roughly plus or minus four, there is systematic bias.
Horizon
Error on the horizon total, per SKU and per forecast origin, before aggregating.
Means add up, quantiles do not. For each week’s demand \(Y_t\):
Probabilistic
Quantile (pinball) loss for quantile \(q\), with \(\hat{y}^{q}_t\) the forecast quantile. At \(q = 0.5\) it is half the absolute error.
WQL, weighted quantile loss: the sum over the evaluated quantiles, divided by total sales. AutoGluon multiplies \(\rho_q\) by 2 so that, with only the 0.5 quantile, WQL equals WAPE exactly.
SQL, scaled quantile loss: the same divided by the MASE denominator instead of by sales. With only the 0.5 quantile, it is MASE.
CRPS, with \(F\) the forecast distribution. It is quantile loss integrated over every quantile; if \(F\) is a single number, it gives the absolute error.
Coverage of an interval \([L_t, U_t]\): the share of times sales landed inside. For a 90% interval it should come out near 0.9.
Winkler for a \((1-\alpha)\) interval: the width, plus a penalty for every unit sales landed outside it.
Decision
Forecast value added against a reference process, with \(E\) the same error metric for both.
Newsvendor quantile, with \(c_u\) the cost of a missing unit and \(c_o\) the cost of a surplus one.
Further reading
- AutoGluon, Forecasting Metrics (v1.1.0). Each metric’s formula as the library implements it, including WQL and SQL.
- Nixtla, Evaluation Metrics. When each metric fits and when it does not, with CRPS for probabilistic forecasts.