3. Forecast accuracy, bias and forecast value add
Forecast accuracy, bias and forecast value add
Forecast accuracy
This measures how closely your predicted demand matches actual demand. There are several common metrics used to evaluate this:
Mean absolute percentage error (MAPE): MAPE measures the percentage error between forecasted and actual values. It has its limitations, especially with low-demand products, where the percentages can be misleading. It’s been criticized for this over the years. Weighted MAPE is used more often than the traditional MAPE.
Root mean square error (RMSE): RMSE gives more weight to larger errors, making it useful for forecasts that rely on the mean. If your forecast is a bit off consistently, RMSE won't penalize you too much, but if you have a few big misses (large forecast errors), RMSE will show those clearly by giving them more impact in the final metric.
Mean absolute error (MAE): MAE focuses on the size of the errors without highlighting larger deviations. Compared to RMSE, which amplifies larger errors by squaring them, MAE provides a more straightforward view of forecast accuracy by focusing only on the absolute size of each error and averaging them out. This gives you a sense of the typical error without letting any particularly large mistakes dominate the calculation.
Each of these metrics offers a way to measure the quality of your forecast, but picking the right one depends on the model you're using and the type of demand you're dealing with—whether it's stable, intermittent, erratic, or downright unpredictable.
BCG Gamma’s (now BCG X) extensive article on evaluating forecasts, which includes a wide variety of accuracy & bias metrics:
FVA shows how much value your forecasting process brings by comparing it to a baseline or simple benchmark, like a naive forecast. For example, if your FVA is 20%, that means your forecast model is improving accuracy by 20% over a basic benchmark. You can also use FVA to see the added value of sales teams' or planners’ inputs.
Why accuracy matters
Managing uncertainty and buffer creation: Forecast accuracy and demand variability play a big role in inventory management decisions. If your forecast is consistently off, you’ll need to build larger buffers. If you’ve got a good handle on demand, you can create smaller buffers. For example, unpredictable products will need more safety stock than stable ones.
Demand lags: Lags refer to the time between when a forecast is made and when the actual demand happens. Different industries and products have varying lead times, making a specific lag period more or less important depending on the business:
Short-term lags: Common in industries with fast inventory turnover or perishables, like groceries. You need accurate near-term forecasts to ensure freshness.
Mid-term lags: Found in industries like consumer electronics, where forecasts are made months in advance, to balance production and shipping times.
Long-term lags: Typical in industries like automotive or large-scale manufacturing, where production cycles are long, and forecasts are made years in advance. These forecasts depend heavily on market trends and external factors, so there’s more uncertainty.
The impact of demand lag/horizon
Naturally, the longer the lag, the harder it becomes to predict demand accurately. Short-term forecasts tend to be more reliable, while long-term ones are more likely to diverge from actual demand.
Forecasting bias
Forecasting bias happens when there’s a consistent difference between forecasted demand and actual demand, whether it is overestimating or underestimating. Bias isn’t just an error - it’s a systematic tendency to lean one way or the other.
Positive bias: This is when your forecast consistently overestimates demand. The outcome? You’re probably left with excess inventory, higher storage costs, and tied-up capital.
Negative bias: This is when your forecast underestimates demand, leading to stockouts, missed sales, and unhappy customers/salespeople.
How is bias created?
Bias can come from a few sources:
Historical reliance: Some forecasting models depend too much on outdated trends or data, ignoring recent changes.
Inadequate collaboration: If departments like marketing or finance aren’t involved, forecasts can be skewed because key adjustments like promotions or market shifts aren’t taken into account.
To identify bias, companies use metrics like Mean Error (ME) or Mean Forecast Error (MFE), which show whether a forecast consistently over- or underestimates demand.
At what level do you measure accuracy?
It’s easier to forecast overall revenue for next year—you might be off by a few percentage points. But as you drill down, things get trickier. Look at specific countries or product lines, and the variance increases. Now, imagine forecasting at a product level, on a daily basis, for a specific location—the accuracy drops even further. Most companies need forecasts at the lowest level, and that's where accuracy should be measured.
Why forecast at the lowest level?
Most businesses need detailed forecasts because decisions around inventory, procurement, and production rely on granular information. It’s not enough to know you’ll sell 100,000 units over the next year. You need to know how much to produce, where to stock it, and when to make it available. This doesn’t mean you need to generate the forecasts at the lowest level immediately. Aggregation & disaggregation can be used to your advantage and will be explained further on.
Expert resources if you want to delve deeper into forecast accuracy:
A great resource:
Shorter article by Ivan: https://openforecast.org/2024/04/24/best-practice-for-forecasts-evaluation-for-business/
Slimstock’s forecast accuracy guide, which looks at it more from a business perspective