Before we jump into the different forecasting methodologies, it's important to first understand the types of demand you’re dealing with. Each one requires a different approach to get accurate results.
Regular demand: Stable and predictable patterns over time, such as daily or weekly sales in retail.
Seasonal demand: Fluctuates based on seasons or specific time periods, like increases during holidays or different seasons.
Intermittent demand: Irregular and sporadic, with frequent periods of no demand, typically common with slow-moving products.
Lumpy demand: High variability and irregular occurrences, often found in specialized B2B markets.
Event-driven demand: Temporary spikes driven by marketing campaigns, promotions, competitor stockouts, interest spikes,...
Each of these types requires a unique approach.
Statistical methods have been the backbone of demand forecasting for years. They work well for regular and seasonal demand but struggle with more complex or irregular patterns.
Time series models, like ARIMA (AutoRegressive Integrated Moving Average), use historical data to make predictions for the short to medium term.
Best fit for: Regular and seasonal demand.
Drawbacks: These models assume that past patterns will continue into the future, which may not hold in rapidly changing markets. They also require manual tuning and don’t handle large datasets well.
Infrastructure: Minimal resources required so you can run them on standard PCs with tools like Excel, Python (using libraries like statsmodels), or R.
Exponential smoothing methods, like Holt-Winters, forecast based on level, trend, and seasonality. They’re simple, effective, and easy to implement for straightforward demand types.
Best fit for: Regular demand with consistent seasonality.
Drawbacks: Not suitable for complex or highly volatile demand types.
Infrastructure: Easily implemented using tools like Excel, Python, or R.
Machine learning (ML) models bring more flexibility to demand forecasting. They handle larger datasets and can uncover complex relationships between demand drivers that traditional statistical models may miss. This makes them well-suited for various demand types.
Random Forest aggregates the results of multiple decision trees to enhance prediction accuracy. It’s useful for forecasting demand by product category or SKU.
Best fit for: Regular, promotional, and lumpy demand, especially when demand drivers (e.g., price, marketing, economic factors) are known.
Drawbacks: Can be difficult to interpret and prone to overfitting if not tuned properly. Not ideal for time series data unless you transform it first.
Infrastructure: Requires moderate computational power; can be implemented using Python (scikit-learn) or R.
GBM (gradient boosting explained) combines multiple weak models to improve accuracy. Popular implementations like XGBoost and LightGBM are highly efficient for handling large datasets.
Best fit for: Promotional and irregular demand, especially when external features like weather or economic indicators influence demand.
Drawbacks: More computationally demanding and requires expertise to tune hyperparameters.
Infrastructure: High computational resources needed - cloud-based solutions like AWS or Google Cloud are recommended for large datasets.
Deep learning models, particularly neural networks, excel at identifying complex patterns in large datasets. They are especially powerful for handling intricate demand patterns, especially when external factors are involved.
RNNs and LSTMs are designed to handle sequential data, making them ideal for time series forecasting. LSTMs, in particular, can capture long-term dependencies and trends.
Best fit for: Intermittent, lumpy, and seasonal demand where past sequences heavily influence future demand.
Drawbacks: Requires large amounts of historical data, high computational power, and expertise to tune properly. They can also overfit if not managed carefully.
Infrastructure: Requires high-performance computing (HPC) or cloud services like Google Cloud AI, AWS Sagemaker, or Microsoft Azure to train models efficiently.
CNNs, usually known for image recognition, have been adapted for time series forecasting. They capture local relationships in the data and can detect complex patterns.
Best fit for: Complex seasonal and lumpy demand where hidden structures in the data influence demand.
Drawbacks: Like RNNs, CNNs are computationally expensive and require large datasets.
Infrastructure: Requires high computational power, often needing GPU-based systems or cloud platforms.
Prophet, developed by Facebook, is a business-friendly time series forecasting framework. It’s particularly good at handling regular and seasonal demand, automatically accounting for trends, seasonality, and holidays. Prophet is easy to use, even for those with minimal statistical expertise, and it handles missing data and outliers well.
Best fit for: Predictable, seasonal demand with consistent trends.
Drawbacks: Prophet struggles with highly irregular or intermittent demand. In these cases, machine learning models like Random Forest or deep learning models like LSTM are more effective.
Demand Type | Statistical (ARIMA, ETS) | Machine Learning (Random Forest, GBM) | Deep Learning (RNN, LSTM) |
Regular | High | Medium | Low |
Seasonal | High | Medium | High |
Intermittent | Low | Medium | High |
Lumpy | Low | High | High |
Promotional | Low | High | High |
While machine learning and deep learning techniques are powerful tools for demand forecasting, they also come with their own set of challenges:
Data requirements: Advanced methods like LSTM and CNN need large datasets for training. If your data is small or sparse, these models may not perform very well.
Overfitting: The complexity of these models can lead to overfitting, meaning the model works great on historical data but struggles to generalize for future predictions.
Interpretability: Deep learning models often operate like "black boxes," making it hard to understand how they’re making decisions. This can be a problem in industries where transparency is critical or and it can hinder user adoption.
Infrastructure cost: Advanced methods require significant computational resources. Cloud infrastructure is often needed to train these models, adding to both cost as well as complexity.
Choosing the right forecasting method depends on your demand type and the infrastructure you have available.
For regular and seasonal demand: Traditional statistical methods like ETS and ARIMA remain effective and cost-efficient.
For lumpy or promotional demand: Machine learning models like Random Forest and Gradient Boosting Machines are better suited since they can handle external factors and more complex patterns. Prophet with adjustments should also be considered.
For intermittent and lumpy demand with complex patterns: Deep learning methods like LSTM and CNN excel here, but they do require large datasets and significant computational power.
Statistical models: These are lightweight and can run on local machines or basic cloud platforms.
Machine learning models: These require moderate infrastructure, such as multi-core processors or cloud-based solutions, especially for larger datasets.
Deep learning models: These need high-performance GPU-based systems or cloud platforms like AWS, Google Cloud, and Microsoft Azure due to their computational demands.
Demand forecasting using FB prophet: https://towardsdatascience.com/demand-forecasting-using-fb-prophet-e3d1444b9dd8
How to use Facebook’s Prophet: https://demand-planning.com/2018/03/14/forecasting-with-facebooks-prophet/ , https://facebook.github.io/prophet/
Machine learning for demand planning by Nicolas Vandeput, video: https://www.youtube.com/watch?v=ujG3-OZGnTk
Forecasting at scale, a paper on creating explainable regression forecasting models: https://peerj.com/preprints/3190/
About statistical forecasting by IBF: https://demand-planning.com/2023/07/10/the-science-of-demand-planning-demystifying-statistical-forecasting/
An analysis of different state-of-the-art models, like LSTM, DeepAR, NLinear, and others and their performance on different types of datasets: https://arxiv.org/abs/2309.15946
Reproduction of the results of this paper in GitHub: https://github.com/MIMUW-RL/Unified-Long-Horizon-Time-Series-Benchmark?tab=readme-ov-file
Monash Forecasting Repository: time-series datasets for training & testing models: https://forecastingdata.org/
Ivan Svetunkov’s resources on forecasting: https://openforecast.org/
An interesting resource on the judgment of forecast practitioners on which models to use: https://onlinelibrary.wiley.com/doi/10.1016/j.jom.2018.05.005
This suggests that expert opinions can already filter out many of the useless models for a specific dataset, thereby reducing computational power needed