Forecast accuracy is about more than algorithms

At N-SIDE, accuracy is considered one of the main pillars of forecasting. It's a topic that often raises questions from systems operators, and for good reason - achieving accurate forecasts can be quite challenging. In order to assist power systems professionals, we've put together a brief guide outlining the key concepts to help them get started on this important subject.

Accuracy is a system property, not just the property of an algorithm.

As a systems operator, it's crucial to prioritize both accuracy and reliability. While a highly complex algorithm that performs well on benchmarks may seem impressive, it's not practical for production if it's difficult to maintain. Ultimately, accuracy should not be achieved at the expense of practicality and reliability.

Accurate for doing what?

At N-SIDE, we believe that having an accurate algorithm is not enough if it solves the wrong problem. Therefore, we always ask ourselves what problem the forecast is intended to solve. For instance, instead of predicting imbalance, we could determine mFRR activation. Similarly, instead of forecasting grid losses and the DA price of electricity, we could predict when to buy energy. Additionally, instead of an HVDC flow forecast, we could forecast ATC (available transfer capacity). In all these cases, we aim to build the business case into the algorithm itself, rather than just minimizing a vanilla Mean Absolute Error. This is made possible by our advanced algorithms.

Three contributors to forecasting accuracy

We distinguish three areas for accuracy: algorithm quality, the data and features of the forecasting system, and the processes used to maintain the system. Let’s consider each in turn:

Algorithm quality

When people think of forecast accuracy, they usually think of the pure science of data science. It involves staying updated with the latest literature, trying out new ideas, and creating a technically sophisticated product based on advanced mathematical concepts and cutting-edge technologies. At N-SIDE, we have a team of skilled applied mathematicians, physicists, and engineers who have created their own algorithms that have proven to be more effective than commercially available ones for the energy applications we've tested. But we also know that an algorithm by itself won’t give you a good forecast.

Data and Features

To provide good features, having top-quality data is crucial. In the European energy industry, some data streams are required to be publicly accessible by law. Additionally, there are numerous commercial data vendors with relevant and high-quality data, such as weather forecasts for renewable energy production and demand. In addition to mining publicly available sources, we subscribe to various data vendors to make sure our algorithms receive the best data. To transform these data into useful features, we must also process them intelligently. For example, identifying and comprehending long weekends from regular time stamps enhances our ability to predict the demand for a particular region.

Forecast system features

At N-SIDE, we have two powerful tools that help us enhance forecast accuracy: transparent forecasting algorithms and probabilistic forecasts. Although these features are typically geared towards making it easier for forecasters to use, they have also given us valuable insight into the reasoning behind our forecasters' predictions, as well as identifying when the model is uncertain. Such insights are critical to improving the model.

Adaptability is another crucial feature for modern energy forecasts. The world, especially the energy world, is changing very fast. In the “good old days”, we could rely on having a 2-year historical data set with which to train our models. Not so today: two years ago is ancient history in Energy, and our forecasting systems must be adaptable enough to cope with that.

Processes

Many processes and best practices ensure that the theoretical performance is realized in production. These are things like:

• Realistic backtesting: verifying the performance on past data. It is very easy to be overly optimistic about forecaster performance during backtesting by inadvertently leaking data, for example.
• Hyper-parameter tuning: To ensure the optimal combination of parameters that prevent overfitting and maximize accuracy, we employ cutting-edge algorithms, but most importantly: experienced machine learning engineers 😉
• Experiment management: Machine learning engineers often try many different things in search of maximum performance. Keeping track of the results of different experiments and the significance of these results is critical to ensuring that the final production model is the best one.
• Co-creation: We have seen many grids and have built tools that are common to different cases. But our clients know their own grids better than we do. Frequent interactions with experts ensure that we build a forecaster that is accurate and actually useful for the end goal that the grid operator has in mind.
• Continuous improvement: Our forecasting team holds monthly performance improvement workshops where each engineer takes a product they have not worked on before and seeks to improve it. By doing so, we allow a diversity of ideas, which has improved our forecasts noticeably.
• Monitoring and alerting: Sometimes performance does decrease, due to a change in world and the drivers affecting the forecast. One example might be the commissioning of a new HVDC line which changes the market coupling between regions, or the unavailability of a data source. Such changes are usually known to us before, but if we miss something, our monitoring and alerting systems are set up to notify as quickly to take the necessary actions

In conclusion, in forecasting, we often find that accuracy is not simply a property of an algorithm, but rather an emergent quality of a well-coordinated system. Striking the right balance between algorithm quality, data and feature selection, and system maintenance processes is crucial to achieving reliable results. This complexity underscores the importance of maintaining a dynamic approach that not only keeps up with the fast-paced energy sector but also anticipates its trajectory. Our understanding of this landscape, combined with a firm commitment to continuous improvement, allows us to consistently deliver practical and reliable forecasts. As we navigate the ever-evolving energy ecosystem, we invite you to join the conversation and explore the future of grid operations.