 Research
 Open access
 Published:
Determining costefficient controls of electrical energy storages using dynamic programming
Journal of Mathematics in Industry volume 14, Article number: 2 (2024)
Abstract
Volatile electrical energy prices are a challenge and an opportunity for small and mediumsized companies in energyintensive industries. By using electrical energy storage and/or an adaptation of production processes, companies can significantly profit from timedepending energy prices and reduce their energy costs.
We consider a timediscrete optimal control problem to reach a desired final state of the energy storage at a certain time step. Thereby, the energy input is discrete since only multiples of 100 kWh can be purchased at the EPEX SPOT market. We use available price estimates to minimize the total energy cost by a rounding based dynamic programming approach. With our model nonlinear energy loss functions of the storage can be considered and we obtain a significant speedup compared to the integer (linear) programming formulation.
1 Introduction
The climate targets for Germany to become climateneutral by 2045, which were reinforced in 2021, imply that the ambitions for expanding renewable energy must be further increased. As a result, the share of volatile power generation will rise and energy storage will become increasingly important. In 2020, there were already almost 300 hours with negative electricity prices on the dayahead market [www.epexspot.com]. This results, among other things, from the oversupply by renewable power generation. The reversal of the trend for an increasing number of negative electricity prices can be significantly influenced by energy storage. Energy storage systems can assume different functions in the energy system. In multiuse approaches, it is increasingly being investigated how battery storage can be used in times when it is not required for its primary purpose, such as primary control power generation or selfsupply optimization [1]. One option is basically to trade energy on EPEX Spot and take advantage of the price spread between two points in time. Besides mixedinteger optimization models [2], various (approximate) dynamic programming approaches (see, e.g., [3–7]) are used to determine cost efficient controls of electrical storages and/or grids. Most approaches therefore use approximate dynamic programming [8] in order to avoid the “curses of dimensionality” by approximating the value function in each state. In [9] it is shown that the energy storage problem can be solved in polynomial time in a deterministic setting, while it is an NPcomplete problem if prices and energy production are stochastic.
In this paper we consider a timediscrete optimal control problem of an electrical energy storage device and present a rounding based dynamic programming approach, which considers a discrete state space by rounding the energy level in the storage. This also reduces the computation time significantly in contrast to the solution of the mixedinteger programming problem. The rounding of the state space enables us to optimize the control over longer time periods (up to one year), which is of particular interest for the layout of the storage device within a retrospective analysis.
The calculations shown here are based on the example of electrochemical energy storage. In principle, however, these considerations apply to all forms of energy storage (electrical, electromagnetic, electrochemical, mechanical, thermal and chemical) and the suggested algorithm can be applied analogously.
This paper is organized as follows. In Sect. 2 we present a timediscrete model of an electrical energy storage devise which takes energy loss during charging, withdrawal, and selfdischarge into account. Based on this model we develop a mixedinteger programming (MIP) formulation for the optimal control of the energy storage. Due the limited applicability of the MIP for larger instances we present a rounding based dynamic programming approach in Sect. 3, which efficiently approximates the problem. Numerical tests are presented in Sect. 4, including a runtime comparison of the considered models and a tradeoff analysis of the investments in storage devices. Section 5 concludes the article and gives a brief outlook on future research directions based on our approach.
2 Modeling of an electrical energy storage
The energy market has varying energy prices due to supply and demand reasons. On account of, e.g., solar or wind energy, prices are relative to the weather, i.e., low costs correlate with using more ecopower. Therefore, storing energy instead of always buying exactly the required amount may be economically as well as ecologically reasonable (see, e.g., [10–12]). A comprehensive introduction to dynamical energy prices and their impact on industrial processes is given in [13].
2.1 Linear programming
Regarding the dayahead market, trading is only possible at fixed points in time, hence we define the set of discrete trading dates as \(T:=\{1,\ldots ,m\}\), \(m\in \mathbb{N}\). Let then \(V_{t}\) denote the charge level of an electrical storage at the time of \(t\in T\), which is restricted by lower and upper capacity bounds \(c,C\in \mathbb{R}_{\geq 0}\), i.e., \(c\leq V_{t}\leq C\) for all \(t\in T\). Furthermore, let the initial and a final fill level be given, denoted by \(V_{0} = {V_{\mathrm{init}}}\) and \(V_{m} \geq {V_{\mathrm{final}}}\), respectively (see Fig. 1 for a schematic illustration). The lower and upper bounds on the purchased energy per time step are denoted by l and u (\(0\leq l \leq u\)), i.e., \(l \leq x_{t} \leq u\) for all \(t\in T\). The fill level \(V_{t}\) depends on three quantities, firstly on the purchased energy \(x_{t}\), secondly on the external energy consumption \(Z_{t}\in \mathbb{R}_{\geq 0}\) and lastly on the previous fill level \(V_{t1}\). Regarding the latter, we introduce an energy loss function \({g:\mathbb{R}_{\geq 0}\to \mathbb{R}_{\geq 0}}\), which is often assumed to be linear, e.g., \(g(V) = (1\beta )\, V\) for some given value \(\beta \in (0,1)\).
The efficiency of storing to and withdrawing energy from the storage is modeled by the efficiency factors \(\eta _{\mathrm{in}}, \eta _{\mathrm{out}}\in [0,1]\), which are either assumed to be constant factors, or depending on the amount of stored/withdrawn energy \(\eta _{\mathrm{in}}, \eta _{\mathrm{out}}: \mathbb{R}\to [0,1]\). Consequently, it has to be distinguished whether the consumed energy \(Z_{t}\) is taken from the storage or purchased energy is used directly. Let \(y_{t}\in [0,x_{t}]\) be the amount of energy stored in time step t and \(\zeta _{t} :=Z_{t}  x_{t} + y_{t}\) the amount of energy loaded from the storage. In total, we obtain the fill level \(V_{t}\) by
To illustrate this formula we consider the two extreme cases: If the energy consumption \(Z_{t}\) in time step t is taken completely from the storage (since there is not energy input in this time step, i.e., \(x_{t}=y_{t}=0\)) then the energy level in the storage \(V_{t}=g(V_{t1})  \frac{1}{ \eta _{\mathrm{out}}}\,Z_{t}\) is reduced by \(\frac{1}{ \eta _{\mathrm{out}}}\,Z_{t}\) taking into account the energy loss when withdrawing energy from the storage. On the other hand, if the energy consumption equals the energy input \(Z_{t}=x_{t}\) energy level of the storage is \(V_{t}=g(V_{t1})\) unchanged apart from the timedependent energy loss g.
Forecasting models can provide a prognosis about the energy prices \(p_{t}\), \(t\in T\) for the period T. For a retrospective analysis, however, we can also consider the true prices, e.g., to evaluate the capacity of the energy storage. In the following we will concentrate on the running energy costs and neglect acquisition and other types of fixed costs. Thus, we aim to minimize the total costs of purchased energy \(x = (x_{1},\ldots ,x_{m})\), i.e., \(\min \;\, \sum_{t\in T} p_{t} \, x_{t}\). Together with the aforementioned constraints, we formulate the following linear program (LP):
It is well known that linear optimization problems are efficiently solvable (in polynomial time with interior point methods, cf. [14]). However, the energy is often traded in discrete quantities, which makes the energy input x a discrete variable. For example, at the EPEX SPOT market only multiples of 100 kWh can be purchased. We thus obtain a mixedinteger (linear) programming problem (MIP).
2.2 Integer programming
Since the energy input can only attain discrete values, we modify equation (1l) in (MIP) to equation (2c):
Thereby, \(h_{x}\in \mathbb{R}_{\geq 0}\) denotes the discretization step size of the energy input, i.e., x is restricted to multiples of \(h_{x}\): \(x\in \{0,h_{x},2\,h_{x},\ldots \}\). In contrast to LPs, MIPs are in general NPhard optimization problems, which are solvable by a significant computational effort, e.g., using branch and bound based approaches [15]. Moreover, if the energy loss in the storage depends nonlinearly on the fill level, one would obtain a mixedinteger nonlinear optimization problem, which are computationally even more demanding.
On that account, we introduce our dynamic programming approach in the following chapter.
3 Roundingbased dynamic programming
3.1 Dynamic programming
The central idea of dynamic programming is to break down an optimization problem into a sequence of smaller efficiently solvable subproblems. Thereby, dynamic programming relies on Bellman’s principle of optimality [16], which states that a solution can only be optimal if its intermediate solutions (up to a certain state/time) are optimal w.r.t. the corresponding subproblems. Knapsack problems [17] and shortest path problems are the most prominent examples of optimization problems satisfying Bellman’s principle [18], which does not hold for all optimization problems. The discrete electrical energy storage problem (2a)–(2c), however, satisfies Bellman’s principle, since a control \(x_{t}\), \(y_{t}\), \(t=1,\ldots ,\tau \), of the storage up to an intermediate time step \(\tau \in T\) with fill level of \(V_{\tau}\) can only be extended to an optimal solution if there is no other feasible policy reaching this (or a larger) fill level \(V_{\tau}\) at time τ with lower energy cost. The optimal solution of the overall problem can then be derived from the optimal solutions of these subproblems. Applying Bellman’s recursion [16] we determine the cheapest way to reach every feasible fill level at time step t based on the costs at time step \(t1\). The optimal control \((x^{\ast}_{1},\ldots ,x^{\ast}_{m})\) for an arbitrary final fill level can then be reconstructed by a backtracking procedure. Adapted to the previously introduced electrical storage problem (2a)–(2c), we initialize the recursion for the total energy costs \(z_{t}(d)\) up to time step t to reach a given storage fill level d as
where \(x_{1}(d) = (d  g({V_{\mathrm{init}}}) + \frac{\zeta _{1}}{\eta _{\mathrm{out}}})/\eta _{ \mathrm{in}}\) is the amount of energy required to reach the level of d in the current state. Thereby, we assume that energy is only withdrawn from the storage when it is not necessary to reach the desired storage level d in time step t, i.e., \(\zeta _{t}=\max \{0,\eta _{\mathrm{out}}(g(V_{t1})d)\}\), since it is always preferable to directly consume energy over its lossy storage. Consequently, in each time step there can be only either charging of the storage or withdrawel of energy from the storage.
3.2 Rounding in the state space
Since the computational efficiency of dynamic programming algorithms strongly depends on the size of the state space a straightforward application of the Bellman recursion onto the discrete storage problem would lead to numerical difficulties. Due to the energy loss function g it is very unlikely that different policies end up at the same fill level, so the number of states grows exponentially with increasing number of time steps. In order to limit the number of states, we discretize the state space, i.e., the fill level of the storage, with step size \(h_{V}\in \mathbb{R}_{\geq 0}\). A similar approach is proposed in [19] on continuous control problems.
Let
define the ceil function with respect to some step size h,
the corresponding floor function, \(V\in \mathbb{R}_{\geq 0}\). In the rounding based dynamic programming (RBDP) algorithm (Algorithm 1), we underestimate the fill level of the storage, i.e., we round off in the state space and consider the following adapted recursion formula, by which we obtain the optimal control to approximately reach a given fill level d at time step t based on the controls up to the previous time step \(t1\):
Using this recursion formula we can apply a dynamic programming scheme to compute the optimal control for each energy level of the storage \(d=c,c+h_{V},\ldots , C\) for all time steps \(t=1,\ldots ,m\), and each energy input k, by eliminating dominated states, cf. Algorithm 1. In the following we assume that l and u are multiples of the discretization \(h_{x}\).
By doing so, the rounding error
is bounded from above by \(h_{V}\) for one single time step. In order to estimate the total error of our algorithm, we first consider some fill level
with rounding error \(0 \leq \varepsilon _{t} < h_{V}\), \(t\in T\). If we assume that the energy loss function g is a monotonically increasing, linear function, we can derive an explicit bound on the total rounding error:
where \(g^{k}\) denotes the ktimes iterated function, i.e., \(g^{0}=\mathrm{id}\) and \(g^{k}=g(g^{k1})\) with \(k\in \mathbb{N}\). Then, the total rounding error is given by
By definition of the function g as \(g(V)=(1\beta )\, V\) with \(\beta \in (0,1)\), it holds that
Since the rounding based dynamic programming algorithms is an exact method on the discretized state space, the difference between the costs of a solution of RBDP and the cost of the optimal solution of (2a)–(2c) are at most \(m\, h_{V}\cdot \max_{t}\{p_{t}\}\). The bound in equation (5) can be computed and subtracted from the upper capacity bound C in order to guarantee feasibility of the exact solution. Hence, \(h_{V}\) should be chosen depending on the maximal capacity, i.e., \(\lfloor C  \varepsilon _{\mathrm{tot}} \rfloor _{h_{V}} \gg 0\).
4 Numerical tests
We implemented the proposed rounding based dynamic programming algorithm, which can be easily adjusted to different use cases. In a simple setup, we compare our method against both linear and mixed integer linear programming, w.r.t. the obtained objective function value \(z^{*}\), of the respective approach. We further add some experiments that demonstrate the runtime differences of our method compared to integer programming, as well as a tradeoff analysis that contrasts the computed energy costs with the storage capacity. The DP is implemented in Python 3.6, the LP in MATLAB and solved with Gurobi, and the MILP is implemented in Julia and solved with Cbc. All experiments were performed on an Intel(R) Celeron(R) N4000 CPU with 1.10 GHz and 8 GB main memory.
In our framework there are several parameters to be adjusted. All experiments were performed with a linear energy loss function, however, different types of functions can be applied. Further, we assume a constant energy consumption over the overall time period to obtain interpretable results reflecting the energy costs. According to the nature of the dayahead market, we allow to purchase energy in steps of 100 kWh, whereas the storage is discretized with a step size of 1 kWh, i.e., \(V_{t} \in \mathbb{Z}\) for all \(t\in T\). A finer or coarser discretization increases the runtime or the rounding error, respectively. In all our numerical experiments we use a time discretization of 1 h. The storage capacity and the amount of purchasable energy are lower bounded by 0. The maximal fill level is varied in the following experiments, and we restrict the energy that is stored at one time step to be maximum half of the capacity. The efficiency factors are \(\eta _{\mathrm{in}} = 0.9\) and \(\eta _{\mathrm{out}} = 0.95\), the energy loss factor is \(\beta =0.1\).
In Table 1 we compare the results of our approach against linear and mixedinteger linear programming solutions. Linear programming (LP) achieves the best solutions, since it considers a relaxation of the discrete problem. However, these solutions are not feasible energy inputs from the EPEX Spot market. Our dynamic programming (DP) approach yields only slightly worse solutions compared to the exact optimal solutions of the mixed integer linear programming (MILP) problem (obtained with coinor/Cbc [20]). All solutions are computed with initial and final fill level equal to 100 kWh. A visual example for the method comparison is given in Fig. 2. We observe that all three approaches provide qualitatively similar results.
While the runtime of the rounding based dynamic programming algorithm depends not only on the considered time period, but also on the storage capacity, its discretization level and the purchasable energy, the runtime of the MILP is only little impaired by variations of these parameters. However, our approach has the clear advantage compared to the MILP model that the optimization over longer time periods (months/years) or with finer time discretizations (15 min/1 min) is possible. For fixed bounds regarding the storage size and the purchasable energy, its runtime grows only linearly for an increasing number of time steps, while the runtime of the MILP grows exponentially, see Fig. 3.
We provide a tradeoff analysis (Fig. 4) where we consider the solutions of our algorithm applied to time frames of one month and one year for varying storage capacities going from 0 kWh to 5000 kWh and 1000 kWh, respectively, in steps of 10 kWh. Based on historical data, previous months or even years can be optimized for several storages and their respective investment costs can be viewed relative to the appropriate energy cost savings. This multiobjective perspective allows to investigate the potential of further investments in storage devices, since it shows the gradual cost reduction induced by the increasing capacity of the storage device. In the case that future energy prices are known either exactly or through forecasting models optimizing several days or weeks jointly improves the result (Fig. 5).
In Fig. 6 we illustrate the solution obtained by the rounding based DP retrospectively optimizing the energy costs over one year, compared to the hourly energy prices. Here, we consider a storage with a maximal fill level of 1000 kWh and a constant energy consumption of 200 kWh per hour. We observe that most energy is purchased in the hours before the two energy price peaks. This demonstrates that our algorithm employs the energy storage in order to bridge expensive time periods.
5 Conclusions and outlook
In this paper, we show that rounding based dynamic programming is an efficient optimization approach for the optimal control of energy storage devices in the presence of volatile costs. In comparison to mixedinteger (linear) programming models the runtime of RBDP is linear in the number of timesteps, which allows us to optimize over larger time periods. The solution of RBDP is thereby a good approximation of the global optimum obtained by the solution of the MILP model.
The presented computational experiments use simplified load curves. However, it is possible to integrate more complicated load curves and feeding plants, as well as supply from own renewable energy sources. This could be used, for example, to optimize the energy trading of a mediumsized company with its own photovoltaic system and battery storage. In addition, this could also be used to calculate the optimal dimensioning of an energy storage system before the investment.
Availability of data and materials
The prices of the day ahead market are publicly available. The used data are also available together with the code on github: https://github.com/SUhlemeyer/RBDP.
Abbreviations
 DP:

dynamic programming
 LP:

linear programming
 MILP:

mixedinteger linear programming
 MIP:

mixedinteger programming
 RBDP:

rounding based dynamic programming
References
Zeh A, Müller M, Naumann M, Hesse H, Jossen A, Witzmann R. Fundamentals of using battery energy storage systems to provide primary control reserves in Germany. Batteries. 2016;2(3):29.
Kumtepeli V, Hesse HC, Schimpe M, Tripathi A, Wang Y, Jossen A. Energy arbitrage optimization with battery storage: 3DMILP for electrothermal performance and semiempirical aging models. IEEE Access. 2020;8:204325–41.
Löhndorf N, Minner S. Optimal dayahead trading and storage of renewable energies—an approximate dynamic programming approach. Energy Syst. 2010;1(1):61–77.
Salas DF, Powell WB. Benchmarking a scalable approximate dynamic programming algorithm for stochastic control of gridlevel energy storage. INFORMS J Comput. 2018;30(1):106–23.
Goubko MV, Shushko NI. Approximate dynamic programming decompositions for economic dispatch in microgrids. IFACPapersOnLine. 2022;55(9):262–7. 11th IFAC Symposium on Control of Power and Energy Systems CPES 2022.
Scott WR, Powel WB. Approximate dynamic programming for energy storage with new results on instrumental variables and projected Bellman errors. Princeton: Princeton University; 2022. Online.
Rüther T, Mößle P, Mühlbauer M, Bohlen O, Danzer MA. Iterative dynamic programming—an efficient method for the validation of power flow control strategies. Electr J. 2022;3(4):542–62.
Powell WB. Approximate dynamic programming. John Wiley & Sons, Inc.; 2011.
Halman N, Nannicini G, Orlin J. On the complexity of energy storage problems. Discrete Optim. 2018;28:31–53.
Paulus M, Borggrefe F. The potential of demandside management in energyintensive industries for electricity markets in Germany. Appl Energy. 2011;88(2):432–41. 5th Dubrovnik Conference on Sustainable Development of Energy, Water and Environment Systems.
Meese J, NeuselLange N, Zdrallek M, Völschow A, Neuhaus U. Optimal use of demand response potentials in mediumsized industry using dynamic electricity tariffs. In: 23rd International Conference on Electricity Distribution (CIRED 2015). 2015.
Meese J, Dahlmann B, Zdrallek M, Völschow A. Intraday redispatch – optimal scheduling of industrial processes at dayahead and continuous intraday market. In: International ETG Congress 2017. 2017.
Meese J. Dynamische Stromtarife zur Erschließung von Flexibilität in Industrieunternehmen. Berlin: epubli GmbH; 2018.
Gondzio J. Interior point methods 25 years later. Eur J Oper Res. 2012;218(3):587–601.
Wolsey LA. Integer Programming. Series in discrete mathematics and optimization. New York: WileyInterscience; 1998.
Bellman R. Dynamic programming. Princeton: Princeton University Press; 1957.
Kellerer H, Pferschy U, Pisinger D. Knapsack problems. Berlin: Springer; 2004.
Papadimitriou CH, Steiglitz K. Combinatorial optimization – algorithms and complexity. New York: Dover; 1998.
Luus R. Optimal control by dynamic programming using systematic reduction in grid size. Int J Control. 1990;51(5):995–1013.
Forrest J, Ralphs T, Vigerske S, LouHafer, Kristjansson B, Jpfasano, et al. coinor/Cbc: version 2.9.9; 2018.
Acknowledgements
We acknowledge support from the Open Access Publication Fund of the University of Wuppertal. Further we thank Benedikt Dahlmann for the supervision of the master’s thesis, on which this work is build upon.
Funding
Open Access funding enabled and organized by Projekt DEAL. Open Access Publication Fund of the University of Wuppertal.
Author information
Authors and Affiliations
Contributions
MS and SU wrote the main part of the paper with input from all authors. SU developed and implemented the rounding based dynamic programming algorithm under supervision of MS and performed the experiments. MS implemented the integer programming problem. BU, supervised by MZ, contributed his knowledge of the energy market and helped with writing the manuscript, especially the introduction and conclusion. All authors contributed to and approved the final manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Stiglmayr, M., Uhlemeyer, S., Uhlemeyer, B. et al. Determining costefficient controls of electrical energy storages using dynamic programming. J.Math.Industry 14, 2 (2024). https://doi.org/10.1186/s13362024001401
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s13362024001401