Reinforcement Learning with Guarantees

Dr.-Ing. Pavel Osinenko

AutorIn

Dr.-Ing. Pavel Osinenko

Titel

Reinforcement Learning with Guarantees

Zitierfähige Url:

https://nbn-resolving.org/urn:nbn:de:bsz:ch1-qucosa2-968096

Übersetzter Titel (DE)

Verstärkendes Lernen mit Garantien

Datum der Einreichung

05.10.2024

Datum der Verteidigung

14.04.2025

DOI

https://doi.org/10.60687/2025-0069

Abstract (EN)

This dissertation summarizes my scientific work originating from 2015, focusing on reinforcement learning and subjects that surrounded it. Three major research tracks are featured in this dissertation: - Lyapunov-based reinforcement learning with guarantees with its apex approach called CALF ('Critic as Lyapunov Function'). - Predictive reinforcement learning with guarantees, so-called 'stacked reinforcement earning'. - Novel approaches to formal control system analysis, whose tools were widely employed the above two in their evolution. Whereas each track comprises multiple research papers produces throughout the years, for brevity and ease of technical exposition, only CALF is detailed in this dissertation, with only a brief account of stacked reinforcement learning. The reader should consult the referenced works, whose list is given in Section 7.1. The outline of this dissertation is as follows: Section 4 and Section 5 give a brief sketch of history of modern control and, respectively, reinforcement learning. Section 7 summarizes the major contributions and lists the key publications related to the dissertation. Section 8 overviews the contemporary approaches to reinforcement learning guarantees and briefly analyses them. Section 9 presents a detailed account of CALF. Section 10 is dedicated to stacked reinforcement learning. Section 11 presents the studies in stabilization, constructive control and overviews some selected application works. The main text is concluded by a summary and research outlook into future possible directions. The CALF analysis if presented in the appendix. We begin with a short overview of modern automatic control history, with a focus on adaptive and optimal control, and then proceed to a sketch of history of artificial intelligence, emphasizing reinforcement learning and its interplay with automatic control. We believe that precisely the optimal and adaptive control fields feature the most notable overlap with reinforcement learning generally, and reinforcement learning with guarantees in particular.

Freie Schlagwörter (EN)

Reinforcement learning, automatic control, optimal control, stability, safety, guarantees, formal analysis

Klassifikation (DDC)

006.31

006.32

629.8

004.21

004.33

005.1

006.24

Normschlagwörter (GND)

Maschinelles Lernen

Optimierung

Regelungstechnik

Bestärkendes Lernen

Künstliche Intelligenz

GutachterIn

Dr.-Ing. habil. Stefan Streif
Dr.-Ing. habil. Timm Faulwasser
Dr.-Ing. Daniel Görges

Den akademischen Grad verleihende / prüfende Institution

Technische Universität Chemnitz, Chemnitz

Version / Begutachtungsstatus

angenommene Version / Postprint / Autorenversion

URN Qucosa

urn:nbn:de:bsz:ch1-qucosa2-968096

Veröffentlichungsdatum Qucosa

24.04.2025

Dokumenttyp

Habilitation

Sprache des Dokumentes

Englisch

Lizenz / Rechtehinweis

Inhaltsverzeichnis

1 Acknowledgment
2 Glossary
3 Introduction
4 Brief History of Modern Control, with Focus on Adaptive and Optimal Control
4.1 Introduction
4.2 Rise of Optimal Control
4.3 Dynamic Programming
4.4 Emergence of Adaptive Control
4.5 Historical Outlook
5 Brief History of Artificial Intelligence and Reinforcement Learning
5.1 On Physiological Research Roots of Reinforcement Learning
5.2 On More Technical Roots of Reinforcement Learning
6 Conclusion of Historical Overviews
7 Contribution and Related Publications
7.1 Related Publications
8 Related Work
8.1 Introduction
8.2 Popular Baselines
8.3 Reinforcement Learning with Guarantees
8.4 Concluding Analysis of Related Work
8.5 Popular Software
9 Critic as Lyapunov Function
9.1 Technical Description of Problem
9.2 Introduction to CALF
9.3 Lyapunov-like Constraints and Implementation Details
9.4 Transfer of Knowledge
9.5 Formal Analysis
9.6 Simulation Studies with CALF
9.7 Experimental Studies with CALF
9.8 Conclusion from Studies with CALF
10 Stacked Reinforcement Learning
11 Stabilization, Formal Control and Applications
12 Summary and Research Outlook
Appendices
A. Details of Formal Analysis of CALF
A.1 Recalls and Definitions
A.2 Proof of Main Theorem
A.3 On ω-Uniform Convergence Moduli
References
B. Main Publications in Full Text Supporting the Dissertation
B.1
L. Beckenbach, P. Osinenko, and S. Streif.
“A stabilizing reinforcement learning approach for sampled systems with partially unknown models”.
International Journal of Robust and Nonlinear Control (2024)
B.2
P. Osinenko, G. Yaremenko, G. Malaniia, and A. Bolychev.
“An actor-critic framework for online control with environment stability guarantee”.
IEEE Access, Aug. 17, 2023
B.3
P. Osinenko, D. Dobriborsci, G. Yaremenko, and G. Malaniya.
“A generalized stacked reinforcement learning method for sampled systems”.
IEEE Transactions on Automatic Control, Feb. 27, 2023
B.4
D. Dobriborsci, R. Zashchitin, M. Kakanov, W. Aumer, and P. Osinenko.
“Predictive reinforcement learning: map-less navigation method for mobile robot”.
Journal of Intelligent Manufacturing (2023)
B.5
P. Osinenko, G. Yaremenko, and G. Malaniia.
“On stochastic stabilization via non-smooth control Lyapunov functions”.
IEEE Transactions on Automatic Control, 68.8, Oct. 10, 2022
B.6
P. Osinenko and S. Streif.
“On constructive extractability of measurable selectors of set-valued maps”.
IEEE Transactions on Automatic Control, 66.8 (2021)
B.7
P. Schmidt, P. Osinenko, and S. Streif.
“On inf-convolution-based robust practical stabilization under computational uncertainty”.
IEEE Transactions on Automatic Control, 66 (11), Jan. 18, 2021
B.8
P. Osinenko.
“Towards a constructive framework for control theory”.
IEEE Control Systems Letters, 6, Apr. 30, 2021
B.9
P. Osinenko and D. Dobriborsci.
“Effects of sampling and horizon in predictive reinforcement learning”.
IEEE Access, 9, Sept. 13, 2021
B.10
L. Beckenbach, P. Osinenko, and S. Streif.
“A Q-learning predictive control scheme with guaranteed stability”.
European Journal of Control, 56 (2020)
B.11
T. Göhrt, P. Osinenko, and S. Streif.
“Adaptive dynamic programming using Lyapunov function constraints”.
IEEE Control Systems Letters, 3.4 (2019). Presented at CDC
B.12
P. Osinenko, L. Beckenbach, and S. Streif.
“Practical sample-and-hold stabilization of nonlinear systems under approximate optimizers”.
IEEE Control Systems Letters, 2.4 (2018). Presented at CDC
B.13
L. Beckenbach, P. Osinenko, and S. Streif.
“Model predictive control with stage cost shaping inspired by reinforcement learning”.
Conference on Decision and Control (CDC), 2019
B.14
P. Osinenko and G. Yaremenko.
“On stochastic stabilization of sampled systems”.
Conference on Decision and Control (CDC), Dec. 14, 2021

Nutzungshinweise für die digitalen Objekte in Qucosa