Header

UZH-Logo

Maintenance Infos

Training Efficient Controllers via Analytic Policy Gradient


Wiedemann, Nina; Wuest, Valentin; Loquercio, Antonio; Muller, Matthias; Floreano, Dario; Scaramuzza, Davide (2023). Training Efficient Controllers via Analytic Policy Gradient. In: 2023 IEEE International Conference on Robotics and Automation, ICRA 2023, London, United Kingdom, 29 May 2023 - 2 June 2023. Institute of Electrical and Electronics Engineers, 1349-1356.

Abstract

Control design for robotic systems is complex and often requires solving an optimization to follow a trajectory accurately. Online optimization approaches like Model Predictive Control (MPC) have been shown to achieve great tracking performance, but require high computing power. Conversely, learning-based offline optimization approaches, such as Reinforcement Learning (RL), allow fast and efficient execution on the robot but hardly match the accuracy of MPC in trajectory tracking tasks. In systems with limited compute, such as aerial vehicles, an accurate controller that is efficient at execution time is imperative. We propose an Analytic Policy Gradient (APG) method to tackle this problem. APG exploits the availability of differentiable simulators by training a controller offline with gradient descent on the tracking error. We address training instabilities that frequently occur with APG through curriculum learning and experiment on a widely used controls benchmark, the CartPole, and two common aerial robots, a quadrotor and a fixed-wing drone. Our proposed method outperforms both model-based and model-free RL methods in terms of tracking error. Concurrently, it achieves similar performance to MPC while requiring more than an order of magnitude less computation time. Our work provides insights into the potential of APG as a promising control method for robotics. To facilitate the exploration of APG, we open-source our code and make it available atgithub.com/lis-epfl/apg_trajectory_tracking.

Abstract

Control design for robotic systems is complex and often requires solving an optimization to follow a trajectory accurately. Online optimization approaches like Model Predictive Control (MPC) have been shown to achieve great tracking performance, but require high computing power. Conversely, learning-based offline optimization approaches, such as Reinforcement Learning (RL), allow fast and efficient execution on the robot but hardly match the accuracy of MPC in trajectory tracking tasks. In systems with limited compute, such as aerial vehicles, an accurate controller that is efficient at execution time is imperative. We propose an Analytic Policy Gradient (APG) method to tackle this problem. APG exploits the availability of differentiable simulators by training a controller offline with gradient descent on the tracking error. We address training instabilities that frequently occur with APG through curriculum learning and experiment on a widely used controls benchmark, the CartPole, and two common aerial robots, a quadrotor and a fixed-wing drone. Our proposed method outperforms both model-based and model-free RL methods in terms of tracking error. Concurrently, it achieves similar performance to MPC while requiring more than an order of magnitude less computation time. Our work provides insights into the potential of APG as a promising control method for robotics. To facilitate the exploration of APG, we open-source our code and make it available atgithub.com/lis-epfl/apg_trajectory_tracking.

Statistics

Citations

Altmetrics

Downloads

4 downloads since deposited on 27 Feb 2024
4 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Conference or Workshop Item (Paper), refereed, original work
Communities & Collections:03 Faculty of Economics > Department of Informatics
Dewey Decimal Classification:000 Computer science, knowledge & systems
Scopus Subject Areas:Physical Sciences > Software
Physical Sciences > Control and Systems Engineering
Physical Sciences > Electrical and Electronic Engineering
Physical Sciences > Artificial Intelligence
Scope:Discipline-based scholarship (basic research)
Language:English
Event End Date:2 June 2023
Deposited On:27 Feb 2024 16:18
Last Modified:28 Feb 2024 04:56
Publisher:Institute of Electrical and Electronics Engineers
Series Name:IEEE International Conference on Robotics and Automation. Proceedings
ISSN:1050-4729
ISBN:979-8-3503-2365-8
OA Status:Green
Publisher DOI:https://doi.org/10.1109/ICRA48891.2023.10160581
  • Content: Accepted Version
  • Language: English
  • Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)