Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entry

Fabisch, Alexander; Laux, Melvin; De Lucas Álvarez, Mariela; Caroselli, Edoardo; Theis, Julian

Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entry

Alexander Fabisch¹, Melvin Laux¹, Mariela De Lucas Álvarez¹, Edoardo Caroselli², Julian Theis²

¹Robotics Innovation Center, German Research Center for Artificial Intelligence (DFKI GmbH), Germany
²Airbus Defence and Space GmbH, Germany
Reinforcement Learning Conference (RLC) 2026

Paper (RLJ) arXiv Code (rl-blox)

Diagram of the hypersonic re-entry vehicle showing lift, drag, velocity, and gravitational force vectors, the body-fixed axes, and the angle of attack, sideslip angle, and bank angle.

We control the attitude of a lifting-body vehicle during hypersonic re-entry, tracking the commanded angle of attack α and bank angle μ while keeping the sideslip angle β small, using deep reinforcement learning combined with an industry-standard PID controller.

Abstract

Deep reinforcement learning has the potential to solve attitude control problems more adaptively, precisely, and robustly by handling nonlinear dynamics, uncertainties, and failure cases more effectively than traditional attitude control approaches. We explore reinforcement learning (RL) for attitude control in spacecraft re-entry. An industry-standard proportional–integral–derivative controller with gain scheduling serves as a strong baseline for model-free RL and hybrid controllers that combine these two approaches. We formalize the application in the RL framework to apply continuous, off-policy RL. State-of-the-art RL achieves comparable performance to traditional control approaches in this domain. However, its out-of-distribution generalization is not sufficient. Hence, we use dynamics randomization to introduce challenging task variations during training and enforce generalization in a predefined operational envelope. Finally, we assess the best obtained RL-based controllers with application-specific metrics to show superior performance in comparison to traditional controllers in the operational envelope, that is, hybrid controllers are able to track the angle of attack better and are more robust under variations of mass, inertia tensor, and flap actuator bandwidth.

Contributions

State-of-the-art deep RL for hypersonic attitude control. We apply state-of-the-art deep reinforcement learning algorithms for continuous control to the problem of attitude control during hypersonic re-entry of a spacecraft. This is a challenging tracking problem that requires gain scheduling for changing atmospheric conditions. We design a simple and effective reward function for attitude control and find that MR.Q excels without task-specific tuning.
Hybrid control architectures. Although the best learned controller performs better than the industry-standard baseline controller under nominal conditions, we explore hybrid control architectures that combine the baseline controller with reinforcement learning and compare these against pure reinforcement learning to enhance out-of-distribution generalization. The hybrid design is inherently safer: the PID controller acts as a verified baseline whose authority can bound the RL agent.
Robustness via dynamics randomization and task scheduling. To improve robustness and generalization explicitly, we compare dynamics randomization and task scheduling approaches for reinforcement learning in this attitude control problem and determine the best training strategy. We find that the resulting policies considerably improve robustness in comparison to the baseline controller.

Key Insights

MR.Q generalizes to every variation seen during training (e.g., exploring actions).
It does not generalize out-of-distribution (OOD).
OOD generalization requires hybrid control: policy + domain-specific controller.

Results Under Nominal Conditions

We first train and evaluate controllers on the nominal re-entry trajectory, where the vehicle dynamics are fixed. State-of-the-art reinforcement learning matches and surpasses the industry-standard baseline controller. We also test how these nominally trained controllers generalize out-of-distribution to variations in the inertia tensor and flap actuator bandwidth that they never saw during training.

Learning curves comparing RL algorithms against the baseline controller

Learning curves: MR.Q reaches and surpasses the performance of the industry-standard baseline controller.

Probability of improvement of RL controllers over the baseline

Probability of improvement over the baseline controller, computed with rliable.

Learning curves comparing pure RL, additive hybrid, and gain-scheduling hybrid architectures

Comparison of control architectures: pure RL, additive hybrid, and gain-scheduling hybrid control.

Angle of attack tracking over a re-entry trajectory.

Sideslip angle tracking over a re-entry trajectory.

Bank angle tracking over a re-entry trajectory.

Out-of-distribution performance under inertia tensor variations, never seen during nominal training.

Out-of-distribution performance under initial attitude variations

Out-of-distribution performance under variations of the initial attitude, never seen during nominal training.

Robustness under flap actuator bandwidth variations

Out-of-distribution performance under variations of the flap actuator bandwidth.

Results Under Domain Randomization

To improve robustness explicitly, we train with dynamics randomization, exposing the agent to a wide range of dynamics parameters (mass, inertia tensor, flap actuator bandwidth, and initial attitude) during training. The resulting controllers generalize across the operational envelope and considerably improve robustness over the baseline. We also compare uniform dynamics randomization against task-scheduling strategies.

Learning curves under dynamics randomization

Learning curves with dynamics randomization (MR.Q). Lines show median performance over test contexts, shaded areas the full range of returns.

Evaluation of the best controllers under domain randomization with domain-specific metrics

Evaluation of the best controllers over 100 test contexts: success rate and error percentiles of the angle of attack, sideslip, and bank angle. Both additive hybrid and pure RL controllers improve robustness over the baseline.

BibTeX

@article{fabisch2026deeprl,
  title   = {Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entry},
  author  = {Fabisch, Alexander and Laux, Melvin and De Lucas {\'A}lvarez, Mariela and Caroselli, Edoardo and Theis, Julian},
  journal = {Reinforcement Learning Journal},
  year    = {2026}
}