Solo project · Educational 2D simulation
Constrained Rocket Ascent
I built a rocket simulator to explore reinforcement learning for flight control. After PPO struggled to learn consistently, I trained a neural controller using scripted demonstrations and corrections to its own mistakes.
The goal
Reach at least 100 km while keeping dynamic pressure, a measure of aerodynamic loading, below 70 kPa and acceleration below 8.5 g. I wanted to understand how a learned controller handles a task with both a destination and operating limits.
My approach
I built the simulator, training environment, and flight-evaluation tools. The controller used 20 observations, including altitude, velocity, fuel remaining, pressure, and orientation, to command throttle and steering.
I started with PPO, which learns through trial and error. I adjusted its rewards and introduced progressively harder altitude targets.
What I tried
PPO sometimes reached 100 km, but repeated training runs stalled below the target or lost progress. Run 36’s final evaluation recorded seven successful flights out of 100.
I tested scripted controllers to check what the vehicle could achieve. Increasing propellant from 20 to 25 tonnes let those controllers reach roughly 130 km within the limits, providing successful flights for the network to learn from.
What worked
My first imitation model reached only about 13 km on average. It could copy actions from the demonstrations, but its own mistakes led it into unfamiliar situations.
I collected those situations, labeled them with the scripted controller’s actions, and retrained the network. This corrective training, called DAgger, helped it complete the mission.
Starting from a working imitation policy also made PPO useful for fine-tuning. However, longer training could erode its safety margins, and stress testing exposed pressure and acceleration violations.
Those failures guided the next training round. I collected 300 failed or low-margin flights from the PPO-tuned controller and labeled the states it visited with the scripted teacher’s actions. I combined those corrections with the original training data and trained a fresh imitation model.
The results
Each learned controller faced 35 scenarios, with 200 flights per scenario. Tests varied wind, mass, drag, propulsion, sensor noise, and control delay.
| Controller | Mission success | Within pressure limit | Within g-force limit |
|---|---|---|---|
| Imitation + PPO fine-tune | 84.1% | 98.8% | 98.1% |
| Imitation + stress-test DAgger | 87.9% | 100% | 100% |
The final model completed more missions and recorded no limit violations across the suite. Most remaining failures involved reduced engine efficiency or combined adverse conditions: the rocket stayed within its limits but fell short of 100 km.
The simulator supplies minimum throttle during liftoff and while vertical velocity remains low below 8 km, plus attitude stabilization below 25 km. Pressure and acceleration limits are measured rather than automatically enforced.
Next Steps
My hypothesis is that PPO struggled partly because its inputs left out useful information. It could see how fast the rocket was rotating, but not the direction, and vehicle properties changed between flights without being directly provided to the controller. I’m testing whether adding those details helps it learn more consistently.