Reinforcement learning · Solo project
Constrained Rocket Ascent
Teaching a simulated rocket to reach 100 km without crushing itself on the way up. It started as a reinforcement learning problem and turned into a lesson about when to stop trusting the learning algorithm.
What it is
The task sounds simple and turns out to be mean. Fly a 2D rocket to 100 km, the Kármán line, while keeping dynamic pressure under 70 kPa and acceleration under 8.5 g. Those two limits are the whole difficulty. Plenty of throttle profiles reach 100 km. Very few do it without tearing the vehicle apart on the climb.
I built the simulator, the training setup, and the evaluation harness, then spent most of the project figuring out why my first approach kept failing.
The part where PPO kept losing
My first instinct was to let PPO learn the whole ascent from scratch with reward shaping. It did not work, and it kept not working in the same way. Across more than a dozen runs the policy would settle around 85 to 95 km with clean safety margins but never quite enough vertical speed to cross 100 km. I tried finer curriculum milestones, different entropy schedules, and reworked rewards. Same ceiling every time.
What actually moved the project was a check with no learning in it at all. I wrote a deterministic search that tried scripted and optimized controllers directly, and it hit the same wall. With the original propellant load, the vehicle physically could not reach 100 km under the 70 kPa limit. The problem was never the algorithm. The rocket was energy-limited. Once I gave it 25 tonnes of propellant instead of 20, scripted controllers cleared 130 km with margin to spare, and the question changed from "why won't PPO learn this" to "how do I teach it a safe ascent I already know exists."
Imitation, then a careful hand-off
From there it became an imitation learning problem. Plain behavior cloning from twelve scripted expert flights failed, because the moment the policy drifted off the expert's path it was in states it had never seen and made things worse. DAgger fixed that: roll out the failing policy, collect the states it actually visits, relabel them with the expert, retrain, and repeat. That produced a strong controller.
PPO came back only at the very end, as a short fine-tune with hard guardrails that stopped training the moment safety compliance slipped. I had already watched an earlier open-ended fine-tune fly higher and faster while quietly breaking the q and g limits I cared about, so this time the guardrails were the point.
Results
The promoted policy reaches 100 km on 990 of 1,000 held-out episodes, with dynamic pressure and g-load inside their limits on more than 99% of runs. I also put it through a 35-case stress suite covering wind, sensor noise, actuator lag, propulsion loss, and mass and drag errors. A robustness-tuned version held 100% safety compliance while lifting average stress success to about 88%. A few stress cases still fail, and I ran a separate feasibility check to confirm those are genuinely outside the vehicle's envelope rather than something more training would fix.
What I took away
The biggest lesson was to check whether a problem is solvable before blaming the solver. I burned a lot of runs tuning rewards for a target the vehicle could not reach, and a one-afternoon feasibility script would have told me that on day one.
The second was about trusting your own metrics. A rounding bug in my success check was quietly failing valid 100 km flights, where a fuel fraction came back as 1.0000000000000002 instead of 1.0. Fixing it moved reported success from 73% to 99%. After that I stopped believing a number until I had read exactly how it was computed. Sometimes the model gets better at the value you are optimizing and worse at the thing you actually want, and the only way to catch that is to look.