RL Training Pipeline

The pipeline

Two machines work in parallel. A Mac runs game development, evaluation, and pushes training instructions. A Windows machine with an RTX 3090 trains models on GPU. They coordinate through git—the simplest protocol that works. After one 32-hour silent gap (the trainer had quietly idled before picking up a job), the protocol grew a heartbeat: the trainer commits a pickup acknowledgment with an ETA, then a progress file every three hours. Silence became a diagnosable signal—the next phase's completion was projected to the minute.

The simulation

Training happens in a custom JAX physics simulation that mirrors the C++ game engine. 4,096 arenas run in parallel on a single GPU, each simulating complete combat with Newtonian physics, projectile collision, energy management, and up to 17 combat abilities (14 base + 3 ship-specific).

The simulation must match the game exactly or the trained model won't transfer. 33 cross-check tests verify parity: thrust values, wall bouncing, bullet damage, bomb splash radius, mine proximity triggers, ship-specific stats, ability mechanics. When a value changes in C++, the same change must be made in Python. 189 behavior tests and 86 visual tests verify the UE5 side.

Observation vector (100 floats)

The neural network sees the world as 100 numbers, matching what a human player could perceive. The vector grew from 90 to 96 to 100 as the sim gained mechanics; new features append at the end so older models keep working (they read a truncated prefix).

Action space (14–17 discrete actions, 40–46 logits)

The base "team" preset has 14 actions (40 logits). Ship-specific presets add 1–3 extra binary actions for unique abilities:

Constraints & trade-offs

The simulation makes deliberate simplifications to hit 570K steps/sec. Each trade-off has a cost and a way we verify it doesn't break transfer to the real game.

Reward shaping

The reward function has 9 tunable coefficients, each exposed as a CLI flag. The defaults were calibrated through iteration—the first 10 phases established which signals matter, and the eval pipeline tells us when to adjust.

The eval-diagnose-train loop

After each training phase, an automated evaluation runs the model against random opponents and frozen previous-generation models. A diagnosis script compares metrics against behavioral thresholds and recommends reward flag adjustments.

What we measure

Win rate alone can't distinguish a good fighter from a passive one that wins by not dying. The eval tracks 40+ metrics per bot:

Every eval row in the results database also records its conditions—map, ship count, team size, deterministic vs sampled play. That column set exists because its absence once let two incomparable numbers sit side by side and support a wrong conclusion (see the war stories below).

Automated diagnosis

$ python3 eval_diagnose.py

  ISSUES FOUND:
    - Low hit rate: 2.6% (threshold: 5%)
    - Zero item pickups: 0
    - High engage distance: 3575u (threshold: 3000u)

  RECOMMENDED REWARD FLAGS:
    --reward-bullet-bonus 0.2  (reduce spam incentive)
    --reward-engage 1.0        (force closer engagement)
    --reward-pickup 0.15       (reward item collection)

  BEFORE ACTING ON THESE RECOMMENDATIONS:
    1. Compare against previous phase
    2. Pick 1-2 flags max
    3. Check if root cause is structural
    4. Form a hypothesis: "Changed X because Y, expect Z"

The script recommends but doesn't decide. The human reviews, picks 1-2 changes, forms a hypothesis, and pushes instructions. Changing many variables at once makes results unattributable.

Training history: what worked, what didn't

Each training phase follows a strict methodology: form a hypothesis ("increasing engage reward will fix passive play"), change one variable, train 2B steps, evaluate against multiple baselines (random, previous generations, best model), and diagnose using 40+ metrics—not just win rate. If the hypothesis was wrong, the metrics tell you why: was it passive play (high timeouts, low damage), overspecialization (beats one opponent, loses to random), or a reward imbalance (one ability dominates)?

95 phases, each building on the previous. The table below is an experiment log, not a changelog—selected rows; win rates are only comparable within a sim era (the observation space and mechanics grew from 90 to 100 floats over time, so a 95% from the small early sim and an 83% from today's full sim describe different games).

The LR bug (Phases 28–31 invalidated)

Phase 28 ran from scratch with minimal reward. Results looked promising. Then a code review revealed the learning rate schedule was counting minibatch steps instead of PPO updates—the LR decayed to zero after less than one real update. Every model from Phase 1 through Phase 31 had trained with LR≈0 for 99%+ of compute. Phase 25's "79% WR" came from less than half a gradient step of actual learning.

The fix was one line in the optimizer setup. Phase 32 REDO—the first properly trained model—hit 94.4% vs random in a single 2B-step run. More improvement in 7 hours than the previous 31 phases combined.

Self-play works, once (Phases 33–37)

Phase 35 warm-started from Phase 33 with Phase 32 as a frozen opponent at 30% mix. Result: 96.2% vs random, 76.8% vs Phase 13—new best across all benchmarks. But the recipe has strict constraints:

Zone reward as engagement regularizer (Phase 38)

KOTH training added a per-step zone reward (0.3, vs kill reward of 1.0) pulling ships toward the map center. The intended effect was zone-seeking behavior. The actual effect was stronger: the KOTH model beat the best FFA model 73.4% head-to-head in pure FFA combat. Zone reward forced more engagements (ships near center fight more often), producing richer gradient signal. The best combat model came from training for an objective other than combat.

Ship-specific abilities (Phases 41–44)

Each ship archetype gets a unique ability added to the sim and action space. The model trains from scratch with the expanded preset and learns when to use the ability alongside the 14 base combat actions.

Usage patterns reveal strategic learning: FocusFire activates 344 times per match (pre-engagement buff), while MineDash fires only 24 times (escape/engage tool, not spam). The models discover ability timing from reward signal alone—no explicit "use ability before fighting" shaping.

The Phase 13 breakthrough

Phase 13 was the first model to use all 7 combat abilities meaningfully. Prior models relied almost entirely on bullets and bombs. What changed:

The recipe: strong base model (Phase 12, trained with engage + bullet shaping) + one round of self-play. The base model learned what to do. Self-play taught it when.

Eval war stories: when the scoreboard lies

The hardest bugs in this project were never in the training code. They were in the measurement. Four times, the evaluation harness produced numbers that looked authoritative and were wrong—each one shaped a real decision before it was caught. This section exists because "the model got X%" turns out to be a claim about the eval harness as much as about the model.

1. The scoring bug that crowned the wrong champion

Team evals counted deaths among "all ships except ship 0" as the model's kills—which in 3v3 includes the model's own teammates. A team getting wiped registered as a win (two dead teammates outscore one dead opponent). The tell: a model playing a mirror match against itself scored 100% instead of 50%. Every number in the 95–99% comp round-robin was an artifact, and the composition it crowned "undefeated champion" had actually lost. After the fix (kills = enemy-team deaths, with a regression test), the ranking inverted: Gamma (Tempest+Comet+Hydra) beat the former champion Delta in both spawn orderings, ~74% averaged.

2. The ±18-point spawn slot

With scoring fixed, a same-policy mirror still refused to land near 50%. The cause took three diagnostic tools to isolate: map geometry isn't rotationally symmetric, so whichever team spawns in slots 0–2 carries about an 18-point advantage in real matchups. No single team win rate is trustworthy on its own. Every head-to-head since runs twice—normal and spawn-swapped—and reports the average. (The fully symmetric mirror turned out to be its own degenerate case: persistent distance ties broke by ship index and produced a fake 100%, which is why the diagnosis kept pointing in the wrong direction.)

3. The opponent fighting with 9 of its 14 actions

The eval loader detected an opponent's action space by trying each preset until one deserialized without error—and the duel preset was tried first. Flax tolerates extra keys in a parameter file, so every team-preset opponent "successfully" loaded as a duel model and silently fought without stealth, burst, rocket, warp, or portal. Every model-vs-model row in the results database had a handicapped opponent. The fix reads the parameter tree itself (head count and input width) instead of guessing—which also revived two-generation-old models as usable baselines, since input-width detection now truncates observations to fit.

4. Deterministic evaluation inverted the conclusion

The newly trained generalist was evaluated on a map it had never seen, using greedy (argmax) action selection. It scored 1.0% against the resident specialist—four wins in four hundred matches, twice in a row. The obvious conclusion: catastrophic map-transfer failure, bullets-per-match collapsing from 26 to 1, plan the next training phase around it. The trustworthy-recipe re-run (sampled actions, spawn-swap averaged, full-strength opponents) scored the same matchup at 47.4%. Parity. Greedy decoding off-distribution locks into degenerate loops that sampling never enters; the eval didn't soften the model's true performance, it inverted the verdict.

From specialists to a generalist (Phases 83–95)

Through Phase 90 the roadmap assumed the endgame was a library of specialists: a model per ship, a model per composition, pick the right one at runtime. Phases 91–92 tested the obvious next step—harden the champion composition against its strongest rival, once by warm-starting and once from scratch. Both regressed, by nearly identical amounts, on every benchmark. Training against a frozen strong opponent narrows a policy even when it starts from strength. That closed the specialist line.

Phase 93 inverted the premise: instead of fixing the lineup, randomize it. Every episode draws six random ships from the eight archetypes, both teams, no composition locks. One model, forced to play everything. It reached 78.8% vs random with uniform competence across the roster—and on the corridor map it had never trained on, the corrected head-to-heads (war story 4) showed it beating the resident champion 66.5%. Ship randomization generalized across ships. The remaining axis was maps.

Phase 94 applied the same trick to geometry. The trainer now accepts a comma-separated map list and splits its 4,096 environments into per-map sub-batches whose trajectories merge into every gradient update—one JIT compilation serves all maps because wall arrays share a padded shape. The honest cost: two half-size batches under-utilize the GPU, roughly halving throughput (16 hours instead of 8 for 2B steps). The result:

The per-ship column also did exactly what it was added to do: it found the next problem. Seven ships hold 50–71% on the corridor map; Hydra—the strongest ship at home—collapses to 27.4%. Its kit is the likely culprit (MIRV cluster missiles and X-Radar reward open space), and "why does Hydra fail in corridors" is a sharper question than any aggregate win rate has produced all project.

Two hypotheses fit that collapse. Either Hydra's kit is a genuine map mismatch—cluster missiles and X-radar that cannot work in four-second corridor brawls, an unfixable balance fact—or the 128-unit network had simply run out of room to hold corridor-Hydra strategy alongside everything else it had learned. The honest call at the time was the first one: a targeted diagnostic suggested a kit/decoding artifact with a low training ceiling, so the recommendation was to log it as balance and stop. The capacity test ran anyway.

Phase 95 changed exactly one structural thing from Phase 94: it doubled the hidden layers, 128 to 256, with double the training budget to fill the larger network. Random lineups, both maps, the reward—all held fixed. The result settled the question.

The lesson cuts against the recommendation that preceded it. "It's an unfixable kit mismatch" was a confident, specific, and wrong conclusion—and the only reason it got overturned is that the experiment ran despite it. More network capacity doesn't raise the ceiling on the map the model has already mastered; it buys room for the hard cases. That is the kind of thing you only learn by spending the compute instead of arguing about it.

Deployment

Trained models export from JAX (Flax parameters) to ONNX format, which UE5's Neural Network Engine (NNE) runs on CPU at inference time. The bot controller builds the same 100-float observation vector as the training sim and decodes the logit output into ship controls. Input and output sizes are auto-detected from the ONNX graph, so 20-, 30-, and 40-logit models from any era load through one pipeline.

Porting the controller from the duel era to the team era was not a recompile. The observation builder had drifted four floats behind the sim, and the differences were semantic, not just dimensional: the sim sorts teammates into the same tracked-ship slots as enemies (with trailing team bits saying which is which), and it trains with a scoring zone fixed at map center—so feeding zeros for zone features tells the policy "you are standing on the objective" everywhere. Both had to be reproduced exactly. The five team abilities decode through the same gameplay-ability system players use, with stealth gated to the one ship the sim allows. A per-game-mode model override map lets duel modes run duel specialists while everything else gets the generalist.

The export step verifies itself: every converted model must produce argmax-identical decisions to a reference implementation across 256 random observations before it ships. That check replaced a brittle absolute-tolerance assert that once failed a perfectly good model on float32 kernel noise—and the failure was nearly missed because a shell pipe swallowed the exit code. Verification, like evaluation, is part of the product.

The live model is the Phase 95 generalist—two 256-unit hidden layers, ~410KB, one network for every ship on every map—verified headless before deployment: it loads, auto-detects its own dimensions, and fights actively through a full automated match. Inference still runs well under a millisecond per bot per frame; doubling the width roughly triples the matmul cost, but the network is tiny next to everything else the frame is doing. Per-mode overrides can still slot the per-ship duel specialists into 1v1 modes.

Methodology limitations

This pipeline follows good experimental hygiene—single-variable changes, multiple baselines, behavioral metrics beyond win rate—but falls short of academic rigor in several ways worth being explicit about.

Single seed per experiment. Each phase runs once with one random seed. Phase 22 demonstrated seed sensitivity: identical hyperparameters produced 48% vs 37% depending on seed. The academic standard is 20+ seeds with confidence intervals. At 13 hours per run on a single RTX 3090, that's 10+ days per experiment—impractical here. Results should be read as "this seed produced X" rather than "this configuration reliably produces X."

Confidence intervals added late. Phases 11-25 were evaluated without confidence intervals. CIs (Wilson score, 95%) are now computed in the eval script. With 500 matches, the 95% CI is roughly ±3-4%, which means Phase 23 (73%) vs Phase 25 (79%) is likely significant, but Phase 24 (74%) vs Phase 23 (73%) probably isn't. Future phases report CIs; historical phases are point estimates only.

Ablation requires from-scratch training. Phase 27 attempted to ablate 9 reward coefficients by fine-tuning from a converged checkpoint (500M steps each). All 9 produced identical behavior—the policy was locked. Post-LR-fix, Phase 32 REDO trained from scratch with pure kill/death reward (all shaping zeroed) and achieved 94.4% immediately—confirming the problem was the LR bug, not reward design.

Learning curves underutilized. Training logs capture entropy, policy loss, and value loss per update, but these aren't systematically analyzed for plateau detection or collapse warnings. Plotting learning curves across phases would give earlier signal on whether a run is worth continuing.

Eval conditions were unrecorded for most of the project. The results database stored win rates without the map, ship count, team size, or action-selection mode that produced them. Two numbers from different conditions got compared at least once and briefly supported a wrong ranking. Conditions are recorded per row now; historical rows remain comparable only where the original logs pin down their setup. Related: cross-era comparisons on this page are intentionally avoided—the sim itself changed under the numbers.

What's next

Where is the real ceiling? Phase 95 showed that doubling the network fixed the hard cases (corridor Hydra, the rival specialists) but did nothing at home—83% there held flat across both sizes. That points at a ceiling set by the observation space or the sim itself, not by capacity. The open question is whether 83% is the game's honest skill cap for this observation, or whether a richer observation (longer memory, more tracked entities) moves it—and whether 256 units is itself now the new capacity wall for the hard maps, or just a waypoint.

Sampled action decoding in-engine. War story 4's footnote: the deployed bot plays its greedy policy, which is exactly the regime that degenerates off-distribution. Switching the controller to sample from the policy (as training does) is a small change with visible gameplay implications—it needs a playtest, not just an eval.

Fix multi-opponent rollout. The --opponent flag accepts multiple paths but only uses the first (frozen_pool[0]). Fixing this would enable opponent diversity without manual rotation—training against a pool of 3–5 prior generations simultaneously, which matters more now that frozen single opponents are known to cause regression.

Multi-map throughput. The per-map sub-batch design costs half the GPU's throughput at two maps. Alternating full-size batches between maps per update would restore it—worth building before a third map joins the mix.