RL Training Pipeline

The pipeline

Two machines work in parallel. A Mac runs game development, evaluation, and pushes training instructions. A Windows machine with an RTX 3090 trains models on GPU. They coordinate through git—the simplest protocol that works.

The simulation

Training happens in a custom JAX physics simulation that mirrors the C++ game engine. 4,096 arenas run in parallel on a single GPU, each simulating complete combat with Newtonian physics, projectile collision, energy management, and up to 17 combat abilities (14 base + 3 ship-specific).

The simulation must match the game exactly or the trained model won't transfer. 31 cross-check tests verify parity: thrust values, wall bouncing, bullet damage, bomb splash radius, mine proximity triggers, ship-specific stats, ability mechanics. When a value changes in C++, the same change must be made in Python. 189 behavior tests and 86 visual tests verify the UE5 side.

Observation vector (96 floats)

The neural network sees the world as 96 numbers, matching what a human player could perceive:

Action space (14–17 discrete actions, 40–46 logits)

The base "team" preset has 14 actions (40 logits). Ship-specific presets add 1–3 extra binary actions for unique abilities:

Constraints & trade-offs

The simulation makes deliberate simplifications to hit 570K steps/sec. Each trade-off has a cost and a way we verify it doesn't break transfer to the real game.

Reward shaping

The reward function has 9 tunable coefficients, each exposed as a CLI flag. The defaults were calibrated through iteration—the first 10 phases established which signals matter, and the eval pipeline tells us when to adjust.

The eval-diagnose-train loop

After each training phase, an automated evaluation runs the model against random opponents and frozen previous-generation models. A diagnosis script compares metrics against behavioral thresholds and recommends reward flag adjustments.

What we measure

Win rate alone can't distinguish a good fighter from a passive one that wins by not dying. The eval tracks 40+ metrics per bot:

Automated diagnosis

$ python3 eval_diagnose.py

  ISSUES FOUND:
    - Low hit rate: 2.6% (threshold: 5%)
    - Zero item pickups: 0
    - High engage distance: 3575u (threshold: 3000u)

  RECOMMENDED REWARD FLAGS:
    --reward-bullet-bonus 0.2  (reduce spam incentive)
    --reward-engage 1.0        (force closer engagement)
    --reward-pickup 0.15       (reward item collection)

  BEFORE ACTING ON THESE RECOMMENDATIONS:
    1. Compare against previous phase
    2. Pick 1-2 flags max
    3. Check if root cause is structural
    4. Form a hypothesis: "Changed X because Y, expect Z"

The script recommends but doesn't decide. The human reviews, picks 1-2 changes, forms a hypothesis, and pushes instructions. Changing many variables at once makes results unattributable.

Training history: what worked, what didn't

Each training phase follows a strict methodology: form a hypothesis ("increasing engage reward will fix passive play"), change one variable, train 2B steps, evaluate against multiple baselines (random, previous generations, best model), and diagnose using 40+ metrics—not just win rate. If the hypothesis was wrong, the metrics tell you why: was it passive play (high timeouts, low damage), overspecialization (beats one opponent, loses to random), or a reward imbalance (one ability dominates)?

44 phases, each building on the previous. The table below is an experiment log, not a changelog.

The LR bug (Phases 28–31 invalidated)

Phase 28 ran from scratch with minimal reward. Results looked promising. Then a code review revealed the learning rate schedule was counting minibatch steps instead of PPO updates—the LR decayed to zero after less than one real update. Every model from Phase 1 through Phase 31 had trained with LR≈0 for 99%+ of compute. Phase 25's "79% WR" came from less than half a gradient step of actual learning.

The fix was one line in the optimizer setup. Phase 32 REDO—the first properly trained model—hit 94.4% vs random in a single 2B-step run. More improvement in 7 hours than the previous 31 phases combined.

Self-play works, once (Phases 33–37)

Phase 35 warm-started from Phase 33 with Phase 32 as a frozen opponent at 30% mix. Result: 96.2% vs random, 76.8% vs Phase 13—new best across all benchmarks. But the recipe has strict constraints:

Zone reward as engagement regularizer (Phase 38)

KOTH training added a per-step zone reward (0.3, vs kill reward of 1.0) pulling ships toward the map center. The intended effect was zone-seeking behavior. The actual effect was stronger: the KOTH model beat the best FFA model 73.4% head-to-head in pure FFA combat. Zone reward forced more engagements (ships near center fight more often), producing richer gradient signal. The best combat model came from training for an objective other than combat.

Ship-specific abilities (Phases 41–44)

Each ship archetype gets a unique ability added to the sim and action space. The model trains from scratch with the expanded preset and learns when to use the ability alongside the 14 base combat actions.

Usage patterns reveal strategic learning: FocusFire activates 344 times per match (pre-engagement buff), while MineDash fires only 24 times (escape/engage tool, not spam). The models discover ability timing from reward signal alone—no explicit "use ability before fighting" shaping.

The Phase 13 breakthrough

Phase 13 was the first model to use all 7 combat abilities meaningfully. Prior models relied almost entirely on bullets and bombs. What changed:

The recipe: strong base model (Phase 12, trained with engage + bullet shaping) + one round of self-play. The base model learned what to do. Self-play taught it when.

Deployment

Trained models export from JAX (Flax parameters) to ONNX format, which UE5's Neural Network Engine (NNE) runs on CPU at inference time. The bot controller builds the same 96-float observation vector as the training sim and decodes the logit output into ship controls. Ship-specific models use different action presets (42–46 logits) but share the same observation space, so one inference pipeline handles all ship types.

Model size is ~130–140KB (two 128-unit hidden layers). Inference runs in <0.1ms per bot per frame. The game loads the correct model per ship archetype via the ability loadout system—Viper bots use the FocusFire model, Lurker bots use the MineDash model, and generic ships fall back to the base KOTH model.

Methodology limitations

This pipeline follows good experimental hygiene—single-variable changes, multiple baselines, behavioral metrics beyond win rate—but falls short of academic rigor in several ways worth being explicit about.

Single seed per experiment. Each phase runs once with one random seed. Phase 22 demonstrated seed sensitivity: identical hyperparameters produced 48% vs 37% depending on seed. The academic standard is 20+ seeds with confidence intervals. At 13 hours per run on a single RTX 3090, that's 10+ days per experiment—impractical here. Results should be read as "this seed produced X" rather than "this configuration reliably produces X."

Confidence intervals added late. Phases 11-25 were evaluated without confidence intervals. CIs (Wilson score, 95%) are now computed in the eval script. With 500 matches, the 95% CI is roughly ±3-4%, which means Phase 23 (73%) vs Phase 25 (79%) is likely significant, but Phase 24 (74%) vs Phase 23 (73%) probably isn't. Future phases report CIs; historical phases are point estimates only.

Ablation requires from-scratch training. Phase 27 attempted to ablate 9 reward coefficients by fine-tuning from a converged checkpoint (500M steps each). All 9 produced identical behavior—the policy was locked. Post-LR-fix, Phase 32 REDO trained from scratch with pure kill/death reward (all shaping zeroed) and achieved 94.4% immediately—confirming the problem was the LR bug, not reward design.

Learning curves underutilized. Training logs capture entropy, policy loss, and value loss per update, but these aren't systematically analyzed for plateau detection or collapse warnings. Plotting learning curves across phases would give earlier signal on whether a run is worth continuing.

What's next

More ship abilities. Three Lattice-faction item abilities are designed (Hex Burst, Cascade Bomb, Stun Mine) and planned for Bastion and Titan. The action space is forward-compatible—reserved slots for faction-specific items avoid retraining existing models.

Fix multi-opponent rollout. The --opponent flag accepts multiple paths but only uses the first (frozen_pool[0]). Fixing this would enable opponent diversity without manual rotation—training against a pool of 3–5 prior generations simultaneously.

Larger models for ship-specific training. Ship-specific models start from scratch and need 4B+ steps to match the generic model's accumulated lineage (6B+ across P32→P38). A 256×256 network or longer training budget would close this gap.