ctde_v0.run_serial32¶
Serial-32 orchestrator — run every 32²/10 unit ONE AT A TIME, pack the light units in parallel.
WHY: one 96 GB card. A Lagrangian 32²/10 training needs ~50 GB (dual + aux λ̂₂ estimator back-propagated through the 100-step rollout); the fixed-penalty variant ~17 GB; 24²/6 ~8.8 GB; 16²/4 ~3.5 GB. Two heavy units = ~100 GB > 96 → OOM. So the 32² work MUST be serialized. The light units are cheap and can safely run alongside the single heavy one.
TWO TIERS (launched as two coexisting tmux sessions by run_serial32.sh): --tier heavy : every grid==32 unit across T2/T3/T4, jobs=1 → strictly ONE at a time (~50 GB). --tier light : every grid<32 unit across T2/T3/T4, jobs=J → packed (~J×9 GB). Peak = 1 heavy (~50 GB) + J light (~J×9 GB). J=3 → ~77 GB, comfortably under 96 with compile headroom. Because 'heavy' is jobs=1 and 'light' only ever holds grid<32 units, the number of 32² trainings on the card is provably ≤ 1 at all times — bulletproof, no cross-session race.
Both tiers write checkpoints into the SAME runs/{roles_t2,lag_t3,critic_t4} tree, so the existing _report_for.sh renders each suite unchanged, and every unit is resumable (a run-dir with a model.eqx is skipped). Each unit carries its own suite's iters (T2=1500, T3/T4=2000). All units are FROM SCRATCH (no warm-start, no cross-unit deps) so scheduling is a plain bounded pool.
# plan only (no training):
PYTHONPATH=.:../../../FiedlerValueEstimation ~/ZymeraLab/.venv/bin/python -m ctde_v0.run_serial32 --tier heavy --dry-run
# the tiers (run_serial32.sh launches both as tmux zheavy/zlight):
... -m ctde_v0.run_serial32 --tier heavy --jobs 1 --seeds 3 --rollouts 16
... -m ctde_v0.run_serial32 --tier light --jobs 3 --seeds 3 --rollouts 16
===>>> This launcher LAUNCHES REAL TRAINING; run it deliberately. <<<===