ctde_v0.run_ab_overnight

run_ab_overnight

Overnight A/B batch launcher — the SuperBlue connectivity-coverage arms.

Launches, on the FIXED honest spec — comm_r = 5 every rung, hard collision-mask on, soft/learned connectivity (no guardrail), frontier-attn explorer, 100-step horizon — the four arms across the density-pinned ladder, --seeds seeds each. Everything is graded on the metrics the trainer now logs every iter: coverage, the REAL connectivity bar (λ₂>0.5) alongside the trivial one, behavioural diversity (SND) and role-distinctness.

base : the shared frontier-attn explorer, warm-start ladder 16->24->32. The REFERENCE, and the single-Actor BOOTSTRAP the fork replicates from. armA : curriculum + randomness — the same ladder with --warmstart-noise at each rung — then a DTE tail (--critic-mode decentral) warm-started from the top rung. bfork : warm-start from base's single-Actor ckpt AT EACH RUNG, replicate into 2 groups (--fork-groups 2); explorer/relay specialization emerges from per-group gradients. bdico : the shared policy + a per-agent mean-zero diversity residual (--diversity-residual on), warm-start ladder — controlled diversity without forking.

DEPENDENCY-AWARE + RESUMABLE: each run is a unit whose run-dir is skipped once it holds a model.eqx; a unit launches only once its needs predecessor's ckpt exists (base-rung -> bfork-rung; rung i-1 -> rung i within a ladder; armA-top -> armA-DTE). --jobs N runs up to N train_ctde subprocesses at once — balthar is MEMORY-BOUND (~3 concurrent @32²/10, ~6 @24²/6), so keep N small at the top rung. --dry-run prints the full plan + deps and launches nothing.

# see the whole plan first (no training):
JAX_PLATFORMS=cpu PYTHONPATH=.:../../../FiedlerValueEstimation     /Users/bijanmehr/Project.Zymera/zymera_lab/.venv/bin/python         ctde_v0/run_ab_overnight.py --dry-run

# the real overnight batch on balthar (drop JAX_PLATFORMS so it uses the GPU), 2 at a time:
PYTHONPATH=.:../../../FiedlerValueEstimation ~/ZymeraLab/.venv/bin/python         ctde_v0/run_ab_overnight.py --out runs/ab --seeds 3 --iters 1500 --rollouts 16 --jobs 2

===>>> This launcher LAUNCHES REAL TRAINING; run it deliberately. <<<===

Unit

Unit(uid, run_dir, rung, seed, extra, init_from_dir, needs_dir)

One train_ctde run: an id, its run-dir, the extra flags, the seed/rung, and the run-dir of a predecessor whose model.eqx must exist before it can start (or None).

Source code in experiments/ctde_v0/run_ab_overnight.py

def __init__(self, uid, run_dir, rung, seed, extra, init_from_dir, needs_dir):
    self.uid = uid
    self.run_dir = run_dir
    self.rung = rung                     # (g, n, r)
    self.seed = seed
    self.extra = extra
    self.init_from_dir = init_from_dir   # run-dir whose model.eqx warm-starts this, or None
    self.needs_dir = needs_dir           # run-dir that must be done first, or None
    self.proc = None