ctde_v0.run_obstacle_overnight

run_obstacle_overnight

Overnight OBSTACLE batch launcher — the full-factorial obstacle/exploration grid.

Launches, on the FIXED honest spec — comm_r = 5 every world, hard collision-mask on, soft/learned connectivity (no guardrail), frontier-attn explorer, 100-step horizon — a FULL FACTORIAL over four axes that probe how the SuperBlue arms cope as the world fills with walls. Every run is graded on the metrics the trainer logs every iter: coverage, the REAL connectivity bar (λ₂>0.5) alongside the trivial one, behavioural diversity (SND) and role-distinctness. Unlike the skills ladder, EVERY run here is trained FRESH — no warm-start, no inter-run deps (needs_dir=None, init_from=None for every unit).

FOUR axes, crossed in full (3 × 2 × 3 × 3 = 54 units per seed):

ARM (3) — what coordinates the team: role : the hand-coded explorer/relay split (--role-picker expl_relay). base : the homogeneous shared explorer, no extra flags. The REFERENCE. sel : the learned role-selector (--selector on --flock scripted --congestion off).

BARRIER (2) — the connectivity barrier penalty weight: boff : barrier off (--barrier-weight 0). bon : barrier on (--barrier-weight 5).

EXPLORE (3) — the exploration drive: eoff : no extra exploration shaping. ebump : a heavier coverage reward (--w-coverage 3). einfo : an explicit info-gain bonus (--explore-infogain on).

WORLD (3) — grid size / agent count / terrain (comm_r FIXED 5 everywhere): o16 : 16×16, 4 agents, open terrain (--terrain open). r24 : 24×24, 6 agents, 3 rooms (--terrain rooms --rooms 3). r32 : 32×32, 10 agents, 4 rooms (--terrain rooms --rooms 4).

Per seed: 3 arms × 2 barrier × 3 explore × 3 worlds = 54 units. With --seeds 1 (the DEFAULT) that is 54 runs; --seeds 2 is 108. Each unit's run-dir is <out>/seed{S}/{world}/{arm}_{barrier}_{explore} (e.g. runs/obstacle/seed0/r24/sel_bon_einfo).

DEPENDENCY-AWARE + RESUMABLE (but every unit here is independent): each run is a unit whose run-dir is skipped once it holds a model.eqx; since nothing warm-starts anything, all units are immediately ready. --jobs N runs up to N train_ctde subprocesses at once — balthar is MEMORY-BOUND (~3 concurrent @32²/10, ~6 @24²/6), so keep N small if the r32 world is in play. --dry-run prints the full plan (every unit: id, grid/N, all extra flags) and launches nothing.

# see the whole plan first (no training):
JAX_PLATFORMS=cpu PYTHONPATH=.:../../../FiedlerValueEstimation     /Users/bijanmehr/Project.Zymera/zymera_lab/.venv/bin/python         ctde_v0/run_obstacle_overnight.py --dry-run --seeds 1

# the real overnight batch on balthar (drop JAX_PLATFORMS so it uses the GPU), 2 at a time:
PYTHONPATH=.:../../../FiedlerValueEstimation ~/ZymeraLab/.venv/bin/python         ctde_v0/run_obstacle_overnight.py --out runs/obstacle --seeds 1 --iters 1500         --rollouts 16 --jobs 2

===>>> This launcher LAUNCHES REAL TRAINING; run it deliberately. <<<===

Unit

Unit(uid, run_dir, rung, seed, extra, init_from_dir, needs_dir)

One train_ctde run: an id, its run-dir, the extra flags, the seed/world, and the run-dir of a predecessor whose model.eqx must exist before it can start (here always None — every obstacle run is FRESH).

Source code in experiments/ctde_v0/run_obstacle_overnight.py

def __init__(self, uid, run_dir, rung, seed, extra, init_from_dir, needs_dir):
    self.uid = uid
    self.run_dir = run_dir
    self.rung = rung                     # (g, n, r)
    self.seed = seed
    self.extra = extra
    self.init_from_dir = init_from_dir   # run-dir whose model.eqx warm-starts this, or None
    self.needs_dir = needs_dir           # run-dir that must be done first, or None
    self.proc = None