Ctrl-G × NWA — benchmark suite & HMM program

Headline — constrained decoding vs own baseline, per benchmark

Task metric on the full test split; paired on identical examples. flips = examples lost/gained vs baseline.

table view

TinyZero split · equation-correctness. Zero losses on every mask condition; mask_bon beats free BoN with a validity guarantee.

table view

IBM scorer. Full Ctrl-G (λ=0.1) ties mask-only FSM with perfect validity and the best Partial of any condition.

table view

FSM approaches mask-only from below as λ→0; validity is perfect at λ≥0.5. Dashed line = mask-only reference.

table view

Chain-validity is the metric masking cannot move (its residual is truncation); the guide converts it.

table view

Distance from the true constrained distribution vs guide capacity. Guidance beats masking only past a quality bar; oracle = exactly 0.

table view

Upper bound on mask-addressable mass: walker-invalid (parse) + incomplete vs semantic failures, per unconstrained run.

table view

Local 9B/8B + constraint vs frontier API models. Bars: frontier range per task; dots: our best constrained local result.

table view

Racket verifier all-False on compute nodes — rackunit lived in per-node ~/.local; fixed via shared PLTADDONDIR. Labels rescored; Llama baseline 0.006→0.193 after restatement-aware verification.
Three mask regressions (nested_arith −5.4, planetarium −2.3, racket-Llama −9.4) — all constraint-language-too-tight; fixed by the projection audit rule: widen the walker until verifier-correct baseline answers are never rejected. All flipped to no-loss.
Walker dead-end → NaN → silent mask-off at the arith line cap (138/384 records) — walker made provably viable; processor now forces EOS instead of emitting all-−inf rows.
Warm-up budget mismatch made all countdown constrained runs cluster at 0.789 — replicate the baseline's full generation config; at budget parity every mask condition is a strict win.
HMM lane "net harmful" (M15) overturned — composite-token scoring (alphabet), full-alphabet corpus (distribution), λ tuning (strength). Now: perfect validity at FSM parity.
Ops: two disk-quota outages mimicked node failures (probe writes first); node-5 GPU 7 squatted; 12h-wall shards resumable by design.