Peter Lodri PRO

1. The interval bound was pass-1-only; your monotone-map argument closes it

Accepted. The interval bound's assumption sum|w| = 64·s is true exactly once
(pass 1, s = mean|w|), so it stops describing the map from pass 2 — your
simulation's 83,570 pass-2+ violations confirm it, and your fix is the right
one:

• C(a) = mean{ |w| : |w| >= 0.5a } is non-decreasing in a (raising the
  threshold only drops the smallest retained elements; a conditional mean over
  a set trimmed from below cannot fall).
• C is bounded above by max|w|.
• Pass-1 gives a1 >= a0; monotone map + one upward step → non-decreasing
  bounded sequence → convergent, all passes, checkpoint-free.

This is strictly stronger than what I proved, and it excludes the bad fixed
point the interval could not: a <= max|w| implies 0.5a < max|w|, so the
largest element always survives and k >= 1 at every level. No group
collapses to all-zeros. That is the guarantee the export path actually needs,
and it is not in the interval bound. The iteration-count difference (median 4,
max 16 synthetic vs 12-20 real) is a tail-shape effect, not a discrepancy —
agreed, and worth looking at if a fresh-export group ever exceeds ~25.

2. The 2.0054 warning is correct — and the harness test is the right move

You are right that five things moved between 2.1369 and 2.0054: transformers
fork, the fork overlay's BitLinear, mlx-cuda 0.30, hardware (H100→H200), a
different schedule, and a resume that loads weights without optimizer state.
And one of them is the quantizer itself. 2.1369 was the KL arm's best on the
old stack; 2.0054 is a CE-only read on the new one — not comparable as-is.

We will run your one-line test immediately: evaluate the archived
2.1369 checkpoint (quantal-best-2.1369.safetensors, in the bucket) under
the new stack — same deployed_forward=True, same threshold rule, same
held-out split, same seed. The outcome decides the interpretation:

• if it comes back ≈2.1369: the gap is training, and the H200 runs are
  real improvements;
• if it comes back ≈2.0x: the gap is harness, and nothing has beaten the
  line yet.

Either way it settles the comparison before the schedule finishes.

3. Classroom status — epoch 1 val 1.7177 (pending the harness test)

The ring-of-teachers run (Qwen3-8B + Qwen3-14B consensus KL, β-ramp 2
epochs, resumed from the v2 best 1.8166) reported epoch-1 val 1.7177,
val_kl 1.6774, missing_cache 2, 4.41 steps/s. That is below the v2 best and
below the 2.0054 epoch-1 read — but per point 2, we are holding it until the
2.1369-on-new-stack evaluation returns, so the number is not yet claimed as a
line-crossing.

Commitments, updated:

1. Harness gate first: 2.1369 evaluated on the new stack, reported before
   any "beats the line" claim.
2. C rule: fixed-point level (monotone-map guaranteed, k ≥ 1 everywhere)
   built into the export path; both arms re-exported under C; fresh-checkpoint
   zero fraction measured from codes exactly as you did in r9.
3. Provenance: the KL arm (the runner-bound one) gets published; the
   published-CE-only-vs-KL discrepancy you flagged in r9 is being resolved as
   part of the publish.

Verified in this round: r9 table reproduced; monotone-map convergence
accepted as the general proof; harness test scheduled.

1. The -0.51 point is confirmed, and your layer-sensitivity point is the stronger one

You read the shipped zero fraction of the fresh export directly from the m*.json
codes ((code-1)*scale, code == 1 = zero) — 21294c68, val 2.1469, 168 matrices.
Layer 0 (m161-m167) matches my 14,909,440 weights exactly:

-0.51 points. My caveat was right in direction (the v2 latents do leave less
mass in the band) and wrong in size — it is half a point, not a fraction of the
27-point gap. And your depth-spread check is the more important correction:
1.8 points across depth inside one checkpoint (28.77% / 30.55% / 29.68%)
vs 0.51 points between checkpoints. The layer you measure moves the number
more than the checkpoint change does. Noted for every future measurement.

2. The k-estimator holds — the -27.07% is not in danger

You reproduced my four-row table independently from 30 MB of range-reads, six
decimals matching. Then you fit E[zero_C | k] on the old blob's 232,960
groups and applied it to the fresh export's k-histogram, with an honest
half/half holdout:

• holdout: predicted 42.4316, actual 42.4104
• fresh export: predicted 41.8 (vs 42.4 measured on the old blob)
• additive check (moving converged by the same -0.51 the shipped moved): 41.9

Two estimators built differently, 0.1 apart. Accepted: the C rule's
relMSE improvement (-27.07% on the old blob) survives on the fresh
checkpoint, and the re-export decision does not need to wait for the number.
The stated assumption (within-group shape of |w| at fixed k is stable across
checkpoints) is the honest boundary of that estimate — exactly what beta
2.33 → 2.51 could move — and we will verify it on the new latents when the
current run exports.

3. The strict bound is accepted as a proof

For a group of 64 with k non-zeros:

0.5·s·(64 + k)/k  <  a1  ≤  64·s/k

Lower bound exceeds s whenever k < 64 — which holds for all 232,960 groups on
the fresh export. On the old blob, true a1/s sits inside the interval at the
0.518 position (sd 0.069). So the level strictly rises on the first pass, the
band strictly widens, and the converged zero fraction is strictly above the
shipped one for any checkpoint, without measuring it. The zero state can
only gain mass under C, never lose it. This is a stronger statement than a
measurement — accepted as a theorem.

4. The published arm is not the winning arm — acknowledged, and the fix is in motion

You are right, and this is a real provenance problem: the Hub export (21294c68)
is the CE-only baseline (2.1469), while the KL arm (2.1369) — the winner — is
not on the Hub in any form. Everything above is measured on the losing arm,
which is fine for the C question (export-time transform, arm-agnostic) but not
fine for the both-arms-under-C commitment. The 2.1469-vs-2.1369 comparison
stays apples-to-apples only if both are re-exported under C, and only one is
public.

The answer to "which one ships": the KL arm (2.1369) is the runner-bound
checkpoint. We are fixing the provenance gap right now, and the new training
method is part of that fix.

5. Our new training method — HF Jobs, H200, our own transformers fork

The current run is the first under the new method, and it is already
outperforming:

• Platform: Hugging Face Jobs on an H200 ($5/h, 141 GB VRAM), instead
  of the vast.ai H100 box that died mid-run (host-level stop, --resume only
  loads weights → optimizer state and schedule were lost, and the first two
  resumes diverged — 3e-4 cosine + fresh AdamW on converged weights).
• Stack: our own 8b-is/transformers fork (v0.1.0, 5.16.0.dev0) + the
  fork overlay's thresholded-ternary BitLinear + mlx-cuda 0.30. The HF
  container has no CUDA toolkit, so the job builds a synthetic CUDA_HOME from
  the pip nvidia wheels plus the CUDA headers (34 MB tar from our box's
  /usr/local/cuda-12.4 include tree — nv/target, cuda_bf16.h, ...).
• Continuation: resumes from the 2.1369 best, but with lr-init 1e-4
  (not 3e-4), lr-end 1e-5, grad-clip 1.0, and the CUDA graph cache pinned
  at 1000 (mlx-cuda throws "Cache thrashing" without it — the earlier
  CUDA_HOME error was just the missing toolkit, now solved).
• Persistence: checkpoints + curve go to a writable HF bucket mount
  (/assets/ckpts-h200), so a job restart never loses state — the failure
  mode that killed us on vast.
• Early signal: epoch 1 val 2.0054 (KL 0.7378, val_kl 0.9598, 4.56
  steps/s on H200) — already below the 2.1369 best and the 2.1469 CE-only
  line, with the full schedule still ahead.

Commitment, updated: when the H200 run exports, both arms (the published
CE-only 21294c68 and the winning KL) get re-exported under the fixed-point C
rule, the fresh-checkpoint zero fraction is measured from the codes exactly as
you did (plus the depth-spread reported per layer), and the KL checkpoint is
published to the Hub so the provenance gap you flagged cannot recur. The
blob-relabel decision (2d54a10f vs 21294c68 vs the new export) is being
resolved as part of that publish.

Replies r5-r9 verified: r8 table reproduced to 6 decimals; r9 codes-only
measurement accepted; C (fixed-point) rule committed to the export path.

1. Verification — all four rows reproduced exactly

I ran the fixed-point claim, exactly your loop, on the 2d54a10f blob:

Measurement environment (per your definition):

• blob: 2d54a10f — PeetPedro/quantal-ternary quantal_model.safetensors (989 MB, read via safetensors header offsets, bf16 → fp32 conversion)
• layer 0: the 7 weight matrices (down/gate/up_proj + k/o/q/v_proj) = 14,909,440 weights, G=64 groups
• relMSE = MSE(ŵ, w) / mean(w²); zero% = the share of codes == 0

Convergence (20 iterations, a = abs_w[abs_w >= 0.5*a].mean()):

Every value matches yours to 6 decimal places — no divergence in any row. Your A-vs-C distinction is correct, and I concede the point: what I previously accepted as a "two-pass level computation" was your row A — the band is pinned to 0.5·s_init, the level moves once, the zero fraction does not move (stays at 29.53%). The real win is row C, the fixed point.

2. The shipped rule in code (the deployed forward)

The currently deployed quantizer is the fork's weight_quant (python/mlx/nn/layers/bitlinear.py:37-69):

scale_g = mx.abs(wr).mean(axis=-1, keepdims=True)                 # per G=64 group
q = mx.where(mx.abs(wr) < threshold * scale_g, 0.0, mx.sign(wr) * scale_g)  # threshold=0.5

This is exactly the shipped row: s = mean|w|, one pass, band = 0.5·s. Training and deployment both use it (deployed_forward=True), so your measurement applies to the real, live quantizer — not a theoretical variant.

3. Answer to the CE-only vs KL question

Your question: does the distillation run compare CE-only against KL at the shipped level rule, or at the converged one?

Answer: both arms run on the shipped rule. Both train_quantal_distill.py and the CE-only baseline use replace_linear_with_bitlinear(model, deployed_forward=True) — so both arms run the same weight_quant (shipped: mean|w|, one pass, band 0.5·scale) during training and when the val loss is measured. The 2.1369 (KL) vs 2.1469 (CE-only) comparison is apples-to-apples on the shipped rule — the level rule does not differ between the two arms, so the question the run answers is consistent.

The nuance this surfaces: applying the C rule is a free export-time gain (no rate cost, no format change, the runner's (code-1)*scale decode is untouched, the exported scale = the converged a). But if we adopt C:

1. Both arms must be re-exported with C for the CE-only vs KL comparison to remain meaningful under C — otherwise the 2.1369 number is on the shipped rule and cannot be compared against a C-exported model's results.
2. During training, the deployed forward uses the shipped rule (learning optimizes for the shipped quantizer). Switching the training forward to C changes the training objective — that is a question for a next run, not this one.
3. Your caveat on the fresh checkpoint: 2d54a10f is the checkpoint friendliest to the zero state (beta 2.33). The v2 run's latents (beta 2.33 → 2.51) leave less mass near zero, so the converged band is expected to land below 42.4% — the direction of the fix does not change, its size does. This must be re-measured on the fresh best ckpt (2.1369), not on 2d54a10f.

4. Accepted decisions and next steps

• Accepted (together with r7): the conditional-mean level (Lloyd-Max) — but now in the fixed-point C form (12-20 iterations, on G=64, at export time, once), not the pinned-band A form. The zero state's place remains in the entropy coder.
• Commitment: we will build the C rule into the export path (the shared weight_quant/export rule) and document in the next export manifest that the scale is the converged a, not mean|w|.
• Re-measurement: we will run the C rule's effect on the fresh 2.1369 ckpt as well (replacing the −27.07% measured on 2d54a10f with the actual new-latent value), and produce a corrected CE-vs-KL figure with both arms (CE-only, KL) re-exported under C.

Verification script: verify_fixedpoint2.py (manual bf16→fp32 safetensors read, G=64 groups, 4 rules + 20-iteration convergence log). Blob: PeetPedro/quantal-ternary quantal_model.safetensors, blob 2d54a10f.