Dipankar Sarkar PRO

The framing is the right one. Cost per token is the axis, not parameter count, and inference is where the bill actually runs. That part I will defend all day.

The 23.4x is where I would push. 25.7 to 601 tok/s on a 35B-A3B MoE (3B active) on a B200: the ceiling number is aggregate-under-concurrency, but 25.7 reads like single-stream unbatched generate. Those are different axes. Batching alone moves an MoE by an order of magnitude before any VKAE mechanism kicks in. The honest speedup is same-concurrency both sides, same batch, same harness, VKAE on vs off. Otherwise part of what you are measuring is 'we turned batching on.'

Same for 'no quality loss.' Measured on what? Aggregate perplexity or a task average sits still while long-context recall or the rare hard token quietly moves. The loss lives in the tail, not the mean.

When the paper lands, will the headline number be VKAE-on vs VKAE-off at identical concurrency, or optimized-throughput vs a single-stream baseline?

Fair. You did disclose all three, and I read the write-up now. The halting Q&A is the best part of it.

The regularizer finding is the real result. 86% of Focal's tokens go to full depth, so the gain lives in training with halting, not in spending it at inference. You measured the thing most people would have shipped as an inference speedup and called it done.

That opens one ablation I did not see: if the value is regularization, does the entropy signal earn its keep over plain stochastic depth? Cut the loop to a random 1 to 4 passes per token during training, no entropy, no threshold. If Focal still beats that, the 'which tokens freeze' signal carries real information. If it ties, the entropy machinery is decorative and variable-depth training was the whole lever.

Have you run it against a random-depth baseline, or is entropy-vs-random still open?

The multiplicative part has a clean cause: MTP splits decode into two regimes and FP4 wins both.

Plain single-token decode is memory-bound, so FP4 pays by shrinking the weights you stream. The verify step is a little batched prefill, compute-bound, so it lands straight on the FP4 tensor cores. Speculative decoding normally only buys you the memory-bound half. You are collecting the compute-bound half too.

Which is why the whole thing rides on acceptance holding. Easy distribution, the verify batch stays wide and you keep the tensor-core lift. Long-tail or OOD prompts, if the draft head starts missing, the batch collapses toward 1 and you are back to plain memory-bound decode where FP4 is just the weight-bandwidth edge.

Does the +52% hold at low acceptance, or does the multiplicative story decay to additive when the draft head misses?

The harness that kills your own best result is the whole job. Nice.

The trap you caught has a direction, which is what makes it dangerous. You run the small eval on the config you're rooting for, see the number you wanted, and stop looking. Confirmation bias and low n multiply. The 8-prompt spectacular is never the config you doubt.

One thing I'd promote to first-class: report the seed band, not the mean. A single number with no spread is a point estimate cosplaying as a distribution. '+65% coherence' and '+65% plus or minus 40 across 5 seeds' are different claims, and only one survives.

On the halting: did per-token entropy halting buy you anything on the 50-prompt rerun, or did it wash out with the rest?

Since KV1 is training-aware not drop-in, you can't borrow a bigger model's competence for free. That's the real bind with going small.

So skip needle-in-haystack (it needs capability an 8M won't have) and use a long-range task an 8M CAN learn: induction/copy where the target token depends on one planted ~60 positions back. Train that, then 2-bit vs 16-bit K/V and measure recall of the planted token by depth.

Same size, isolates the cache under a real long-range dependency. KLD averages straight over that, a copy task can't hide it.

Would that fit your current training budget?

5.33x shrink is the easy number. The one I'd chase is task-level, not KLD.

0.0916 nats/token is an average over 372k positions, and a KV cache lives or dies on the rare position that mattered: the token at 90k you have to attend back to. Average KLD can look tiny while long-context recall quietly falls off, because the easy majority of positions dominate the mean.

Before "128K on a normal machine," I'd want a needle-in-a-haystack sweep: same passkey at depth 8k / 32k / 128k, 2-bit KV vs 16-bit, pass/fail not perplexity. That's the eval that proves the claim.

Have you run anything retrieval-shaped over long context yet, or just WikiText so far?

Good catch, that path is dead. I folded the four standalone demos into one Space to stay under HF's 3-slot cpu-basic quota, so grite moved.

Live here, running over local git data (the grite tab):
https://huggingface.co/spaces/neullabs/agent-infra

Thanks for flagging it. Did it 404 for you, or hang on a cold boot?