244 to 3.4 ms/tok: the NPU optimization sprint
Seven engine versions in four days took NPU inference from 1,930 ms/tok to 3.4 ms/tok, a 72x speedup through batch decode, fused dispatch, and INT8 GEMM. Every throughput figure here is historical and unsourced, and this post is the record of that sprint, not the claim.
Four days, seven versions
Seven engine versions in four days took NPU inference from 1,930 ms/tok to 3.4 ms/tok. The range this post measures, the one behind the 72x, runs from the last baseline we trusted at 244 ms/tok down to the final 3.4 ms/tok. The sprint was not one lucky rewrite. It was a sequence of measured steps, each one recorded, each one kept only when the improvement reproduced on the next run.
| Dimension | Value |
|---|---|
| Engine versions | 7 |
| Days | 4 |
| Start of the sprint | 1,930 ms/tok |
| Last trusted baseline | 244 ms/tok |
| End of the sprint | 3.4 ms/tok |
| Speedup | 72x |
Batch decode
The first lever was batch decode. The NPU sat idle in the batch dimension, paying the full dispatch round trip for one token at a time. Batching the decode work into a single pass turned the same compute into many tokens, and the first large step down came from that alone.
Fused dispatch
The second was fused dispatch. Attention and feed-forward were still separate round trips, and every round trip on this NPU carries launch and wait overhead that dwarfs the actual multiply. Fusing the layer into a single dispatch removed the repeated cost, one call per transformer layer instead of several. It is the difference between an engine that works and an engine that is fast.
INT8 GEMM
The third was INT8 GEMM. Moving the matrix multiply to 8-bit integer math changed the cost of the part of the pipeline that was actually busy. It was the last lever, and it was the one that put the sprint over the line.
The record
The throughput figures in this post are historical and unsourced. The 1,930 ms/tok start, the 244 ms/tok baseline, the 3.4 ms/tok end, and the 72x speedup were recorded during the sprint before we had a reproducible harness, and all of them are quarantined in benchmarks/latest.json _unverified. The techniques are real. The numbers are not yet evidence.
A four day sprint that goes 72x is only worth posting if the numbers survive the harness. The engine ships today with the batched, fused, INT8 path this sprint produced, and it ships with the same discipline that keeps these figures quarantined until they can be re-measured. The sprint is the record, not the claim.