TokenRouter: token-level multi-backend inference
Strix Halo carries both an NPU and a GPU, and using only one leaves half the machine idle. TokenRouter is our open-source answer: route the token stream across both, with the NPU drafting and the GPU verifying, and real logprobs on the wire.
The divide
On AMD Strix Halo, the two engines sit a few millimeters apart and rarely talk. The NPU is good at quick speculative drafts, the GPU is strong at verifying them in bulk. Most software picks one backend and leaves the other idle. TokenRouter does not pick.
Token level, not model level
Routing at the model level means a whole request goes to one backend and stays there. TokenRouter works one token at a time: each step decides where the work happens. That is finer-grained than a request split, and it is the part that took real engineering.
Draft and verify
The pattern is simple to describe and hard to make fast: the NPU produces a draft token, the GPU verifies it. Keeping the two in sync without stalling either side is where the interesting work sits.
Real logprobs on the wire
A router that guesses about likelihood is guessing. TokenRouter carries real logprobs on the wire, computed on the actual backend, so the acceptance decision is made from evidence rather than approximation.
Open source
The routing layer is released open source, like the rest of the stack. This is the kind of code that is only useful if other people can read it, and only trustworthy if they can rebuild it. Read it, run it, and tell us where the numbers disagree.