Qwen3.5-4B-MTP-ONNX-WebGPU
The multi-token prediction (MTP) layer that Qwen/Qwen3.5-4B ships in its checkpoint
(mtp.*), converted to ONNX for speculative decoding in a web browser with ONNX Runtime Web's WebGPU backend, as used
by Quick Inference. It drafts the next tokens for onnx-community/Qwen3.5-4B-ONNX, which checks them: the
output is exactly the model's own.
What it computes
As the MTP layer of Qwen's checkpoint (vLLM's Qwen3_5MultiTokenPredictor): for a position t, the embedding of the
token at t + 1 and the model's final hidden state at t (after its final norm) are normalised, concatenated and mapped
to the hidden size, then one full-attention decoder layer (gated attention, its own KV cache at the
model's positions), a final norm and the model's output layer give the guess for t + 2. A chain of guesses feeds the
layer's own output hidden state back.
- Inputs:
input_ids[1, n],hidden[1, n, hidden] (float16),position_ids[1, n],last(the row to guess from),seqlens_kandtotal_sequence_length(GroupQueryAttention's),past_key_values.0.key/value(a static cache, updated in place). - Outputs:
token[1, 1] (the guessed token id),hidden_out[1, 1, hidden] (for the next guess),present.0.key/value.
Changes from the original
- The layer's matrices are quantised to 8 bits (round to nearest, block size 32).
- The output layer and the token embedding table are the 4-bit ones of the ONNX model it drafts for, keeping the first 32,768 token ids only: guesses come from those (token ids past them are clamped for the embedding lookup). This makes it several times smaller and faster, for a few percent fewer kept guesses.
- Norm weights include the 1 of Qwen3.5's zero-centred RMSNorm.
These are modified files: their outputs are close to, but not identical to, the original layer's.
Licence
Licensed under the same terms as the original model (Apache 2.0): see LICENSE.