jesdga/Qwen3.8-Flash-Next-Basalt

The fastest inference engine for Qwen3.8-Flash-Next on Blackwell

Website: basalt.jgulfo.com

Model files for Basalt, a local inference engine for the Qwen3.8-Flash-Next mixture-of-experts model. Basalt runs on Linux with any NVIDIA Blackwell GPU (sm_120). A second GPU can serve as a peer expert tier, and experts that do not fit in VRAM stay in system RAM. It has an OpenAI- and Anthropic-compatible API.

Each .basalt file is self-contained: weights, tokenizer, MTP drafter, vision projector and the int8 hyper-connection table the engine reads at start. The expert profile (which experts start in VRAM) ships with the engine. Expert formats: Unsloth UD quants (IQ1_S, IQ1_M, IQ2_XS, IQ3_XXS, IQ4_XS, Q4_K_XL, Q5_K_XL, Q6_K_XL), Q8_0 and NVIDIA NVFP4.

Files: Qwen3.8-FN-NVFP4-DQ6_K, -NVFP4-DQ8, -UD-IQ1_S, -UD-IQ1_M, -UD-IQ2_XS, -UD-IQ3_XXS, -UD-IQ4_XS, -UD-Q4_K_XL, -UD-Q5_K_XL, -UD-Q6_K_XL and -Q8_0 (each .basalt), plus convert/.

These are .basalt format version 2 files, which Basalt 0.2.0 and later require. Version 2 drops engine policy data from the pack (the expert profile now ships with the engine, so a better one never needs a new download) and stores the int8 hyper-connection table, so the engine no longer rebuilds it at every start. Weights are unchanged and output is identical. Already have a version 1 file? Re-downloading is optional: python3 tools/migrations/basalt_v1_to_v2.py OLD.basalt NEW.basalt converts it locally without re-quantizing (needs numpy and room for a second copy). See the file format changelog.

Speed

Best measured, single stream, RTX 5090 at 400 W + RTX 5060 Ti peer at 180 W, tokens/s. Numbers move with every release.

pack	prose	structured	chat
NVFP4-DQ6_K	307	601	335
UD-IQ3_XXS	342	658	399
UD-IQ4_XS	300	608	363
UD-IQ2_XS	345	641	396
UD-IQ1_S	389	763	457
UD-IQ1_M	364	732	434
Q8_0	112	166	101

Quality

Measured with basalt-bench --quality: 100 chunks of wikitext-2 (512 tokens each), scored through the decode path against full precision: a llama.cpp BF16 reference of the original model (reference PPL 3.273 on the scored rows). int8 KV, RTX 5090 at 400 W + RTX 5060 Ti at 180 W. Lower KLD and higher top-1 agreement are closer to full precision. PPL is the pack's own perplexity on the same text; it is not a closeness measure. Only the official Basalt packs are listed.

pack	source	file	experts	PPL	KLD mean	same top-1
UD-IQ1_S	Unsloth	65.6 GiB	31.1 GiB	3.405	0.2676	82.76%
UD-IQ1_M	Unsloth	67.1 GiB	32.4 GiB	3.379	0.2560	83.19%
UD-IQ2_XS	Unsloth	74.4 GiB	39.4 GiB	3.289	0.1829	86.21%
UD-IQ3_XXS	Unsloth	81.6 GiB	46.8 GiB	3.242	0.1188	88.67%
UD-IQ4_XS	Unsloth	90.9 GiB	56.2 GiB	3.231	0.0818	90.52%
UD-Q4_K_XL	Unsloth	106.7 GiB	71.7 GiB	3.321	0.0496	92.63%
NVFP4-DQ6_K	NVIDIA	122.0 GiB	63.3 GiB	3.304	0.0624	91.65%
NVFP4-DQ8	NVIDIA	123.3 GiB	63.3 GiB	3.297	0.0561	92.25%
UD-Q5_K_XL	Unsloth	150.5 GiB	91.6 GiB	3.293	0.0255	94.82%
UD-Q6_K_XL	Unsloth	160.6 GiB	101.7 GiB	3.283	0.0209	95.22%
Q8_0	Unsloth	178.3 GiB	119.5 GiB	3.279	0.0200	95.34%

Both NVFP4 packs keep NVIDIA's NVFP4 experts bit for bit; the head and embedding are Q8_0. The D suffix names the dense projections' format: Qwen3.8-FN-NVFP4-DQ6_K stores them as Q6_K, quantized once from BF16, and Qwen3.8-FN-NVFP4-DQ8 stores them as Q8_0, which removes most of the 6-bit pack's measurable rounding (KLD 0.0059 lower; every rounding removed, BF16 dense included, is 0.003 lower still) and decodes about 7% slower on prose and 3% on structured output. Every rounding the engine does is measured in the precision docs.

The UD-IQ1_S, UD-IQ1_M, UD-IQ2_XS, UD-IQ3_XXS, UD-IQ4_XS and NVFP4 packs carry a self-distilled MTP layer (convert/mtp/rt/dense.bin) for higher draft acceptance; the others keep the original.

Run

Build the engine as described in the Basalt README, then:

hf download jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-NVFP4-DQ6_K.basalt --local-dir models

basalt_config.json:

{
  "version": 2,
  "models": [{ "model": "models/Qwen3.8-FN-NVFP4-DQ6_K.basalt", "gpu": "0,1", "kv": "int8" }],
  "engine": { "resident-experts": true },
  "server": { "port": 8080 }
}

bin/basalt serve --config basalt_config.json

Conversion kit

convert/ holds the inputs for tools/convert/convert.py (MTP layer, vision projector, draft vocabulary), so any GGUF quant of this model can become a .basalt file. See docs/model-file.md.

Sources

• UD quants, Q8_0: unsloth.
• NVFP4 experts: NVIDIA.
• MTP drafter: Qwen/Qwen3.8-Flash-Next BF16 MTP tensors.
• Basalt is a fork of Strata (MIT).

License

All files here are derived from Qwen3.8-Flash-Next and are distributed under the Qwen Community License 1.0, whose notice is included in this repository and inside every .basalt file. Businesses offering Model as a Service or an AI coding/office assistant need a separate license from Qwen for commercial use, and products above 100M monthly users or US$20M monthly revenue must display the model name. The LICENSE has the binding terms. The Basalt engine itself is MIT.