Qwen3.8-Flash-Next-W4A16-Merlin

Three full 262K sessions resident at once on four RTX 3090s. Qwen3.8-Flash-Next with an 806,792-token KV pool (three full 262,144-token sessions measured resident at once on v2; v2.2 re-tested at 3 × 256K), MTP speculative decoding and vision on consumer Ampere (sm_86, 24 GB each, PCIe, no NVLink bridge), with thinking-on decode at parity with the single-session TP4 build and prefill 10–26 % faster, served through the engine in the companion repository halt95/qwen38-flash-next-3090s. v2 (2026-09-17) changes the serving shape, not the checkpoint: tensor parallel 2 × pipeline parallel 2 with expert parallel, the FP8 n-gram table served to the GPUs over a host-mapped, fail-closed pull transport, and the engine's token-embedding copies for both the target model and the MTP drafter moved out of VRAM into pinned host memory, read by a device-mapped lookup inside the captured cudagraph. v2.2.0 (2026-09-25): rebased on vLLM 0.30.0, requires the new runtime. The same weights and the same serving shape, on an engine rebased from a 0.28-era nightly onto vLLM 0.30.0 (the public v0.30.0 tag plus 58 commits; What changed in v2.2.0). It is a reliability release. With four determinism switches on, prefix caching off and thinking off, greedy (T=0) output repeated request to request in every sampled cell on release candidates, within one compile cache. A fail-closed fault on any GPU rank now stops the engine (workers gone in 3.0–3.5 s in the injection tests) instead of hanging it. A rare illegal-address engine death is contained, not proven absent: ring rows are validated, and a mismatch stops the engine with a named cause instead of faulting. Orderly shutdown is faster (measured on release candidates before the final tree). Decode is at parity with v2: 0.97–1.04 of the v2 gate medians at 4K–261K in both thinking modes (v2.2.0: one release-candidate boot; v2: five-boot gate medians). Builds for v2.0.x do not run it: it needs the vLLM 0.30.0 runtime, its own compiled extensions and a fresh compile cache. v2.0.1 (2026-09-18) is the v2 shape plus one engine fix, structured output under concurrency (What changed in v2.0.1, below). The v1 build (TP4, 342,912-token pool) still serves this checkpoint and stays in the repository.

What is in the checkpoint

This is an Ampere build. Stock vLLM 0.28 could not serve Flash-Next with an FP8 KV cache on sm_86 (Triton there has no fp8e4nv type, so the FP8 read path here is a custom integer-decode kernel); stock vLLM 0.30 was not tested with this checkpoint. The model's n-gram embedding table does not fit on 24 GB cards: it is 102 GB in Intel's BF16 release and ~48 GiB in the FP8 form used here, so it has to live in host memory. vLLM 0.30.0 includes an upstream UVA-based PLE offload (#54371). This checkpoint was qualified on the companion repository's fail-closed host-mapped pull transport instead, which builds on the community PLE-offload work of #53899 (closed without merging; its author pointed to #54371). The checkpoint was qualified only on that patched vLLM, published with pinned build scripts in the companion repository. If your cards have native FP8 and enough memory for the BF16 table, you likely do not need this checkpoint; Intel's release is the natural starting point.

component	format	source
routed experts (512 per layer)	INT4 g128 symmetric, compressed-tensors	Intel/Qwen3.8-Flash-Next-W4A16-AutoRound, converted losslessly from auto_round:auto_gptq
MTP draft-head experts (512)	INT4 g128 symmetric, packed in place	ours (packer adapted from DominikBucko/qwen38-flash-next-2x3090)
Gated DeltaNet projections in_proj_qkv, in_proj_z, out_proj	INT8 per-channel symmetric	ours; in_proj_b, in_proj_a and conv1d stay BF16
n-gram (PLE) embedding table, 51B parameters	FP8 with a global scale, CPU-offloaded at serve time	RadixArk/Qwen3.8-Flash-Next-NVFP4 (its FP8 table replaces Intel's BF16 shard)
the 12 attention layers of the main model, the draft head's attention, norms, token embeddings, lm_head, shared expert, vision tower, indexer	BF16, unchanged	Intel / Qwen
KV cache at serve time	FP8 E4M3 with calibrated static scales for the 12 main-model attention layers (qsa_kv_scales_262k.json)	ours

The shard files are numbered with gaps (model-00002-of-00017 and model-00016-of-00017 do not exist); the index references only the 27 weight files that do, so a download that lists 25 model-* shards (15 model-000NN-of-00017 and 10 model-plefp8-*) plus model_extra_tensors and mtp-routed-experts-int4 is complete.

The v2 delta is one key in config.json, inside the text_config object: "ple_embedding_dtype": "float8_e4m3fn", which is how the v2 engine selects the FP8 n-gram path (the v1 engine used the VLLM_PLE_FP8_GLOBAL_SCALE=1 environment variable and reads the key too). Every weight file is byte-identical to the 2026-09-08 upload. If your copy of config.json predates the key, scripts/make-e1-config.py in the companion repository inserts it (in text_config; a top-level copy of the key is ignored by the engine) without touching anything else. This repository's config.json carries the key since the v2 refresh.

No VnimanieAI tensor is in this checkpoint; that lineage is the previous checkpoint it replaced and the baseline in the quality table below.

Attention is left in BF16 on purpose: only 12 of the main model's layers are true attention, every token passes through all of them, and their K/V projections feed a cache that is already FP8. Packing them to INT4 would save about 0.37 GiB per card; whether it costs quality was not tested, so it was not done.

What changed in v2.2.0

Rebased on vLLM 0.30.0: requires the new runtime. The engine moves from a 0.28-era nightly to vLLM 0.30.0 (qualified with Python 3.13, torch 2.13.0+cu130, triton 3.7.1, flashinfer 0.6.18.post1). The two compiled extensions this release changes (the top-k and Marlin MoE kernels) are built for sm_86 from the published source; their device code matches the qualified builds for every kernel on this model's path (identified by kernel name), and they load on glibc 2.34 or newer. The first serve compiles FlashInfer and Triton kernels, so the host needs a C/C++ compiler, ninja, the Python 3.13 headers, glibc 2.34 or newer and a driver with CUDA 13.0 or newer (the build pins the CUDA 13.0 nvcc wheels). The checkpoint, config.json and the KV-scale sidecar are unchanged.

• Greedy repeatability. Four switches (MERLIN_FULL_K, MERLIN_QSA_SORT, MERLIN_TIE_DET, MERLIN_TIE_RECENT) remove three sources of run-to-run variation: the MoE split-K reduction order, the sparse-indexer top-k output order, and tie-breaking at the top-k cutoff. On release candidates, with prefix caching off and thinking off, eight identical T=0 requests gave one distinct output in every sampled cell from 3,960 to 261,802 prompt tokens; with them off, the deep cells gave 4 to 8 of 8. They cost 1.2–2.2 % median step time at 65K–262K. An in-kernel sorted emission (MERLIN_TOPK_SORTED_EMIT=1, output-identical to the Python sort it replaces) gives 1.9–3.0 % lower median step time than that sort (one boot per arm). These results are sampled and hold within one compile cache (see Known behaviours).
• Fail-closed on any rank. A fail-closed PLE fault on a non-zero tensor-parallel rank used to hang the engine: the dead worker's process sentinel was held open by an inherited helper process, so its death went unnoticed. It was present in v2.0.x too, and was never seen before because no earlier test hit a non-zero rank. Now each worker has a fault pipe that signals before exit, exits are also watched by pidfd, and the inherited pipes are close-on-exec. With an injected fault on rank 1, detection is immediate and all workers are gone in 3.0–3.5 s, with the GPUs free in 3.8–4.6 s. The client gets HTTP 500. This covers fail-closed faults and worker exits, not a GPU that hangs without exiting.
• Rare illegal-address engine death: contained, not proven absent. Each sparse-attention raw-key ring row carries a tag that is checked before the row is used, so a stale or uncommitted row is never read as a key or a position. A mismatch is counted at an always-fatal guard site: with VLLM_E2_GUARD_MODE=warn (the shipped serve script) the engine stops with a named cause instead of an illegal-address fault; with the guard reader off (the code default) the row is masked and counted but nothing reads the counter. A seeded replay faults on the old engine and is caught on the new one; the check did not fire in the release candidates' 90-minute soak or stress runs, but the original fault was seen only twice in the whole campaign, so no rate is claimed. The KV pool is unchanged.
• Faster orderly shutdown. Drain-order fixes shorten it to 2.65–3.55 s, down from about 9 s; a late SIGTERM during teardown is survived, and a failure that races shutdown still exits 1 (measured on release candidates before the final tree; the final tree also changes the fault-detection path and parts of the same teardown code).
• Guard telemetry: serve with VLLM_E2_GUARD_MODE=warn (the shipped serve-v2.2.sh default). Guard hits are then reported, and the designated fatal sites (among them the ring-row check) stop the engine with a named cause instead of masking the row silently.

What changed in v2.0.1 (structured output under concurrency)

Structured output now works under concurrency. In v2 a response_format request could fail with HTTP 500 while other requests were decoding: the draft hand-off kept one unidentified slot that the alternating pipeline-parallel microbatch overwrote, the -1 placeholders left behind gave the grammar bitmask an all-allowed row, and the unconstrained token then failed the state machine. v2.0.1 ports two upstream pull requests, both open upstream at the time of writing: PR #54442 (never leave an unmasked row for a draft slot the scheduler did not schedule) and PR #56802 (key draft snapshots by scheduler step, so a request is verified against the drafts its own step consumed).

Measured with the shipped serve command, 80 structured requests over four load cells, 10 idle and 10 under load each:

Concurrent decodes	Thinking	v2: schema-valid responses of 10, under load	v2.0.1: same
1	on	8/10	10/10
2	on	1/10	10/10
3	on	2/10	10/10
2	off	6/10	10/10

Idle was 10/10 in every cell on both; every v2.0.1 response body validated against its schema, and the run logged no grammar rejection and no terminated request. No throughput regression was found: on the reference host's two-boot ladder across 4K, 32K, 131K and 261K in both thinking modes v2.0.1 lands between 0.978 and 1.028 of the v2 median, median inter-token latency within 0.21 ms, tokens per step unchanged. The prefix-cache equivalence block fails identically on v2 and v2.0.1 (open behaviour, below), unchanged by the fix. The checkpoint, the serve command and the KV budget are untouched — this is an engine fix, and the weights are byte-identical. These figures are maintainer-reported; their record is not published, and the two upstream pull requests are the independently readable evidence.

Serving

v2.2.0 (recommended: vLLM 0.30 runtime, 806K pool, three 262K sessions)

Quick start with the container, the Proxmox LXC recipe, a health check and troubleshooting: the companion repository's README. The steps below are the bare-metal route.

From the companion repository at release v2.2.0. That release pins its vLLM source (the head of the bundle's branch v2.2.0-public) = commit 9c27ca9a0657338852c7d3e59561d424fc036fde, tree eebcd6480dba463294e9ca34f24eb770a45e11fe, on top of upstream vLLM v0.30.0 (ced6857afa0ea7b2e3f0846a62e1394e90f15607).

1. Build with scripts/build-v2.2.sh: download v2.2.0-from-upstream-v0.30.0.bundle.gz and build-artifacts-sm86-py313-cu130-v2.2.0.tar.gz from the v2.2.0 release first (the README's build block shows the commands). The build takes upstream v0.30.0 plus the v2.2.0 delta (the git bundle, verified by commit and tree hash), a fresh Python 3.13 venv on the vLLM 0.30.0 runtime, and the two sm_86 extensions this release changes (_C_stable_libtorch, _moe_C_stable_libtorch), taken from the release's build-artifacts asset (the default, pinned by sha256) or built from source with BUILD_OWN=1 (needs cmake 3.26 or newer, ninja and a C/C++ compiler; it compiles with the venv's CUDA 13.0 nvcc). It also installs the CUDA 13.0 nvcc wheels the first serve compiles kernels with.
2. Add the config key if your copy of config.json predates it: python3 scripts/make-e1-config.py /path/to/Qwen3.8-Flash-Next-W4A16-Merlin (a no-op otherwise; serve-v2.2.sh refuses to start without the key).
3. Serve with scripts/serve-v2.2.sh /path/to/Qwen3.8-Flash-Next-W4A16-Merlin. Same shape and flags as v2.0.1 (below), with these environment changes:
   • VLLM_E2_PLE_PULL_TRANSPORT=1 replaces VLLM_PLE_CPU_OFFLOAD=1. vLLM 0.30 uses that old name for its own pinned-host PLE path with the opposite default; the engine refuses a configuration that turns both on.
   • VLLM_USE_BREAKABLE_CUDAGRAPH=0 (exactly 0).
   • MERLIN_FULL_K=1 MERLIN_QSA_SORT=1 MERLIN_TIE_DET=1 MERLIN_TIE_RECENT=1 MERLIN_TOPK_SORTED_EMIT=1. These are compile-cache factors: change them and the next boot recompiles.
   • VLLM_E2_GUARD_MODE=warn.
   • CUDA_HOME pointing at a toolkit no newer than the driver (default: the venv's CUDA 13.0 wheels), plus a C/C++ compiler, ninja and the Python 3.13 headers on the host: the script checks them before starting.
   • The two logging-only counters are off by default (COUNTERS=1 turns them on; the published numbers were measured with them on).
4. Use a new VLLM_CACHE_ROOT. v2.0.x compile artefacts do not carry over. The first boot compiles (about 6 minutes); later boots load the cache (about 3 minutes).

Thinking is on by default, so a short max_tokens can end inside the reasoning with empty content; send "chat_template_kwargs":{"enable_thinking":false} or a larger max_tokens. The endpoint has no API key unless you set VLLM_API_KEY. A healthy boot logs GPU KV cache size: 806,792 tokens; /v1/models answers even when the engine is dead, so check health with a real completion.

Settings that are load-bearing on v2 (the v2.0.1 settings; they apply unchanged to v2.2.0):

• Host RAM: 96 GiB (98,304 MiB) is the qualified allocation. The ~48 GiB FP8 n-gram table and about 4.2 GiB of pinned embedding tables (1,064 MiB per rank) live there; the measured resident floor is about 69 GiB, the rest being reclaimable page cache, and the boot peak is not measured. /dev/shm needs at least 1 GB (a Bus error at boot means it is undersized).
• --kv-cache-memory 4100000000 is the highest pin that boots with a complete clean row set on 24 GB cards; the non-KV headroom that remains is about 1 GB per card. The 25/23 layer partition is part of that budget.
• The four cards must be headless, with no other CUDA process on them: the pool is pinned in bytes, so a display server or another process on one card can stop the boot or exhaust that headroom.
• --max-num-batched-tokens stays at 1024 on this shape. The deep-prefill transients that engine-killed 2048 and 4096 on earlier configurations sit outside the torch pool.
• Capture sizes cover max-num-seqs × (K+1) = 32.
• VLLM_HOST_EMBED_TABLE=1 is on by default and is a registered torch compile-cache factor; turning it off costs the 608 MiB per card back and a fresh compile.
• Do not set PYTORCH_CUDA_ALLOC_CONF=expandable_segments (v1 measured it at −27 % four-stream aggregate).
• Clients retry once on an empty completion (Known behaviours).
• CUDACachingAllocator ... OOM warning lines can appear when several long sessions grow at once: the upstream sparse-attention indexer's prefill logits buffer (1 KiB per token of context, 268 MB at 262K) asked for a contiguous block, the allocator flushed its cache and the retry succeeded. Warnings, not failures (maintainer-observed: 22 lines, 0 failed requests in a three-session run); a failing retry would mean the ~1 GB non-KV headroom per card is exhausted.

v2.0.1 (previous: 806K pool, three 262K sessions, vLLM 0.28-era base)

From the companion repository at release v2.0.1. That release pins vLLM fork tag v2.0.1 = commit ad5c3c223999de577b04cdb9caeab2dcb76b61b9, tree 5426cf19586a66946a1d8f7e68f48bdc7f52d62d (upstream/PIN-v2); scripts/build-v2.sh asserts both the commit and the tree after checkout, so a build that does not match the pin fails rather than serving something else.

1. Build with scripts/build-v2.sh: the pinned upstream vLLM commit plus the community Flash-Next commits and the v2 delta (shipped as a git bundle release asset, verified by commit and tree hash), a fresh Python 3.13 venv from 199 pinned packages, compiled ops from the upstream precompiled cu130 wheel pinned by sha256 (no C++ change in v2). The first serve still JIT-compiles FlashInfer and Triton kernels, which needs a CUDA toolkit no newer than the driver, a C compiler, ninja and the Python headers (see v2.2.0 above).
2. Add the config key if your download predates it: python3 scripts/make-e1-config.py /path/to/Qwen3.8-Flash-Next-W4A16-Merlin.
3. Serve with scripts/serve-v2.sh /path/to/Qwen3.8-Flash-Next-W4A16-Merlin. It is the served entry with every variable exported; the shape it runs is TP2 × PP2 + EP (VLLM_PP_LAYER_PARTITION=25,23), MTP K=3 probabilistic, --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 1024, --kv-cache-memory 4100000000 --kv-cache-dtype fp8_e4m3 with the sidecar, FULL_AND_PIECEWISE cudagraphs with captures to 32, prefix caching, the PLE pull transport (VLLM_PLE_CPU_OFFLOAD=1, home GPU 0) and the host-resident embeddings (VLLM_HOST_EMBED_TABLE=1). The exact command and environment are in the script and in the bench card.

The load-bearing settings above apply to it.

v1 (TP4, 342,912-token pool)

scripts/build.sh and scripts/serve.sh in the same repository: vLLM v0.28.0, the eight community Flash-Next commits, six patches, the compiled-ops wheel pinned by sha256 and the sm_86 GDN decode kernel rebuilt (needs CUDA 13 and gcc ≥ 11.3). Serve shape: TP4 + EP, MTP K=3, captures [1,4,8], --max-num-seqs 2, --max-num-batched-tokens 2048, --kv-cache-memory 2600000000 as shipped in serve.sh (2.9e9 = pool 342,912 tokens on the reference host since 2026-09-10; the v2 gate's v1 arm ran at 2.9e9), VLLM_PLE_FP8_GLOBAL_SCALE=1, VLLM_PLE_OFFLOAD_HOME_DEVICE=3. The 2026-09-08 card's load-bearing notes still apply to it: the capture list must cover max-num-seqs × (K+1) (a [1,3] list once collapsed two-stream throughput to 18 tok/s per stream), and 4096 batched tokens peaked at 24.1 of 24 GiB at 262K.

Measured

Hardware for everything below: 4× RTX 3090 at 220 W, Gen4 x16 to every card, peer-to-peer over the open-kernel-module P2P patch, an EPYC 7532 with 192 GB ECC of which the serving container is allocated 96 GB. Four runs are reported, and they are not the same configuration; the three older ones are folded below the first.

Serving: v2.2.0 requalification (2026-09-24), same shape, determinism switches on, thinking on and off

One boot of the v2.2.0 release candidate (kernels and model code as released; the final release adds only process-supervision fixes), measured against the v2 gate's five-boot medians below; no confidence interval is claimed. Single-stream decode, T=0 with a seed, 3 sends per cell (the served entry defaults to T=1.0 / top-p 0.95 / top-k 20; these numbers were not re-measured under those defaults):

prompt tokens	thinking	v2.2.0 tok/s	v2 gate tok/s	ratio	median event interval, ms (v2.2.0 / v2)	MTP tokens per step (v2.2.0 / v2)
4,096	on	160.0	162.2	0.986	18.98 / 18.72	3.02 / 3.02
4,096	off	119.7	118.2	1.013	19.12 / 18.66	2.30 / 2.21
32,768	on	165.6	166.3	0.996	19.05 / 18.73	3.12 / 3.07
32,768	off	127.9	123.0	1.040	18.98 / 18.81	2.43 / 2.31
131,072	on	168.9	169.4	0.997	19.50 / 19.10	3.19 / 3.13
131,072	off	121.3	123.3	0.984	19.25 / 18.97	2.31 / 2.35
261,120	on	175.1	176.2	0.994	19.54 / 19.33	3.32 / 3.28
261,120	off	121.8	125.2	0.973	19.55 / 19.15	2.35 / 2.37

The event interval is 0.9–2.5 % longer in every cell, a combined difference of the determinism path, the 0.30 base and the instrumentation (v2.2.0 was measured with the guard reader in warn and both logging counters on, v2 with those readers off; in the v2 campaign they cost ~0.3–0.5 ms per step). Throughput lands at 0.97–1.04 of v2; tokens per step are equal or higher in six of the eight cells. Eight-stream aggregate is 530.3 tok/s (v2.0.x: 526.6) and single-stream aggregate 118.9 (118.1). GSM8K-200 thinking off is 198/200 on both. Divergence from the BF16 teacher (defined under the v2 gate table below) on the same 24 held-out prompts is 0.0338: one boot, within the check's preset ceiling (0.0388), and not a gain, since one boot is not comparable with the spread of the gate's five boots (0.0365–0.0378). Also in the release candidate's requalification run:

• a 90-minute soak with two ~200K sessions and an 8-way load: 12,115 requests, 0 errors;
• three 256,000-token sessions resident at once (v2 held three full 262,144-token sessions);
• the KV pool driven to 100 % with preemptions and no failed request;
• 80/80 structured-output requests.

Over context depth on the final v2.2.0 tree (one boot, the reference host's serve environment with the logging counters on): cold prefill 5,123–5,410 tok/s from 10K to 200K tokens; decode with thinking on (reasoning_effort low, 512 tokens) 153.5–186.7 tok/s (median of three different prompts per depth), at a median event interval of 18.8 ms at 10K rising to 19.4 ms at 200K. Most of the decode spread tracks how many drafted tokens each text accepts rather than depth. Chart and per-depth table: benchmarks/2026-09-24/BENCH-CARD.md.

These are maintainer measurements on the reference host; the records are not published.

Serving: the v2 gate (2026-09-17), TP2 × PP2 + EP, MTP K=3

Numbers from the pre-registered two-arm gate that qualified the v2 tag: five boots of v2 alternating with five boots of the v1 TP4 build on the same box, one frozen manifest, three streamed sends per cell, T=0 with a fixed seed, judged as medians (benchmarks/2026-09-17/BENCH-CARD.md in the companion repository, every boot's numbers in it).

	v2	v1 TP4 build (same gate, same box; † see the note under the table)
shape	TP2 × PP2 + expert parallel, MTP K=3	TP4 + EP, MTP K=3
KV pool, FP8 tokens, whole box	806,792 (3.08 × a full-context request; three full 262,144-token sessions resident, measured at a peak of about 95 % of the pool)	342,912 (1.31 ×)
context per request	262,144	262,144
concurrent sequences admitted (--max-num-seqs, configured; the shapes listed were measured resident)	8 (3 × 262K, 1 × 262K + 3 × 131K, 5 × 131K, 8 × 65K, 8 × 32K)	2
decode, single stream, thinking on, 4K / 32K / 131K / 261K prompt	162 / 166 / 169 / 176 tok/s (medians of 5 boots)	163 / 165 / 172 / 170†
decode, thinking off, same depths	118 / 123 / 123 / 125	124 / 122 / 120 / 125†
median event interval, thinking on, same depths	18.72 / 18.73 / 19.10 / 19.33 ms	18.61 / 18.73 / 19.17 / 19.37† ms
MTP tokens per step, thinking on, same depths	3.02 / 3.07 / 3.13 / 3.28	3.00 / 3.04 / 3.17 / 3.24†
prefill, 10K / 100K / 261K prompt	4,944 / 5,282 / 5,122 tok/s	4,498 / 4,194 / —
time to first token, cold, 4K / 32K / 131K / 261K	0.91 / 6.19 / 24.77 / 50.68 s	0.97 / 7.16 / 31.94 / — s
quality vs the BF16 teacher (24 held-out prompts, 5 boots; lower is closer; defined below)	0.0365–0.0378	0.0378–0.0385
tool-call structure (150 cases) / exact recall (160 cases)	130/150 / 160/160	reference quant 125/150 / 160/160 (parity band, ≥ ref − 2)
vision	on, 2 images per request	on

† v1 at 261K was measured on 2026-09-18, after the gate, in one boot with three sends (170.1 / 124.6 tok/s; median event interval 19.37 / 19.40 ms; 3.24 / 2.39 tokens per step, thinking on / off). Every other v1 cell is a median of five boots; the 261K depth was preregistered for the v2 arm only, so v1 prefill and time to first token at that depth are not measured.

Quality vs the BF16 teacher. The teacher is a reference run of this same checkpoint with a BF16 KV cache, eager and without speculative decoding, captured once on 24 held-out prompts. Each served boot is scored teacher-forced on the teacher's own continuations: the mean absolute difference in per-token log-probability over all 1,476 positions of the 24 prompts, averaged over 4 captures per boot; lower is closer. It measures what the serving configuration adds (FP8 KV cache, compiled graphs, parallel layout, engine patches), not the loss from quantising the original model. The gate passed v2 if the one-sided 95 % upper bound (Welch) of the mean v2-minus-v1 difference stayed at or below 0.0015, one within-boot standard deviation of the reference captures, fixed in advance; the measured difference was −0.0009.

Decode is the streamed event rate over the answer, client-timed. With MTP the honest pair is the median event interval (v2 18.7 / 18.7 / 19.1 ms against v1 18.6 / 18.7 / 19.2 at 4K / 32K / 131K) and tokens per step, both in the card. One cell of the gate was recorded as a shortfall: thinking-off decode at 4K read 0.92–0.99 of the v1 build on every one of the five boots, at an identical event interval (fewer accepted draft tokens per step); the cause was not established in the gate. A later interleaved re-test (2026-09-23; 15 boots: the v1 build, and the v2.2 release candidate with the determinism switches on and off, five each) put the same cell at 1.011 of the v1 build with the switches on (0.987 off), with tokens per step equal (1.007 [0.994, 1.021]). That is consistent with a content draw (at T=0 a near-tie at the first prose tokens decides which of a few equally valid phrasings the answer takes, and the drafter predicts some better than others) rather than an engine defect; the v2 shortfall itself was not re-run. The served entry defaults to T=1.0 / top-p 0.95 / top-k 20; the gate numbers were not re-measured under those defaults.

Serving: the v1 card (2026-09-08), TP4 + EP, MTP K=3

Captures [1,4,8], the shipped sidecar, through the front door, thinking on, an image in every depth prompt (benchmarks/2026-09-08/BENCH-CARD.md; 2 clip-counter lines over the run, zero errors): single-stream decode 166 / 174 / 172 / 163 tok/s at 4K / 32K / 131K / 260K prompt tokens, time to first token 0.9 / 7.0 / 32.0 / 75.7 s at the same depths, deepest measured prompt 260,566 tokens, MTP acceptance 2.43 in the card's logging window (vLLM's mean acceptance length under served sampling; not comparable with the tokens-per-step figures above) (2.4–2.9 across served logging windows, maintainer-reported). End to end, a cold 260K prompt is dominated by its prefill: about 76 s on the v1 build and 51 s on v2 (the v2 gate table).

Quality: paired per question (2026-09-03), v1 build, MTP K=2

A different instrument from the gate above, on a different serving shape (captures [1,3,6]). The baseline is the checkpoint this one replaced: the same MTP and GDN packs applied to the VnimanieAI W4A16 checkpoint, whose attention projections are INT4 where Intel's are BF16.

quality (paired)	previous	this checkpoint	delta
GSM8K-1319, thinking off	95.68 %	96.44 %	+0.76 [−0.08, +1.60], p = 0.11
ARC-Challenge-1172	96.93 %	97.27 %	+0.34 [−0.19, +0.87], p = 0.34
MMLU (1,000, seed 0)	86.60 %	87.30 %	+0.70 [−0.81, +2.21], p = 0.44
verbosity at xhigh reasoning effort, 40 hard MMLU questions, 32K budget (median tokens)	3,841	2,126	−45 %
250K daily check (needle at 259K, vision ×2)	4/4	4/4

Read those honestly. None of the three task deltas reaches significance at these sizes. An earlier comparison (2026-09-02, the unpacked Intel graft at a different serving shape, so not this exact checkpoint) had the same direction and was significant on GSM8K and ARC; the two runs share a baseline lineage and are not independent replications. The verbosity result is one run of 40 questions at xhigh (accuracy 32/40 vs 31/40); at the served low effort verbosity was unchanged (median 1,054 vs 1,059) and accuracy read 32/40 vs 35/40. In that K=2 gate this checkpoint decoded about 5 % slower than the previous one at short context and equal at depth. The v2 gate's quality row (divergence from the BF16 teacher on 24 held-out prompts, above) is a different instrument and is not comparable to this table. An earlier single-capture quality screen of the v2 campaign is recorded as a FAIL and stays cited as such in the companion repository; the two-arm gate above replaced it as the instrument.

Known behaviours of the Qwen3.8-Flash-Next architecture in vLLM

These are behaviours of the architecture's serving path in vLLM (the hybrid Gated-DeltaNet / sparse-attention / MTP execution path and its hybrid KV manager). None is introduced by this checkpoint's quantisation; whether the prefix-cache loss is upstream's or the engine patches' is not established. Each is documented in the companion repository, and none produced a wrong answer in the qualification runs (the empty completion returns no answer at all; retry it). Read them before serving. They apply to v2.2.0 unless marked.

• An occasional empty completion on a warm repeat of a long cached prefix (upstream class, vllm-project/vllm #53912): HTTP 200, finish_reason: "stop", zero tokens. Not reproduced on the v2.2 release candidate (0 of 200 warm requests over five boots without preemption; a request-level 95 % upper bound of 1.9 %, Wilson score), but the mechanism is not identified, so it is not claimed fixed. Clients retry once; that returned the correct answer in every observed case (a mitigation, not a guarantee).
• Prefix-cache blocks of a session that finishes while other long sessions are still decoding can be dropped (in the hybrid KV manager path). Eviction under KV-pool pressure is not excluded, but the mechanism is not established. Cost: first-token latency on that session's next turn; the answer is unaffected. Prefix-cache hits land in 3,200-token aligned blocks, so prompts under about 6.4K tokens cannot hit.
• Greedy T=0 repeats only within one compile cache. With the four determinism switches on, repeated T=0 requests gave identical output in every sampled cell (sampled, not a batch-invariance guarantee). Two fresh compiles of the same engine can still differ: the six prefill kernels of the Gated-DeltaNet path autotune, and their near-tied picks, cached per compile directory, differed between two fresh caches; Inductor's reduction autotuning is a second candidate, and which kernels account for the difference is not fully attributed. Keep one VLLM_CACHE_ROOT per deployment if byte-identical repeats matter. Any numeric change (parallel layout, kernel build, version) can also flip near-tied choices to a different, equally valid phrasing, which moves MTP acceptance a few percent per prompt.
• Guard telemetry defaults to off in code. Serve with VLLM_E2_GUARD_MODE=warn so that guard hits are reported and the designated fatal sites stop the engine with a named cause rather than masking a row silently.
• Contained in v2.2.0: the rare illegal-address engine death on the sparse-attention path (ring rows are now tagged and validated; with VLLM_E2_GUARD_MODE=warn a mismatch stops the engine with a named cause; not proven absent). Fixed in v2.2.0: a hang when a non-zero GPU rank hit a fatal fault (see What changed in v2.2.0). On v2.0.x both remain as described in the v2.0.1 notes.

The KV-scale sidecar

qsa_kv_scales_262k.json holds static FP8 scales for the 12 attention layers of the main model. It is the same file for v1 and v2 (byte-identical). It was calibrated on this checkpoint on 2026-09-08: eager (no CUDA graphs), no speculative decoding, text at 1,971 / 7,989 / 31,975 / 127,805 / 199,888 / 254,875 prompt tokens plus image prompts at 2,848 and 8,841 tokens, merged as the maximum over the four TP ranks with a 10 % margin. The rank dumps and the merge script ship with the companion repository and reproduce this file byte for byte.

Gated at the v1 served shape (MTP K=3, full CUDA graphs, 262K, prefix caching on), each arm a fresh server: daily lane check 4/4 (text, one image, two images, exact needle at 207,495 tokens); 24 clip-counter increments on 3 layers during the gate, against 496 on 9 layers for the earlier sidecar calibrated on the previous checkpoint. Served by a build made from the public repository with the final v1 patch revision (strict loader, float32 clip ceilings), the same file read 0 increments and 4/4. Without a sidecar the FP8 path runs at scale 1.0. The draft head's own attention layer is not in the sidecar and runs at scale 1.0. The v2 gate ran 10 boots with the clip counter compiled in and off; the sidecar's effect on v2 decode was not measured separately.

Limitations

• Requires the patched vLLM above (the companion repository's v2.2.0 release on the vLLM 0.30.0 runtime, the v2.0.1 tag, or the v1 build). Upstream vLLM merged Flash-Next model support on 2026-08-31 (#53896), and vLLM 0.30.0 adds its own UVA-based PLE offload (#54371); whether a stock build serves this checkpoint (its compressed-tensors config, the FP8 n-gram table, FP8 KV on sm_86) was not tested.
• The v2 evidence boundary: the gate is a maintainer measurement on the reference host with a frozen, hash-pinned manifest; the bench card is published in the companion repository, the underlying close records, raw streams and boot logs are not. The v2.0.1 requalification figures (in What changed in v2.0.1) are maintainer-reported and their record is not published (the two upstream pull requests that fix ports are public and can be read independently); the same holds for the v2.2.0 requalification figures. What you can reproduce independently: the source tree (commit and tree hash), the environment, the sidecar and the serve command. The v2 build was reproduced from the bundle on the reference host; no third-party reproduction is documented.
• Quality gains over the previous checkpoint are directionally consistent but individually non-significant; the clearer effect is verbosity at high reasoning effort, from one 40-question run.
• The empty warm completion and the prefix-cache loss remain open (documented, not fixed). Greedy repeatability is sampled and per compile cache. The illegal-address engine death is contained in v2.2.0 (a named engine stop instead of a fault; not proven absent), the non-zero-rank fault hang is fixed in v2.2.0, and structured output under concurrency is fixed since v2.0.1.
• Qualified on one hardware configuration (4× RTX 3090, 220 W, Gen4 x16, P2P enabled). It also runs without peer-to-peer (a stock driver): checked on a rented 4× RTX 3090 with correct answers and lower prefill (about 4,000 t/s at 55K tokens against 5,410 at 50K; that host also had a PCIe x8 link per card and two NUMA nodes).
• Attention, norms, embeddings and lm_head are BF16, so this is not a maximally compressed checkpoint.

Lineage and credit

• Qwen for Qwen3.8-Flash-Next and the licence (Qwen Community License 1.0, vendored here as LICENSE; its notice applies to these weights as a derivative work).
• Intel for the AutoRound expert quantisation; the maintainer's reading is that the expert rounding is where the measured quality difference comes from, though attention format also differs between the two compared checkpoints.
• RadixArk for the FP8 PLE table.
• DominikBucko for the MTP INT4 packing recipe this one adapts.
• The community PLE-offload patch stack (peakcrosser7, vLLM PRs #53896 / #53899; #53899 was closed unmerged, and upstream now carries a UVA PLE offload, #54371) that makes Flash-Next serveable on Ampere, and the upstream fixes the engine carries or re-ports (credited by PR and author in the companion repository's NOTICE).
• aikitoria for the open-kernel-module patch that enables peer-to-peer on consumer Ampere.
• alesha-pro, whose benchmark metric definitions the v1 cards' instrument ports.
• The v2 host-mapped pull transport and the host-resident target and drafter embeddings inside the cudagraph are the maintainer's own work.