GLM-5.3-Flash-DFlash2 — EXL3 3.00 bpw (draft)
EXL3 (ExLlamaV3 trellis) quantization by r0b0tlab of the
incoai/GLM-5.3-Flash-DFlash2
block-diffusion speculative-decoding draft, for use with
r0b0tlab/GLM-5.3-Flash-EXL3-2.25bpw-sm121.
- Format: EXL3 3.00 bpw, mul1 codebook, 6-bit head, output scales on,
calibration 250x2048 (
quantization_config.jsoncarries the full module/tensor manifest). - Size: 610 MiB, single
model.safetensorsshard. - Role: DFlash2 K=5 drafter (TabbyAPI
draft_mode: model,draft_num_tokens: 5). Not a standalone language model.
Qualified configuration (single NVIDIA GB10 / DGX Spark, SM121, aarch64)
Runtime: pinned ExLlamaV3 v1.5.2 + TabbyAPI in a native ARM64 CUDA 13 container
(ExLlamaV3 12414d0a, TabbyAPI 816c321, torch 2.13.0+cu130).
Launcher, guard, steward and sanitized evidence:
r0b0tlab/glm53-flash-exl3-exllamav3-gb10.
Container image (ARM64/SM121, no weights baked in):
docker pull ghcr.io/r0b0tlab/glm53-flash-exl3-exllamav3-gb10:b25c467
(registry digest sha256:e321670c…88c02).
FP16 KV, max_seq_len = cache_size = 262144, batch size 1,
sysmem_recurrent_cache: 512.
Verified result — near-window single-key retrieval (2026-09-27)
Token-exact needle-in-a-haystack on a fresh serve epoch: the client proves the
server-inserted BOS token against /v1/token/encode, fits the prompt with the
local tokenizer, and requires exact equality with the API's
usage.prompt_tokens.
- 259,993 actual prompt tokens (of a 262,144-token window, 2,048-token
output reserve, 103 tokens to spare): needle retrieved,
finish_reason=stop, clean answer, local/server token parity exact. - Matched 8K C1 five-repeat medians with this draft at K=5 (AR -> K5): structured 22.91 -> 52.90 tok/s, code 22.61 -> 49.96 tok/s, prose 22.54 -> 20.56 tok/s (prose regressed; not a blanket speedup). K5 counters: 4,832 draft tokens accepted / 3,673 rejected.
Q200v2 quality kit (2026-09-27, same profile, serial, effort=low)
GSM8K 79/80 (98.75%) · HumanEval 39/40 (97.5%) · IFEval 31/40 (77.5%) ·
hard_reasoning 19/20 (95.0%, manual review) — text-180: 168/180 (93.33%);
BFCL v4 multi_turn_base hard-20: 6/20 — full 200: 174/200 (87.0%).
All rows finish_reason=stop, zero transport errors. Evidence in the runtime
repo's evidence/glm53-exl3-q200v2-20260927/.
Not qualified
Multi-key near-window output formatting, concurrency > 1, throughput at 256K,
and long-run stability were not qualified on this profile. A memory guard
(host MemAvailable floor 6,144 MiB, 2-second sampling, fail-closed on new
NVRM/OOM journal events) bounded every run; no new NVRM allocation failure
occurred during the verified requests.
Use
# TabbyAPI draft_model block (see runtime repo for the full guarded launcher)
draft_model:
draft_mode: model
draft_model_dir: <this-repo>
draft_cache_mode: FP16
draft_num_tokens: 5
Provenance and license
Quantized by r0b0tlab from incoai/GLM-5.3-Flash-DFlash2, which is released
by Inco AI under CC BY-NC-ND 4.0
for research and evaluation. This repository distributes a quantization of that
checkpoint for research/evaluation use only, on the same non-commercial basis;
see the upstream model card for commercial licensing. EXL3 format follows
exllamav3 (MIT). The target-model pack is licensed MIT from
zai-org/GLM-5.3-Flash and is distributed separately.
- Downloads last month
- 318