GLM-5.3-Flash-DFlash2 — EXL3 3.00 bpw (draft)

EXL3 (ExLlamaV3 trellis) quantization by r0b0tlab of the incoai/GLM-5.3-Flash-DFlash2 block-diffusion speculative-decoding draft, for use with r0b0tlab/GLM-5.3-Flash-EXL3-2.25bpw-sm121.

  • Format: EXL3 3.00 bpw, mul1 codebook, 6-bit head, output scales on, calibration 250x2048 (quantization_config.json carries the full module/tensor manifest).
  • Size: 610 MiB, single model.safetensors shard.
  • Role: DFlash2 K=5 drafter (TabbyAPI draft_mode: model, draft_num_tokens: 5). Not a standalone language model.

Qualified configuration (single NVIDIA GB10 / DGX Spark, SM121, aarch64)

Runtime: pinned ExLlamaV3 v1.5.2 + TabbyAPI in a native ARM64 CUDA 13 container (ExLlamaV3 12414d0a, TabbyAPI 816c321, torch 2.13.0+cu130). Launcher, guard, steward and sanitized evidence: r0b0tlab/glm53-flash-exl3-exllamav3-gb10. Container image (ARM64/SM121, no weights baked in): docker pull ghcr.io/r0b0tlab/glm53-flash-exl3-exllamav3-gb10:b25c467 (registry digest sha256:e321670c…88c02).

FP16 KV, max_seq_len = cache_size = 262144, batch size 1, sysmem_recurrent_cache: 512.

Verified result — near-window single-key retrieval (2026-09-27)

Token-exact needle-in-a-haystack on a fresh serve epoch: the client proves the server-inserted BOS token against /v1/token/encode, fits the prompt with the local tokenizer, and requires exact equality with the API's usage.prompt_tokens.

  • 259,993 actual prompt tokens (of a 262,144-token window, 2,048-token output reserve, 103 tokens to spare): needle retrieved, finish_reason=stop, clean answer, local/server token parity exact.
  • Matched 8K C1 five-repeat medians with this draft at K=5 (AR -> K5): structured 22.91 -> 52.90 tok/s, code 22.61 -> 49.96 tok/s, prose 22.54 -> 20.56 tok/s (prose regressed; not a blanket speedup). K5 counters: 4,832 draft tokens accepted / 3,673 rejected.

Q200v2 quality kit (2026-09-27, same profile, serial, effort=low)

GSM8K 79/80 (98.75%) · HumanEval 39/40 (97.5%) · IFEval 31/40 (77.5%) · hard_reasoning 19/20 (95.0%, manual review) — text-180: 168/180 (93.33%); BFCL v4 multi_turn_base hard-20: 6/20 — full 200: 174/200 (87.0%). All rows finish_reason=stop, zero transport errors. Evidence in the runtime repo's evidence/glm53-exl3-q200v2-20260927/.

Not qualified

Multi-key near-window output formatting, concurrency > 1, throughput at 256K, and long-run stability were not qualified on this profile. A memory guard (host MemAvailable floor 6,144 MiB, 2-second sampling, fail-closed on new NVRM/OOM journal events) bounded every run; no new NVRM allocation failure occurred during the verified requests.

Use

# TabbyAPI draft_model block (see runtime repo for the full guarded launcher)
draft_model:
  draft_mode: model
  draft_model_dir: <this-repo>
  draft_cache_mode: FP16
  draft_num_tokens: 5

Provenance and license

Quantized by r0b0tlab from incoai/GLM-5.3-Flash-DFlash2, which is released by Inco AI under CC BY-NC-ND 4.0 for research and evaluation. This repository distributes a quantization of that checkpoint for research/evaluation use only, on the same non-commercial basis; see the upstream model card for commercial licensing. EXL3 format follows exllamav3 (MIT). The target-model pack is licensed MIT from zai-org/GLM-5.3-Flash and is distributed separately.

Downloads last month
318
Safetensors
Model size
0.3B params
Tensor type
F16
·
I16
·
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for r0b0tlab/GLM-5.3-Flash-DFlash2-EXL3-3.00bpw

Quantized
(5)
this model