Instructions to use c-bf/GLM-5.2-AutoRound-W4G64-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use c-bf/GLM-5.2-AutoRound-W4G64-MTP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="c-bf/GLM-5.2-AutoRound-W4G64-MTP")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("c-bf/GLM-5.2-AutoRound-W4G64-MTP")
model = AutoModelForCausalLM.from_pretrained("c-bf/GLM-5.2-AutoRound-W4G64-MTP")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use c-bf/GLM-5.2-AutoRound-W4G64-MTP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "c-bf/GLM-5.2-AutoRound-W4G64-MTP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "c-bf/GLM-5.2-AutoRound-W4G64-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/c-bf/GLM-5.2-AutoRound-W4G64-MTP

SGLang

How to use c-bf/GLM-5.2-AutoRound-W4G64-MTP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "c-bf/GLM-5.2-AutoRound-W4G64-MTP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "c-bf/GLM-5.2-AutoRound-W4G64-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "c-bf/GLM-5.2-AutoRound-W4G64-MTP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "c-bf/GLM-5.2-AutoRound-W4G64-MTP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use c-bf/GLM-5.2-AutoRound-W4G64-MTP with Docker Model Runner:
```
docker model run hf.co/c-bf/GLM-5.2-AutoRound-W4G64-MTP
```

GLM-5.2 AutoRound W4G64

This is an unofficial community quantization of zai-org/GLM-5.2. It is not an official Z.ai release and is not affiliated with Z.ai.

The checkpoint targets high-throughput reasoning and coding-agent serving on a single 8-GPU Hopper node. Routed MoE expert weights use symmetric 4-bit AutoRound quantization with group size 64. Attention, the DSA/IndexShare indexer, shared experts, sensitive projections, and the output head remain in BF16. The MTP layer is included and can be used for speculative decoding.

Checkpoint summary

Property	Value
Base model	`zai-org/GLM-5.2`
Architecture	`GlmMoeDsaForCausalLM`
Quantization	AutoRound W4A16, symmetric W4G64
Packing	`auto_round:auto_gptq`
Checkpoint layout	81 backbone shards + 5 MTP shards
Approximate storage	405 GiB
MTP	One quantized next-token-prediction layer
License	MIT, inherited from the base model

Quantization procedure

This is post-training weight-only quantization; the model was not fine-tuned. The run used AutoRound 0.14.0 on four NVIDIA H20 GPUs with:

Setting	Value
Weight bits	4
Group size	64
Symmetric	Yes
Optimization iterations	200
Calibration samples	512
Calibration sequence length	2,048
Batch size	2
Gradient accumulation	4
Scale dtype	FP16

The calibration set consisted of packed coding-agent-style text samples. The calibration corpus is not distributed with this checkpoint.

The mixed-precision policy was intentionally conservative:

Backbone layers 3–77: routed-expert gate_proj, up_proj, and down_proj matrices are W4G64.
Backbone layers 0–2, all attention and indexer modules, shared experts, eh_proj, weights_proj, embeddings, and lm_head remain BF16.
MTP layer 78 was processed separately: 768 routed-expert weights are W4G64; 23 non-expert tensors remain BF16. The MTP shared head reuses the BF16 target head at runtime.

Validated runtime

The tested software stack was:

Component	Version
vLLM	`0.23.1rc1.dev471+ge312c5cb2`
vLLM commit	`e312c5cb25427e76fc3830ab14e7b6bc0963a55c`
Python	3.12.12
PyTorch	2.11.0+cu130
CUDA	13.0
Hardware	8 × NVIDIA H20 96 GB, tensor parallel size 8

apply_vllm_glm52_patches.py contains the source-level compatibility changes used for this checkpoint. It is strictly hash-pinned to the vLLM commit and Triton source above, stages all edits before installation, compiles and verifies every output, installs files atomically, and is idempotent. It refuses unknown or partially modified sources instead of attempting a fuzzy patch.

The patch covers:

GLM-5.2 sparse-indexer and missing-parameter guards, plus AutoRound routed expert and MTP checkpoint namespace compatibility.
Header-first safetensors filtering so MTP eager loading does not read every unrelated backbone shard payload.
CUDA Graph pre-capture warmup and real cubin generation through ptxas.
Explicit PyNCCL loader/device handling and correct CUDA-device binding for the CPU KV-cache pinning thread.

These changes affect loading and serving only; they do not modify model weights. Use a separate environment for this pinned runtime. A newer vLLM version may already contain equivalent fixes and should be validated independently.

vLLM serving example

The following is a public, minimal reproduction of the tested serving profile. It assumes that vLLM has already been built at the commit above and that ptxas is available on PATH. Replace the example paths and public model name.

export VENV=/path/to/vllm-venv
export MODEL_DIR=/path/to/GLM-5.2-AutoRound-W4G64

"$VENV/bin/python" "$MODEL_DIR/apply_vllm_glm52_patches.py" \
  --venv-root "$VENV" \
  --model-dir "$MODEL_DIR" \
  --apply

# Keep compilation and CUDA Graphs enabled. These settings match the validated
# single-node runtime; adapt NCCL transport selection to your topology.
export VLLM_DISABLE_COMPILE_CACHE=1
export VLLM_DISABLE_PYNCCL=0
export NCCL_CUMEM_ENABLE=0
export NCCL_CUMEM_HOST_ENABLE=0
export VLLM_ALLREDUCE_USE_SYMM_MEM=0
export VLLM_USE_NCCL_SYMM_MEM=0
export TRITON_STORE_BINARY_ONLY=0
export CUDA_MODULE_LOADING=EAGER

"$VENV/bin/vllm" serve "$MODEL_DIR" \
  --served-model-name glm-5.2-autoround-w4g64 \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 8 \
  --dtype auto \
  --max-model-len 130000 \
  --gpu-memory-utilization 0.94 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --chat-template-content-format string \
  --trust-remote-code \
  --safetensors-load-strategy eager \
  --distributed-timeout-seconds 1800 \
  --disable-custom-all-reduce \
  --enable-prefix-caching \
  --compilation-config.pass_config.fuse_allreduce_rms false \
  --speculative-config '{"method":"mtp"}' \
  --spec-tokens 2 \
  --kv-offloading-size 256 \
  --kv-offloading-backend native \
  --disable-hybrid-kv-cache-manager

The 256 GiB CPU KV offload setting is optional and requires sufficient pinned host memory. Reduce or remove it for systems without that capacity.

Example OpenAI-compatible request:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "glm-5.2-autoround-w4g64",
    "messages": [{"role": "user", "content": "Explain and solve x^2 = 1 (mod 105)."}],
    "max_tokens": 8192,
    "temperature": 0.6,
    "top_p": 0.95,
    "stream": true
  }'

Reasoning models may use several thousand tokens before final content. For long-horizon coding-agent workloads, a 32K output allowance was materially more reliable than an 8K allowance.

Measured performance

Measurements below used the validated stack on one 8 × H20 node with TP8, 130K maximum context, MTP speculative depth 2, max_num_seqs=8, an 8,192-token batch ceiling, prefix caching, CUDA Graphs, and 256 GiB native CPU KV offload.

The fixed-shape serving test used the same 117-token prompt, exactly 512 output tokens per request, streaming responses, and a 60-second load window. Throughput is the server generation-token counter divided by wall time.

Concurrency	Successful requests	Aggregate generation	Mean / P95 TTFT	Mean prefill	Mean decode
3	30 / 30	229.09 tok/s	0.411 / 0.859 s	0.351 s	6.116 s
5	40 / 40	310.30 tok/s	0.438 / 0.679 s	0.361 s	7.586 s
8	56 / 56	441.11 tok/s	0.437 / 0.692 s	0.346 s	8.417 s

No request preemption or metrics scrape error occurred in these windows. MTP accepted 57.7%–58.7% of drafted tokens, for an effective accepted length of about 2.16 tokens per verification iteration.

A separate reasoning smoke test produced 4,083 tokens at 110.16 tok/s after a 0.266-second TTFT, stopped normally, returned both reasoning and final content, and correctly enumerated the eight solutions of x² ≡ 1 (mod 105).

On isolated coding-agent tasks, the model passed all public and held-out tests for the four completed single-concurrency tasks, ranging from configuration merging to a parser/formatter/evaluator. In concurrent agent runs, aggregate generation was 218.20 tok/s at concurrency 3 and 278.04–292.79 tok/s at concurrency 5. These are workload observations, not standardized leaderboard results.

Performance varies with prompt length, output distribution, MTP acceptance, cache state, host-memory bandwidth, software build, and GPU topology. The numbers above should not be compared with other model cards unless the harness and serving configuration are matched.

Quality and limitations

This checkpoint has not been evaluated against the BF16 base model with a comprehensive, identical public benchmark harness. No claim of lossless quantization is made.
Only the vLLM commit and 8 × H20 configuration documented above were fully validated. Other vLLM revisions, runtimes, GPU architectures, and tensor parallel sizes may require additional work.
Quantization is concentrated in routed experts, but quality can still differ from the BF16 base model, especially on rare domains or long reasoning paths.
The checkpoint inherits the capabilities, risks, usage restrictions, and limitations described in the official GLM-5.2 model card.
Users are responsible for application-specific safety, privacy, reliability, and output-quality evaluation before deployment.

License and references

This quantized checkpoint follows the base model's MIT license. Review the base repository and license before redistribution or production use.

Downloads last month: 174

Safetensors

Model size

122B params

Tensor type

I32

BF16

F16

F32

Model tree for c-bf/GLM-5.2-AutoRound-W4G64-MTP

Base model

zai-org/GLM-5.2

Quantized

(107)

this model

Papers for c-bf/GLM-5.2-AutoRound-W4G64-MTP

GLM-5: from Vibe Coding to Agentic Engineering

Paper • 2602.15763 • Published Feb 17 • 200

Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs

Paper • 2309.05516 • Published Sep 11, 2023 • 13