glm-5.2-visual-runtime / deployment_report.md

Switch local vision stack to Qwen Omni and add weights bundle manifest

c4b8a5d verified 5 months ago

4.28 kB

# GLM Visual Runtime Deployment Report

## Summary

`glm-5.2-visual-runtime` is a training-free multimodal model gateway. It exposes an OpenAI-compatible model ID while routing text reasoning to GLM-5.2 and visual work to persistent visual variables and lazy lenses.

This repo is ready as a one-click all-local deployment manifest. It does not duplicate the upstream checkpoint weight shards because the official checkpoints are pulled from Hugging Face during deployment:

- BF16: `zai-org/GLM-5.2`
- FP8: `zai-org/GLM-5.2-FP8`
- Vision / omni: `Qwen/Qwen3-Omni-30B-A3B-Instruct`
- Optional alternate reasoning: `Qwen/Qwen3.6-27B`

For one-click deployment, run `one_click/docker-compose.all-local.yml`. It starts the gateway, local GLM-5.2 vLLM, local Qwen Omni vLLM-Omni, local OCR, PostgreSQL, and MinIO.

## Deployment Topology

```text
OpenAI client
  -> glm-5.2-visual-runtime gateway
  -> local reasoning model
       vLLM serving zai-org/GLM-5.2-FP8
  -> local vision model
       vLLM-Omni serving Qwen/Qwen3-Omni-30B-A3B-Instruct
  -> local OCR container
       repo-built OCR service
  -> local persistence
       PostgreSQL + MinIO
```

## vLLM Requirements

- vLLM `>= 0.23.0` or `vllm/vllm-openai:glm52`.
- Multi-GPU server suitable for a 753B-class MoE FP8 model.
- Hugging Face access to `zai-org/GLM-5.2-FP8`.
- Hugging Face access to `Qwen/Qwen3-Omni-30B-A3B-Instruct`.
- Optional Hugging Face access to `Qwen/Qwen3.6-27B`.
- OpenAI-compatible vLLM port, default `8001` in the included examples.

## One-Click Command

```bash
cp one_click/.env.all-local.example one_click/.env.all-local
docker compose -f one_click/docker-compose.all-local.yml --env-file one_click/.env.all-local up --build
```

## Reasoning vLLM Command

```bash
vllm serve zai-org/GLM-5.2-FP8 \
  --served-model-name glm-5.2 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice
```

## Vision / Omni vLLM Command

```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --omni \
  --served-model-name qwen3-omni \
  --tensor-parallel-size 2 \
  --limit-mm-per-prompt '{"image": 8, "video": 1, "audio": 1}'
```

## Optional Qwen3.6-27B Reasoning Command

```bash
vllm serve Qwen/Qwen3.6-27B \
  --served-model-name qwen3.6-27b \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

## Gateway Environment For All-Local Mode

```bash
GLM_BASE_URL=http://glm52-vllm:8000/v1
GLM_MODEL=glm-5.2
VISION_BASE_URL=http://vision-vllm:8000/v1
VISION_MODEL=qwen3-omni
OCR_BASE_URL=http://ocr:8080
VISUAL_RUNTIME_MODE=local
```

## Single-Repo Weights Bundle

The deployment package includes `weights_manifest.json` and `scripts/materialize_weights.py`.

To physically place all required model weights in this same repo:

```bash
pip install huggingface_hub
python scripts/materialize_weights.py --profile glm52_plus_qwen_omni
hf upload-large-folder wassemgtk/glm-5.2-visual-runtime models --type model
```

Alternate all-Qwen profile:

```bash
python scripts/materialize_weights.py --profile qwen36_plus_qwen_omni
hf upload-large-folder wassemgtk/glm-5.2-visual-runtime models --type model
```

These downloads are large. Confirm Hugging Face quota, local disk, and upload time before materializing full checkpoints.

## Current Acceptance Status

- Training-free runtime: complete.
- HF model report: complete.
- HF Docker Space: running in deterministic test mode.
- OpenAI SDK compatibility: smoke-tested.
- Standard vLLM reasoning deployment files: included.
- Local Qwen Omni vLLM deployment files: included.
- Single-repo checkpoint materialization manifest/script: included.
- Local OCR container source: included.
- All-local one-click Compose profile: included.

## Important Constraint

Do not run:

```bash
vllm serve wassemgtk/glm-5.2-visual-runtime
```

This repo is a runtime deployment package. Run the included one-click compose profile; it pulls `zai-org/GLM-5.2-FP8` and `Qwen/Qwen3-Omni-30B-A3B-Instruct` as part of deployment, or materialize `models/` if you want the checkpoint files stored in the same repo.

	# GLM Visual Runtime Deployment Report

	## Summary

	`glm-5.2-visual-runtime` is a training-free multimodal model gateway. It exposes an OpenAI-compatible model ID while routing text reasoning to GLM-5.2 and visual work to persistent visual variables and lazy lenses.

	This repo is ready as a one-click all-local deployment manifest. It does not duplicate the upstream checkpoint weight shards because the official checkpoints are pulled from Hugging Face during deployment:

	- BF16: `zai-org/GLM-5.2`
	- FP8: `zai-org/GLM-5.2-FP8`
	- Vision / omni: `Qwen/Qwen3-Omni-30B-A3B-Instruct`
	- Optional alternate reasoning: `Qwen/Qwen3.6-27B`

	For one-click deployment, run `one_click/docker-compose.all-local.yml`. It starts the gateway, local GLM-5.2 vLLM, local Qwen Omni vLLM-Omni, local OCR, PostgreSQL, and MinIO.

	## Deployment Topology

	```text
	OpenAI client
	-> glm-5.2-visual-runtime gateway
	-> local reasoning model
	vLLM serving zai-org/GLM-5.2-FP8
	-> local vision model
	vLLM-Omni serving Qwen/Qwen3-Omni-30B-A3B-Instruct
	-> local OCR container
	repo-built OCR service
	-> local persistence
	PostgreSQL + MinIO
	```

	## vLLM Requirements

	- vLLM `>= 0.23.0` or `vllm/vllm-openai:glm52`.
	- Multi-GPU server suitable for a 753B-class MoE FP8 model.
	- Hugging Face access to `zai-org/GLM-5.2-FP8`.
	- Hugging Face access to `Qwen/Qwen3-Omni-30B-A3B-Instruct`.
	- Optional Hugging Face access to `Qwen/Qwen3.6-27B`.
	- OpenAI-compatible vLLM port, default `8001` in the included examples.

	## One-Click Command

	```bash
	cp one_click/.env.all-local.example one_click/.env.all-local
	docker compose -f one_click/docker-compose.all-local.yml --env-file one_click/.env.all-local up --build
	```

	## Reasoning vLLM Command

	```bash
	vllm serve zai-org/GLM-5.2-FP8 \
	--served-model-name glm-5.2 \
	--kv-cache-dtype fp8 \
	--tensor-parallel-size 8 \
	--speculative-config.method mtp \
	--speculative-config.num_speculative_tokens 5 \
	--tool-call-parser glm47 \
	--reasoning-parser glm45 \
	--enable-auto-tool-choice
	```

	## Vision / Omni vLLM Command

	```bash
	vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
	--omni \
	--served-model-name qwen3-omni \
	--tensor-parallel-size 2 \
	--limit-mm-per-prompt '{"image": 8, "video": 1, "audio": 1}'
	```

	## Optional Qwen3.6-27B Reasoning Command

	```bash
	vllm serve Qwen/Qwen3.6-27B \
	--served-model-name qwen3.6-27b \
	--tensor-parallel-size 2 \
	--max-model-len 262144 \
	--reasoning-parser qwen3 \
	--enable-auto-tool-choice \
	--tool-call-parser qwen3_coder
	```

	## Gateway Environment For All-Local Mode

	```bash
	GLM_BASE_URL=http://glm52-vllm:8000/v1
	GLM_MODEL=glm-5.2
	VISION_BASE_URL=http://vision-vllm:8000/v1
	VISION_MODEL=qwen3-omni
	OCR_BASE_URL=http://ocr:8080
	VISUAL_RUNTIME_MODE=local
	```

	## Single-Repo Weights Bundle

	The deployment package includes `weights_manifest.json` and `scripts/materialize_weights.py`.

	To physically place all required model weights in this same repo:

	```bash
	pip install huggingface_hub
	python scripts/materialize_weights.py --profile glm52_plus_qwen_omni
	hf upload-large-folder wassemgtk/glm-5.2-visual-runtime models --type model
	```

	Alternate all-Qwen profile:

	```bash
	python scripts/materialize_weights.py --profile qwen36_plus_qwen_omni
	hf upload-large-folder wassemgtk/glm-5.2-visual-runtime models --type model
	```

	These downloads are large. Confirm Hugging Face quota, local disk, and upload time before materializing full checkpoints.

	## Current Acceptance Status

	- Training-free runtime: complete.
	- HF model report: complete.
	- HF Docker Space: running in deterministic test mode.
	- OpenAI SDK compatibility: smoke-tested.
	- Standard vLLM reasoning deployment files: included.
	- Local Qwen Omni vLLM deployment files: included.
	- Single-repo checkpoint materialization manifest/script: included.
	- Local OCR container source: included.
	- All-local one-click Compose profile: included.

	## Important Constraint

	Do not run:

	```bash
	vllm serve wassemgtk/glm-5.2-visual-runtime
	```

	This repo is a runtime deployment package. Run the included one-click compose profile; it pulls `zai-org/GLM-5.2-FP8` and `Qwen/Qwen3-Omni-30B-A3B-Instruct` as part of deployment, or materialize `models/` if you want the checkpoint files stored in the same repo.