File size: 4,283 Bytes
6216b58 55fc7ef 6216b58 c4b8a5d 6216b58 c4b8a5d 6216b58 55fc7ef c4b8a5d 55fc7ef 6216b58 c4b8a5d 6216b58 55fc7ef 6216b58 c4b8a5d 55fc7ef c4b8a5d 55fc7ef 6216b58 55fc7ef 6216b58 55fc7ef c4b8a5d 55fc7ef 6216b58 c4b8a5d 6216b58 55fc7ef c4b8a5d 55fc7ef 6216b58 c4b8a5d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 | # GLM Visual Runtime Deployment Report
## Summary
`glm-5.2-visual-runtime` is a training-free multimodal model gateway. It exposes an OpenAI-compatible model ID while routing text reasoning to GLM-5.2 and visual work to persistent visual variables and lazy lenses.
This repo is ready as a one-click all-local deployment manifest. It does not duplicate the upstream checkpoint weight shards because the official checkpoints are pulled from Hugging Face during deployment:
- BF16: `zai-org/GLM-5.2`
- FP8: `zai-org/GLM-5.2-FP8`
- Vision / omni: `Qwen/Qwen3-Omni-30B-A3B-Instruct`
- Optional alternate reasoning: `Qwen/Qwen3.6-27B`
For one-click deployment, run `one_click/docker-compose.all-local.yml`. It starts the gateway, local GLM-5.2 vLLM, local Qwen Omni vLLM-Omni, local OCR, PostgreSQL, and MinIO.
## Deployment Topology
```text
OpenAI client
-> glm-5.2-visual-runtime gateway
-> local reasoning model
vLLM serving zai-org/GLM-5.2-FP8
-> local vision model
vLLM-Omni serving Qwen/Qwen3-Omni-30B-A3B-Instruct
-> local OCR container
repo-built OCR service
-> local persistence
PostgreSQL + MinIO
```
## vLLM Requirements
- vLLM `>= 0.23.0` or `vllm/vllm-openai:glm52`.
- Multi-GPU server suitable for a 753B-class MoE FP8 model.
- Hugging Face access to `zai-org/GLM-5.2-FP8`.
- Hugging Face access to `Qwen/Qwen3-Omni-30B-A3B-Instruct`.
- Optional Hugging Face access to `Qwen/Qwen3.6-27B`.
- OpenAI-compatible vLLM port, default `8001` in the included examples.
## One-Click Command
```bash
cp one_click/.env.all-local.example one_click/.env.all-local
docker compose -f one_click/docker-compose.all-local.yml --env-file one_click/.env.all-local up --build
```
## Reasoning vLLM Command
```bash
vllm serve zai-org/GLM-5.2-FP8 \
--served-model-name glm-5.2 \
--kv-cache-dtype fp8 \
--tensor-parallel-size 8 \
--speculative-config.method mtp \
--speculative-config.num_speculative_tokens 5 \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--enable-auto-tool-choice
```
## Vision / Omni vLLM Command
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
--omni \
--served-model-name qwen3-omni \
--tensor-parallel-size 2 \
--limit-mm-per-prompt '{"image": 8, "video": 1, "audio": 1}'
```
## Optional Qwen3.6-27B Reasoning Command
```bash
vllm serve Qwen/Qwen3.6-27B \
--served-model-name qwen3.6-27b \
--tensor-parallel-size 2 \
--max-model-len 262144 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
```
## Gateway Environment For All-Local Mode
```bash
GLM_BASE_URL=http://glm52-vllm:8000/v1
GLM_MODEL=glm-5.2
VISION_BASE_URL=http://vision-vllm:8000/v1
VISION_MODEL=qwen3-omni
OCR_BASE_URL=http://ocr:8080
VISUAL_RUNTIME_MODE=local
```
## Single-Repo Weights Bundle
The deployment package includes `weights_manifest.json` and `scripts/materialize_weights.py`.
To physically place all required model weights in this same repo:
```bash
pip install huggingface_hub
python scripts/materialize_weights.py --profile glm52_plus_qwen_omni
hf upload-large-folder wassemgtk/glm-5.2-visual-runtime models --type model
```
Alternate all-Qwen profile:
```bash
python scripts/materialize_weights.py --profile qwen36_plus_qwen_omni
hf upload-large-folder wassemgtk/glm-5.2-visual-runtime models --type model
```
These downloads are large. Confirm Hugging Face quota, local disk, and upload time before materializing full checkpoints.
## Current Acceptance Status
- Training-free runtime: complete.
- HF model report: complete.
- HF Docker Space: running in deterministic test mode.
- OpenAI SDK compatibility: smoke-tested.
- Standard vLLM reasoning deployment files: included.
- Local Qwen Omni vLLM deployment files: included.
- Single-repo checkpoint materialization manifest/script: included.
- Local OCR container source: included.
- All-local one-click Compose profile: included.
## Important Constraint
Do not run:
```bash
vllm serve wassemgtk/glm-5.2-visual-runtime
```
This repo is a runtime deployment package. Run the included one-click compose profile; it pulls `zai-org/GLM-5.2-FP8` and `Qwen/Qwen3-Omni-30B-A3B-Instruct` as part of deployment, or materialize `models/` if you want the checkpoint files stored in the same repo.
|