Instructions to use amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16")
model = AutoModelForMultimodalLM.from_pretrained("amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16

SGLang

How to use amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16 with Docker Model Runner:
```
docker model run hf.co/amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16
```

Qwen3.8-27B-Quark-AWQ-INT4-W4A16

INT4 weight-only (W4A16) quantized version of Qwen/Qwen3.8-27B, produced with the AWQ algorithm using AMD Quark.

Weights: INT4, group size 128
Activations: BF16 (unquantized)
Algorithm: AWQ (activation-aware weight quantization)
Calibration: 128 samples, sequence length 512
Base model: Qwen/Qwen3.8-27B (Apache 2.0)

Benchmark results

Benchmark	Setting	This model (AWQ)	BF16 base	Recovery %
GSM8K, 5-shot (flexible-extract / strict-match)	Thinking: `temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0`, `max_gen_toks=8192`	91.21% / 90.67%	93.33% / 93.33%	97.7%
GSM8K, 5-shot (flexible-extract / strict-match)	Non-thinking: `temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0`, `max_gen_toks=8192`	91.51% / 90.37%	90.67% / 89.76%	100.9%
Wikitext perplexity	Greedy	8.8250	8.4364	95.6%
BFCL Overall Acc (single_turn)*	Greedy (harness default)	24.06%	24.38%	98.7%

* BFCL Overall Acc reflects only single_turn categories, not the full Gorilla-leaderboard formula (multi-turn/web-search/memory categories were not run and would count as 0 against the public leaderboard's own Overall Acc). Sub-metrics: Non-Live AST 86.58% (base 88.52%), Live AST 81.57% (base 83.05%), Relevance Detection 75.00% (base 75.00%), Irrelevance Detection 72.47% (base 72.22%).

Recovery % = quantized / BF16-base, using flexible-extract for GSM8K rows and base/quantized (inverted, since lower is better) for perplexity — both measured by us against verified-upstream Qwen/Qwen3.8-27B weights, not vendor-reported numbers. GSM8K uses lm-evaluation-harness; non-thinking mode is approximated by pre-closing an empty <think></think> block in the prompt, since the harness task is a raw few-shot completion rather than a chat-templated request. BFCL run via the official bfcl_eval harness.

Eval command

GSM8K, thinking mode, via lm-evaluation-harness's native vLLM backend:

lm-eval run \
  --model vllm \
  --model_args pretrained=amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.4,enforce_eager=True,trust_remote_code=True \
  --tasks gsm8k \
  --num_fewshot 5 \
  --gen_kwargs max_gen_toks=8192,do_sample=True,temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=0.0,repetition_penalty=1.0 \
  --batch_size auto \
  --log_samples

GSM8K, no-thinking mode. The prompt suffix <think>\n\n</think>\n\n is prepended to each answer field to suppress the model's thinking preamble (see custom task yaml below).

# gsm8k_nothink.yaml
task: gsm8k_nothink
dataset_path: openai/gsm8k
dataset_name: main
output_type: generate_until
training_split: train
fewshot_split: train
test_split: test
doc_to_text: "Question: {{question}}\nAnswer: <think>\n\n</think>\n\n"
doc_to_target: "{{answer}}"
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: false
    regexes_to_ignore: [",", "\\$", "(?s).*#### ", "\\.$"]
generation_kwargs:
  until: ["Question:", "</s>", "<|im_end|>"]
  do_sample: false
  temperature: 0.0
repeats: 1
num_fewshot: 5
filter_list:
  - name: "strict-match"
    filter: [{function: "regex", regex_pattern: "#### (\\-?[0-9\\.\\,]+)"}, {function: "take_first"}]
  - name: "flexible-extract"
    filter: [{function: "regex", group_select: -1, regex_pattern: "(-?[$0-9.,]{2,})|(-?[0-9]+)"}, {function: "take_first"}]
metadata: {version: 3.0}

lm-eval run \
  --model vllm \
  --model_args pretrained=amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.4,enforce_eager=True,trust_remote_code=True \
  --tasks gsm8k_nothink \
  --include_path <dir containing gsm8k_nothink.yaml> \
  --num_fewshot 5 \
  --gen_kwargs max_gen_toks=8192,do_sample=True,temperature=0.7,top_p=0.80,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0 \
  --batch_size auto \
  --log_samples

--gen_kwargs on the CLI overrides the YAML task's own generation_kwargs defaults (greedy) with the instruct-mode recommended sampling parameters used for the non-thinking scores above.

Quantization command

python3 quantize_quark.py \
  --model_dir Qwen/Qwen3.8-27B \
  --output_dir Qwen3.8-27B-Quark-AWQ-INT4-W4A16 \
  --quant_scheme int4_wo_128 \
  --num_calib_data 128 \
  --seq_len 512 \
  --quant_algo awq \
  --model_export hf_format \
  --data_type auto \
  --device cuda

Run from Quark/examples/torch/language_modeling/llm_ptq using AMD Quark with native qwen3_5 architecture support for AWQ (contributed upstream).

Serving

Requires a Quark-compatible inference runtime with W4A16Int4 scheme support (#48606 and #46110).

vllm serve amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --reasoning-parser qwen3

License

Downloads last month: 86,143

Safetensors

Model size

6B params

Tensor type

I32

BF16

Model tree for amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16

Base model

Qwen/Qwen3.8-27B

Quantized

(860)

this model