steampunque/Qwen3-Coder-30B-A3B-Instruct-MP-GGUF

Mixed Precision GGUF layer quantization of Qwen3-Coder-30B-A3B-Instruct by Qwen

Original model: https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct

The hybrid quant employs different quantization levels on a per layer basis to increase flexibility of trading off performance vs file size. Less parameter bits are used at deep layers and more bits at cortex layers to simulultaneously optimize quantized size and model performance. An extended layer definition E quant Q4_E_H for the model is defined as follows (updated 9/18/2026):

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q5_K_886"],[1 ,"Q5_K_885"],[2 ,"Q5_K_664"],[3 ,"Q4_K_664"],[4 ,"Q4_K_654"],[5 ,"Q4_K_554"],[6 ,"Q4_K_554"],[7 ,"Q4_K_554"],
   [8 ,"Q4_K_444"],[9 ,"Q4_K_444"],[10,"Q4_K_444"],[11,"Q4_K_444"],[12,"Q4_K_444"],[13,"Q4_K_444"],[14,"Q4_K_444"],[15,"Q4_K_444"],
   [16,"Q4_K_554"],[17,"Q4_K_554"],[18,"Q4_K_554"],[19,"Q4_K_554"],[20,"Q4_K_654"],[21,"Q4_K_654"],[22,"Q4_K_654"],[23,"Q4_K_654"],
   [24,"Q4_K_654"],[25,"Q4_K_654"],[26,"Q4_K_654"],[27,"Q4_K_654"],[28,"Q4_K_654"],[29,"Q4_K_654"],[30,"Q4_K_654"],[31,"Q4_K_654"],
   [32,"Q4_K_664"],[33,"Q4_K_664"],[34,"Q4_K_664"],[35,"Q4_K_664"],[36,"Q4_K_664"],[37,"Q4_K_664"],[38,"Q4_K_664"],[39,"Q4_K_664"],
   [40,"Q4_K_665"],[41,"Q4_K_665"],[42,"Q4_K_665"],[43,"Q5_K_665"],[44,"Q5_K_665"],[45,"Q5_K_666"],[46,"Q5_K_668"],[47,"Q6_K_866"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The layer quants were optimized for strong performance across a small set of code gen prompts on both cuda backend and CPU backend with CPU expert offload while sized to approximately Q4_K_M bit efficiency.

Comparison:

Quant	size	PPL	Comment
Q4_K_M	18.6e9	9.6	default embed and output
Q4_E_H	18.3e9	9.4	Q6_K embed Q6_K output

Usage:

This moe model can be efficiently run by offloading expert tensors to CPU via -ot exps=CPU to open up very large context space. The smaller size of the optimally quantized parameters will give an effective boost in CPU processing speed due to reducing the memory BW needed to repeatedly copy them from main memory to SIMD regs. It can also run fully offloaded on GPU via RPC or high VRAM GPU.

The model can be speculated using Qwen2.5-Coder-0.5B-Instruct if the inference platform can support vocabulary translation between draft and target. Example performance on humaneval first problem prompt using a downstream speculator with fixed draft block length ND:

Config	ND	F16 context size	Q8 context size	gen rate
2 4070, RPC	0	~55k tokens	~100k tokens	112tps
2 4070, RPC	4	~45k tokens	~75k tokens	142tps
1 4070, exp CPU	0	~105k tokens	~200k tokens	22 tps
1 4070, exp CPU	2	~90k tokens	~155k tokens	25 tps

The model has a native 256k context window which can be extended to 1M with Yarn as follows:

Set base context for yarn rope scale compute to 262144 (256k), then with a context of N tokens > 256k the rope scale = N / 256k.

Example, if N = 512k then rope scale = 2.0.

Then on model start pass --rope-scaling yarn --yarn-orig-ctx 262144 --rope_scale 2.0 (must be ajusted if kv other than 512k)

Later versions of llama.cpp have a bug which soft caps context length to the training context, effectively disabling yarn context extension. Patch server-context.cpp according to https://github.com/ggml-org/llama.cpp/issues/22140 to fix it.

High context performance appears to work verified against a needle in haystack test at 75k tokens.

Benchmarks:

Code evals for the model are given here: https://huggingface.co/spaces/steampunque/benchlm.

Download the file from below:

Link	Type	Size/e9 B	Notes
Qwen3-30B-Coder-A3B-Instruct.Q4_E_H.gguf	Q4_E_H	18.3e9 B	~Q4_K_M size

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040