tiny-random-qwen38-flash-next

A tiny model with random weights and the architecture of Qwen3.8-Flash-Next (qwen4_exp, Qwen4ExpForConditionalGeneration), for testing (optimum-intel / OpenVINO). Its outputs are not meaningful. It needs transformers >= 5.17.

Architecture

Every component type of the real model is present, with the same structure and smaller sizes:

Component	Tiny model	Qwen3.8-Flash-Next
Layers	8 = 2 × (3 Gated DeltaNet + 1 Qwen Sparse Attention)	48 = 12 × (3 + 1)
Hidden size	16	2560
Qwen Sparse Attention (QSA)	4 query / 2 KV heads × 64; indexer 4 heads × 16, blocks of 4 tokens, budget 8 tokens	24 / 2 × 256; indexer 4 × 128, blocks of 4, budget 2048
Gated DeltaNet	2 key / 6 value heads × 8	16 / 48 × 128
MoE	16 experts, top-4, shared expert	512 experts, top-10, shared expert
Hyper-connections	4 streams, low rank 8	4 streams, low rank 320
N-gram PLE (layer 2)	bi- and tri-grams, 2 heads per order, prime vocabularies from 1000, 4 shards	8 heads per order, prime vocabularies from 2·10^7, 128 shards
Multi-Token Prediction head	1 QSA layer (mtp.*)	1 QSA layer (mtp.*)
Vision encoder	depth 2, hidden 32	depth 27, hidden 1152
Vocabulary, tokenizer, chat template, processors	the real ones

The selection budget is small so that QSA is really sparse on short prompts.

The checkpoint keeps the layout of the real one:

• fused 3D experts.gate_up_proj / experts.down_proj;
• the n-gram table split into ngram_embedding.shard_*;
• the MTP head under mtp..

transformers does not build the MTP head (_keys_to_ignore_on_load_unexpected = ["^mtp.*"]) and ignores its weights when loading. Its structure follows vLLM (vllm/models/qwen4_exp):

• fc_embedding projects the next token's embedding;
• a shared fc_hidden projects the main model's pre-final-mixer streams;
• the projected embedding is added to every stream;
• one QSA decoder layer and the head's hyper_connection_mixer follow, then the shared lm_head.

Initialization

The default initialization (std 0.02) gives nearly uniform logits. Greedy tokens then depend on rounding noise, and comparing runtimes means little. The weights are therefore drawn with more variance:

• layers: std 0.25;
• embeddings: std 1;
• lm_head: std 2;
• MoE routers: std 1.

The same scheme is used for the MTP head.

The model was checked on text, image and video prompts. For both the main model and the MTP head:

• outputs are finite, with logits std about 4 and clear greedy top-2 margins (median above 0.3);
• 32 greedy tokens are all or nearly all distinct, with no repetition runs;
• all experts are used;
• QSA drops 23–93% of the visible tokens.

For the main model, the n-gram PLE contributes 18–59% of the residual-stream norm.

For the MTP head:

• its predictions depend on both of its inputs;
• a chained second draft step stays healthy.