Gemma-4-E4B-it-TextOnly-GPTQ-a16w4: Optimized for SiMa.ai Modalix

Overview

This repository contains the Gemma-4-E4B-it-TextOnly-GPTQ-a16w4 model, optimized and compiled for the SiMa.ai Modalix platform.

• Model architecture: Gemma 4 E4B (text-only)
• Quantization: A16W4 (16-bit activations, 4-bit weights)
• Pre-quantization: The source checkpoint was pre-quantized with GPTQ before compilation.
• Maximum context length: 8192
• Source model: google/gemma-4-E4B-it

Performance

The following measurements characterize text generation on Modalix.

Measured with MoLE using batch size 1, five samples per input length, and up to 128 generated tokens. Values are arithmetic means. TTFT includes language-model prefill and the first generated token; generation rate is measured after the first token.

Input tokens	Mean TTFT (seconds)	Mean generation rate (tokens/second)
128	0.16	19.33
256	0.31	19.24
512	0.61	18.97
1024	1.24	18.75
2048	2.51	18.09
3072	3.84	17.84
4096	5.22	17.47
5120	6.71	16.97
6144	8.24	15.52
7168	9.86	14.91

Prerequisites

To run this model, you need a SiMa.ai Modalix device with the SiMa.ai Neat Runtime installed.

Installation and Deployment

Download the precompiled model directly on Modalix:

llima pull Gemma-4-E4B-it-TextOnly-GPTQ-a16w4

The model is installed under:

/media/nvme/llima/models/Gemma-4-E4B-it-TextOnly-GPTQ-a16w4

Alternatively, download it on a host and copy it to Modalix:

hf download simaai/Gemma-4-E4B-it-TextOnly-GPTQ-a16w4 --local-dir Gemma-4-E4B-it-TextOnly-GPTQ-a16w4
scp -r Gemma-4-E4B-it-TextOnly-GPTQ-a16w4 sima@<modalix-ip>:/media/nvme/llima/models/

Usage

Validate the model directly on Modalix:

llima run Gemma-4-E4B-it-TextOnly-GPTQ-a16w4

For application integration, see GenAI with LLiMa.

Limitations

• This repository contains compiled Modalix artifacts and cannot be loaded with transformers.AutoModel.
• Quantization can cause minor differences from the full-precision source model.

Resources

• GenAI with LLiMa
• Serve GenAI Models
• Run an LLM
• Source model