LFM2.5-1.2B-JP — LiteRT-LM

LiquidAI/LFM2.5-1.2B-JP converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (requires litert-lm ≥ 0.14 / a recent AI Edge Gallery). Sibling of litert-community/LFM2.5-1.2B-Instruct.

Update (2026-08-04): the .litertlm files were updated in place to add the ExecutorMetadata section that litert-lm ≥ 0.15 requires to bind the hybrid conv/attention state buffers (without it, 0.15 fails at inference with missing some output TensorBuffers). Weights and graph are byte-identical to the original release, and the files continue to run on litert-lm 0.14.

LFM2.5-1.2B-JP is the Japanese-optimized chat variant of Liquid AI's hybrid conv-attention flagship — a 1.2B model tuned for natural Japanese conversation that runs fully on-device at ~100 tok/s on a laptop CPU.

日本語に最適化された 1.2B オンデバイスモデルです。 スマホの CPU だけで動作し、ネット接続なしで自然な日本語チャットができます。AI Edge Gallery にインポートするだけで試せます。

File	Recipe	Size	GSM8K (n=100)
LFM2.5-1.2B-JP_int8.litertlm	int8 dynamic (linears + embedding; convs float)	1.24 GB	65% (bf16 reference: 63%)
LFM2.5-1.2B-JP_int4.litertlm	int4 blockwise-32 + OCTAV linears, int8 embedding, convs float	736 MB	55%
LFM2.5-1.2B-JP_int4_gpu.litertlm	same int4 recipe, re-exported so it runs on the GPU	736 MB	= int4 (see notes)
LFM2.5-1.2B-JP_int8_gpu.litertlm	same int8 recipe and weights as the int8 file, re-exported so it runs on the GPU	1.24 GB	65% (= int8, measured head to head)

Context (KV cache)	4096 max
Backend	CPU for every file. Two files also run on the GPU with litert-lm ≥ 0.16.0, verified by generation: _int4_gpu (Android OpenCL, macOS, and iOS Metal) and _int8_gpu (Android OpenCL and macOS; not tested on iOS). On iOS set maxNumTokens to 1024 (see the iPhone section — a mismatched value breaks engine creation)
Template	bundled — full chat template (tool-calling supported)
Base model	LiquidAI/LFM2.5-1.2B-JP (LFM Open License v1.0)

Accuracy

GSM8K (English, greedy, 0-shot CoT, max-tokens 1024, n=100, same harness for all rows): PyTorch bf16 63% · LiteRT int8 65% (full parity, +2pt within noise) · int4 55%. English math undersells a Japanese-optimized tune — it is reported for quantization-fidelity transparency, not as this model's headline; both files pass an 8-question sanity gate (7/8, zero degenerate) and Japanese conversation quality was verified by inspection. Note the int8 file quantizes linears only (convs stay float): quantizing this tune's convs costs ~9pt, unlike the Instruct sibling where conv-int8 is free.

Usage

litert-lm run ./LFM2.5-1.2B-JP_int8.litertlm --prompt "海について一文で書いてください。"

GPU — the _int4_gpu file

Pixel 8a (Tensor G3), litert_lm_main built from the litert-lm v0.16.0 release tag, 263-token prompt, generation capped at 256, 3 runs per backend:

Backend	Prefill (263 tok)	Decode	TTFT
GPU (OpenCL)	190–194 tok/s	19.8–21.2 tok/s	1.40–1.44 s
CPU	35–60 tok/s	13.2–20.0 tok/s	4.4–7.5 s

On phone-class hardware the GPU's win is prefill and time-to-first-token (3–6× both); decode is bandwidth-bound and roughly a wash, so long prompts gain far more than long answers. The CPU rows spread because the phone throttles across a run series; the GPU rows repeat to within 2%.

Apple M4 Max (litert-lm benchmark --cache no, litert-lm 0.16.0, -p 256 -d 256 --runs 3, otherwise idle machine; both backends were verified to generate correct text before these numbers were quoted):

Backend	Prefill (256)	Decode	TTFT
GPU	3682 tok/s	316.0 tok/s	0.07 s
CPU	334.6 tok/s	80.3 tok/s	0.78 s

On a desktop GPU the gain is across the board — roughly 11× prefill and 4× decode against the same file on CPU.

GPU — the _int8_gpu file

_int4_gpu above is the small, fast GPU file. _int8_gpu is the accurate one: the same weights as the int8 file — post-hoc int8 on the linears and embedding, convs left float — re-exported on the same litert-torch 0.9.3 lineage so a GPU delegate will take it. Before it existed there was no way to run this repo's int8 weights on a GPU at all; the int8 file reaches 536 of 579 operations and the runtime refuses the partial split.

Galaxy S26 (SM8850, Adreno), litert-lm v0.16.0, 205-token prompt, generation gated separately from the benchmark:

	_int8_gpu	_int4_gpu
Delegation	542/542 on each of the ten multi-token prefill signatures, 501/501 on prefill_1 and 519/519 on decode, zero rejected ops	identical: 542/542 on each of the ten, 501/501 and 519/519, zero rejected ops
Prefill (205 tok)	978.8 tok/s (2 runs: 978.8, 940.4)	1067.8 tok/s (2 runs: 1067.8, 919.0)
Decode	41.9 tok/s (108 tokens generated)	54.8 tok/s (298 tokens generated)
GSM8K (n=100, macOS CPU)	65%, measured on this file	55%, inherited from the int4 file it shares weights with

The two columns were measured a day apart in separate sessions, not side by side. Both are the first of two runs, which is the fair comparison available: this phone slows as it heats, and the two files generate different amounts of text in a benchmark pass, so their second runs are not exposed to the same heat. The spread is shown so you can see it. The decode figures in particular are indicative rather than controlled — the generation lengths differ.

They are a pair, not an upgrade. int4 decodes faster and downloads 41% smaller; int8 answers better. Prefill is close. Pick int4 when tokens per second or download size is the constraint, int8 when accuracy is.

Memory is the surprise. Measured the same way on the same phone — process high-water mark during a generation run — the 1.24 GB int8 file peaked at 556 MB and the 736 MB int4 file at 618 MB. Both figures move a little between runs, and the gap survives that: every reading taken of the int8 file — logged and unlogged — sits below every reading taken of the int4 file. The larger file is the lighter one in use, and neither is anywhere near the several-gigabyte figure a "the GPU holds weights in fp32, so budget about four times the file size" rule of thumb predicts. If you have been sizing GPU headroom that way, measure instead.

Apple M4 Max, litert-lm 0.16.0, -p 256 -d 256 --runs 3 --cache no, each reading taken after a 300 s idle period:

--max-num-tokens	Prefill (256)	Decode	TTFT
1024	5105 tok/s	271.1 tok/s	0.054 s
4096	3946 tok/s	248.5 tok/s	0.069 s

Decode moves with the KV budget here just as it does on CPU — 1024 is the faster setting and a good chat default; the file allows up to 4096. Compare the right rows: the _int4_gpu table earlier in this card was taken at the default 4096 budget, so the like-for-like pair is this file's 4096 row against it, not the 1024 row.

Reproducing these GPU figures. This Mac's GPU slows under back-to-back benchmarking much faster than you would guess. Four runs of one file in a row, at the same effective settings, read 3712 → 3712 → 3529 → 2324 tok/s prefill: flat for two, then a 37% fall, with decode falling further over the same four runs (249 → 247 → 224 → 140). Recovery needs a real pause: 180 s of idle returned prefill to 3712, 300 s to 3946 and 900 s to 3949, so 180 s still leaves you about 6% low. The two rows above were each taken after 300 s, one reading at a time, which is what you need to reproduce them. The _int4_gpu Mac table earlier in this card predates this finding — same --runs 3 protocol, but with no record of what the machine had been doing beforehand — so treat a head-to-head between the two tables as rough rather than exact.

Quality is unchanged by the re-export. Measured head to head against the published int8 file through one harness on one day on the macOS CPU backend, GSM8K n=100 greedy: 65% for both. Asked the same prompt on the CPU backend with greedy decoding, the two files return byte-identical text (144 bytes, character for character) — the re-export changes the graph, not the weights.

GPU — iPhone 17 Pro (iOS Metal)

The engine-creation failure previously tracked in LiteRT-LM#3129 turned out to be a context-sizing issue on the integration side, not a runtime bug. This section is about the _int4_gpu file; the _int8_gpu file has not been tested on iOS. The Metal delegate compiles its kernels against the KV cache that maxNumTokens pre-allocates; a value that does not match the file's exported plan makes engine creation fail with a shader-compile error. On iOS Metal, set maxNumTokens to 1024 (Swift EngineConfig(maxNumTokens:), C API litert_lm_engine_settings_set_max_num_tokens).

iPhone 17 Pro, litert-lm v0.16.0 xcframework, maxNumTokens 1024, single run, device charging: decode 70.0 tok/s, TTFT 91 ms on a short prompt, peak ~560 MB. All Metal kernels compile and generation is coherent.

Run on Android

Install a recent Google AI Edge Gallery, import this repo (or adb push a file and use local import: menu → Models → “+” → From local model file), select the CPU backend, and chat — in Japanese. GPU acceleration needs one of the _gpu files — _int4_gpu or _int8_gpu — and litert-lm ≥ 0.16.0; the Gallery bundles its own runtime, which may lag that version — if its GPU toggle fails there, use CPU.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, CPU backend, -p 256 -d 256 --runs 3 (the tool averages three iterations), warm-up run discarded, otherwise idle machine. Decode on this family depends strongly on the KV budget, so both settings are listed:

Variant	--max-num-tokens	Prefill (256)	Decode	TTFT
int8	1024	1526 tok/s	98.5 tok/s	0.18 s
int4	1024	387 tok/s	119.0 tok/s	0.67 s
int8	4096	1071 tok/s	82.5 tok/s	0.25 s
int4	4096	340 tok/s	79.3 tok/s	0.76 s

Set --max-num-tokens to the smallest value your use case needs — 1024 is a good chat default, and the file allows up to 4096. At 1024 the int4 file decodes fastest; at 4096 the two variants converge.

The _int8_gpu file carries the same weights as the int8 file, and on the CPU backend it measures like it. Run one after the other, -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024: _int8_gpu 396.9 tok/s prefill / 86.3 tok/s decode against int8 413.8 / 87.1. Decode is within 1%. Prefill is noisier: the same GPU file measured 396.9 here and 409.7 on an earlier pass, a 3.1% spread of its own, which is the same order as the 4.1% gap to the control's single reading — so the two files are not cleanly separated on prefill. No separate CPU table is given for the GPU file — use the numbers in this paragraph, which were taken in one sitting; its GPU figures are in the _int8_gpu section.

A caveat about the table above, and it is about the table rather than about any file: those rows were measured by an earlier session, and the same int8 file re-measured with the command just quoted reads 413.8 tok/s prefill against the 1526 shown, and 87.1 tok/s decode against 98.5. What differs between the two measurements has not been run down, so treat the prefill column above as not currently reproducible and the decode column as roughly right.

Use the CPU backend for the int8 and int4 files — neither can create a GPU engine. Both come from the pre-0.9.2 ShortConv export generation, whose prefill graph still carries INT64 ADD/CAST inside Lfm2ShortConv, plus GATHER_ND and a GREATER_EQUAL with const inputs. The GPU delegate takes 536 of the 579 operations and leaves 43 on the CPU, and the runtime then refuses the partial split: Hint fully delegated to single delegate is set, but the graph is not fully delegated. Re-exporting from the post-0.9.2 lineage removes those INT64 ops — that is what the _gpu files are. _int4_gpu runs fully delegated on Android OpenCL, macOS and iOS Metal (with maxNumTokens 1024; see the iPhone section); _int8_gpu carries these same int8 weights and runs fully delegated on Android OpenCL and macOS.

On a Pixel 8a (Tensor G3, CPU backend, measured in AI Edge Gallery) int8 decodes at ~19 tok/s and int4 at ~31 tok/s — on phone-class memory bandwidth the int4 file is about 1.7× faster as well as 41% smaller, so prefer int4 on mid-range devices. Those Android figures are single ship-gate runs, not medians. First device load compiles the graph and can take about a minute; later loads are instant.

Conversion notes

Converted with released litert-torch 0.9.1 with the same exporter fix as the Instruct sibling: the stock LFM2 short-conv block saves its conv state from padded prefill columns, corrupting the first generated token of nearly every reply; the fix derives the chunk's valid length from the attention mask in-graph and gathers the state from the last valid columns (verified token-identical to an exact per-token reference loop). Multi-length prefill signatures (1–1024). Quantization on this finetune: post-hoc int8 on linears + embedding with the convs left float, and post-hoc int4 on linears only. Post-hoc conv quantization breaks generation. Quantizing convs at export time instead costs accuracy here — 9 GSM8K points on this tune, 56 against 65 — so the int8 and int4 files both leave them float.

Why there is a separate _int4_gpu file. The exporter patch above reads the chunk's valid length with index_select and a mask sum, which lower to GATHER_ND and INT64 ops, and GPU delegates reject both — the CPU-only files take 536 of 579 operations on the GPU and the runtime then refuses the partial split. litert-torch 0.9.2 fixed the underlying conv-state bug upstream using valid-token masking and a one-hot matmul state select, which emits neither op. _int4_gpu is that export: litert-torch 0.9.3 + litert-converter 0.3.1, the same int4 recipe and the same weights as the int4 file, plus the executor-metadata section litert-lm ≥ 0.15 needs. It delegates fully on Android OpenCL — 501/501 and 519/519 nodes, zero rejected ops — and runs on the macOS GPU backend. Quality is unchanged: run head to head against the published int4 file on identical prompts, decoding and scoring, both score 38/50 on GSM8K (CPU, n=50), and the GPU file passes an 8-question sanity gate on both backends. The CPU-only files are left exactly as they are rather than replaced, so nothing changes for anyone already using them.

License and changes

Distributed under the LFM Open License v1.0 (see LICENSE, inherited from the base model). Note the license's commercial-use limitation for organizations above US$10M annual revenue. Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified; exporter conv-state fix as described in Conversion notes. This repository is a community conversion and is not affiliated with Liquid AI.