rleo/vision-rs-models

vision-rs models — SigLIP 2 base-patch16-224 (ONNX, uint8 input)

Models ready for vision-rs (vision-server): photo and search-query vectors in one 768-d space, for text→photo search in a photo library.

file	what	changes vs. the source
siglip2_base_224_vision.u8.onnx	image tower	preprocessing added in front of the graph (see below); weights unchanged
siglip2_base_224_text.onnx	text tower	none (copied as is)
siglip2_tokenizer.json	Gemma tokenizer	none (copied as is)

Source and licence

• Weights: google/siglip2-base-patch16-224 by Google, Apache License 2.0 (see LICENSE).
• ONNX export: onnx-community/siglip2-base-patch16-224-ONNX (revision ba1f3b0843), files onnx/vision_model.onnx, onnx/text_model.onnx, tokenizer.json.

Modifications (Apache-2.0 §4b)

siglip2_base_224_vision.u8.onnx was produced by scripts/prepare_models.py (scripts/uint8_frontend.py): the input becomes image_u8, uint8 NHWC BGR 224x224, and five nodes are prepended — Cast to float, BGR→RGB Gather, (x − 127.5) / 127.5, NHWC→NCHW Transpose — feeding the original pixel_values input; the batch dimension is symbolic. Nothing else changed: outputs match the original model fed with the same normalisation (cosine 0.9999995).

Usage

• Image: resize the photo to 224x224 (area interpolation, aspect ratio not kept), BGR uint8, batch N; read pooler_output and L2-normalise.
• Text: lowercase, tokenize, pad / truncate to 64 tokens (pad id 0, EOS appended by the tokenizer); input_ids int64 [N, 64]; read pooler_output and L2-normalise.
• Rank photos for a query by dot product.

With vision-rs:

vision-server serve --models <this repo, downloaded>   # POST /v1/embeddings

Quality and precision

COCO val2017 text→image retrieval (5000 photos, 25k captions), measured with vision-rs: R@1 52.0, R@5 76.2, R@10 83.8. With TensorRT, keep LayerNorm in FP32: plain FP16 drops R@1 to 48.7 at the same speed.