chenyeikai/VisProbe-Qwen3-VL-2B-LoRA

VisProbe · Qwen3-VL-2B LoRA (probabilistic visual classifier)

A LoRA adapter that turns Qwen/Qwen3-VL-2B-Instruct into a probabilistic closed-set visual classifier. At inference the model does not generate free-form text: the answer-position logits are restricted to the candidate codes (A/B/C/D, or A=false/B=true) and renormalised with a softmax, producing an auditable probability distribution.

Released as part of VisProbe — code, data pipeline, and the full experiment log: https://github.com/Hanzoghos/visprobe

Results — frozen baseline vs this adapter

Same test sets, same scoring protocol, single RTX 3090, bf16. Accuracy (and absolute gain):

Test set	Baseline	After LoRA	Gain
SNLI-VE (visual entailment, 3-way, n=21,482)	0.6918	0.8998	+20.80 pp
VQAv2 yes/no (seeded 1/3 subset, n=26,846)	0.6618	0.8558	+19.40 pp
Visual7W (4-way multiple choice, n=6,695)	0.8512	0.8875	+3.63 pp
Macro average (equal weight)	0.7350	0.8810	+14.6 pp

Full accuracy/ECE/Brier tables and loss curves: see results/SUMMARY.md and EXPERIMENTS.md in the VisProbe repository. This is the largest gain of all released VisProbe adapters — the best accuracy-per-compute point.

Training

Item	Value
Data	99,999 mixed examples (33,333 per dataset, stratified by label, seed=42)
Epochs / steps	1 / 25,000 (batch size 4; per-example micro-batches with gradient accumulation)
LoRA	r=8, alpha=16, dropout=0.05, bias=none
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (language decoder only; vision tower frozen)
Optimisation	lr 1e-4, 3% linear warmup + cosine decay, grad clip 1.0, AdamW
Precision	bfloat16, gradient checkpointing (use_reentrant=False)
Checkpoint	final-from-best: lowest recent-window training loss
Hardware / time	1× NVIDIA RTX 3090 24 GB / about 7.6 h
Software	transformers 4.57.1, peft

Trainable parameters: ~18.6M (adapter weights only, 34.9 MB in safetensors).

How to use

With VisProbe (recommended — gives probabilities, not text)

from vlm.scoring.vlm_scorer import VLMScorer

# The adapter id is downloaded from the Hub automatically; the base model is Qwen3-VL-2B-Instruct
scorer = VLMScorer("qwen3vl2b", adapter="chenyeikai/VisProbe-Qwen3-VL-2B-LoRA")
result = scorer.score(image, fields, context_text=context)
# result["fields"][name]["scores"] -> {"A": p, "B": p, ...} candidate probabilities

Generic PEFT loading

import torch
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3-VL-2B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "chenyeikai/VisProbe-Qwen3-VL-2B-LoRA")
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-2B-Instruct")

Scope & limitations

• The adapter is specialised for closed-set choice / boolean questions rendered with the VisProbe prompt schema (single-token answer codes). It is not a general-purpose chat upgrade.
• Training data are three academic visual-reasoning benchmarks; in-domain behaviour is measured by the tables above, out-of-domain behaviour is not.
• Post-finetune calibration error (ECE) changes slightly (still small). VisProbe ships a temperature-calibration utility, but calibrated-temperature results are not part of this release.
• VQAv2 results use a seeded random 1/3 test subset (standard error ≈ ±0.5 pp).

License

Adapter weights: Apache-2.0, following the base model Qwen3-VL-2B-Instruct. Datasets remain under their upstream licenses (Flickr30k / COCO / Visual Genome based).