JEV-HRI-InternVL3-2B

A typed decision model for human-robot interaction (HRI). Given a robot-camera image in which the detected people are marked with a box and an ID (P1, P2, ...), it answers closed questions about each person, such as which interaction state is P2 in?, and returns only a calibrated probability for every option, in one forward pass. It never writes free text, so there is nothing to parse and no invalid answer.

It is OpenGVLab/InternVL3-2B-hf distilled from Qwen/Qwen3-VL-32B-Instruct with a LoRA adapter (merged into these weights). It follows the "System 1" design of autotrust/JEV-27B-VL, applied to robot-camera images.

primitive	output	example in configs/hri_intent.yaml
choice	value, confidence, probabilities	interaction state of P2: ATTENTIVE / AVAILABLE / BUSY / ENGAGED / MOVING
noul	value (bool), probability = P(true)	would P2 welcome being approached right now?
score	expected value Σ i·p_i and the distribution	how costly would it be to interrupt P2? (0-4)

Quick start

git clone https://huggingface.co/ejbejaranos/JEV-HRI-InternVL3-2B && cd JEV-HRI-InternVL3-2B
pip install "transformers>=5.0" accelerate torch pillow pyyaml huggingface_hub num2words   # tested with 5.19

from PIL import Image
from jev_hri.hub import load_jev_hri
from jev_hri.annotate import mark_persons
from jev_hri.policy import select_target
import yaml

jev = load_jev_hri("ejbejaranos/JEV-HRI-InternVL3-2B")                      # bf16, one GPU
img, pids = mark_persons(Image.open("frame.jpg"), boxes)   # boxes from your person detector
out, timing = jev.predict(img, pids)
out["P1"]["intent"]   # {'type': 'choice', 'value': 'ATTENTIVE', 'confidence': 0.81, 'probabilities': {...}}
select_target(out, yaml.safe_load(open("configs/hri_intent.yaml"))["policy"])   # whom should the robot address?

Questions, options, criteria and the robot policy live in configs/hri_intent.yaml: edit them without touching code. Options you add or rename work zero-shot; the model was trained on the questions in configs/hri_distill.yaml.

How it was made (the pipeline)

1. Images. 8,000 COCO train2017 images with 1-6 people (no crowds, no sports or food scenes), resized to 640 px, every person marked with a coloured box and an ID from left to right (distill/make_data.py). No human labels.
2. Teacher. Qwen/Qwen3-VL-32B-Instruct, zero-shot, answers 9 questions for every person: the 5-state interaction intent, openness, interruption cost, plus perception questions (looking at the robot, phone, walking, in a group, sitting, distance). Its full probability distribution over the options is kept as a soft label (distill/teacher_label.py).
3. Student. OpenGVLab/InternVL3-2B-hf with LoRA r=32, alpha=64 on every linear layer of the language model and the lm_head; the vision encoder is frozen. Loss = KL(teacher ‖ student) over the option tokens of each question + 0.1 × format loss (−log P(any option token)), 2 epochs (distill/train.py). Tied input/output embeddings are untied first so that merging the lm_head LoRA does not change the input embeddings. Single image tile. The image is not split into tiles (one 512/448 px view, ~13x fewer image tokens than the base model's default); this setting is saved in the processor config, and the zero-shot rows of the table use it too, so the comparison is fair.
4. Calibration. One temperature per question type fitted on held-out images against the teacher (calibration.json: choice 1.06, noul 1.16, score 1.05); option tokens per type in decision_head.json.
5. Export and checks. LoRA merged into bf16 weights; the merged model is checked against the adapter and all the metrics below are measured on the merged model.

Prompt protocol hri-bare-v1 (no chat template; image first so it can be cached once per frame):

<image>
[state] {context} Detected people: P1, P2.
[kind] choice
[question] Which interaction state best describes person P2?
[options]
A) ATTENTIVE: looking at the robot, eye contact, ...
B) AVAILABLE: ...
[decision]:            <- read the logits of A, B, ... (noul: true/false; score: 0..5)

At inference the image and context are encoded once into the KV cache and all (person × question) rows are decided in one batched forward over that cache (jev_hri/engine.py).

Results

Fidelity to the teacher on 320 held-out images: top-1 agreement 0.875 (before training 0.687), KL 0.1690 (before 0.5968).

HRI evaluation set: 50 COCO val2017 street / station / corridor images, 137 people, 5 interaction states (intent question). Latency: median per image (all people, 3 questions each, ~8 decision rows), one NVIDIA A100 80GB, bf16, 3 repetitions per image. Generative latencies come from the original-model benchmark (generation and Jev interleaved); for the Jev rows of the zero-shot and distilled models of the same size the latency comes from a paired run (one after the other on an otherwise idle GPU) when that run exists. "clear" = excluding the labels marked as doubtful. Peak VRAM: peak memory allocated by PyTorch (weights + activations + KV cache) on the 8 evaluation images with the most people (5-6 people), measured separately for each mode (reports/measure_vram.py); add ~0.5 GB for the CUDA context.

model	size	method	accuracy	macro-F1	accuracy (clear)	latency (ms)	peak VRAM (GB)
InternVL3-2B-hf (original)	2B	generative JSON	0.270	0.236	0.287	1418	4.3
InternVL3-2B-hf zero-shot Jev (no training)	2B	Jev (1 forward)	0.365	0.255	0.377	59	5.1
JEV-HRI-InternVL3-2B (this model)	2B	Jev (1 forward)	0.635	0.482	0.697	74	5.1
Qwen3-VL-32B-Instruct (teacher, zero-shot Jev)	32B	Jev (1 forward)	0.679	0.531	0.721	543	69.7
Qwen3-VL-32B-Instruct (teacher, generative)	32B	generative JSON	0.642	0.479	0.697	3752	67.1
autotrust/JEV-27B-VL (System 1, reference)	27B	Jev (1 forward)	0.672	0.578	0.730	1903	–
autotrust/JEV-27B-VL (System 2, generative)	27B	generative JSON	0.715	0.594	0.762	6213	–

Other models of the family:

model	size	method	accuracy	macro-F1	accuracy (clear)	latency (ms)	peak VRAM (GB)
JEV-HRI-InternVL3-1B-hf	1B	Jev (1 forward)	0.628	0.472	0.656	52	2.7
JEV-HRI-LFM2-VL-1.6B	1.6B	Jev (1 forward)	0.620	0.460	0.648	57	3.9
JEV-HRI-LFM2-VL-450M	450M	Jev (1 forward)	0.635	0.504	0.648	39	1.5
JEV-HRI-Qwen3-VL-2B	2B	Jev (1 forward)	0.628	0.511	0.664	96	6.6
JEV-HRI-Qwen3-VL-4B	4B	Jev (1 forward)	0.672	0.529	0.721	134	11.7
JEV-HRI-Qwen3-VL-8B	8B	Jev (1 forward)	0.657	0.509	0.697	170	19.5
JEV-HRI-SmolVLM2-2.2B-Instruct	2.2B	Jev (1 forward)	0.599	0.467	0.631	71	5.8
JEV-HRI-SmolVLM2-256M-Video-Instruct	256M	Jev (1 forward)	0.599	0.443	0.623	67	0.9
JEV-HRI-SmolVLM2-500M-Video-Instruct	500M	Jev (1 forward)	0.562	0.380	0.607	69	1.5

Tests

pytest tests -k "not gpu"                                   # protocol, calibration, HRI policy (CPU)
JEV_MODEL=ejbejaranos/JEV-HRI-InternVL3-2B pytest tests -k gpu      # end-to-end on a COCO image (GPU)

Limitations

• Evaluation labels are not yet human-verified. Claude (2 pasadas: completa + revision ampliada de dudosas). PENDIENTE revision humana; 15 of 137 labels are marked as doubtful. 137 people is a small test set: differences of a few points are within noise.
• The student learns the teacher, including its mistakes; its accuracy is bounded by Qwen3-VL-32B-Instruct's.
• COCO photos are not robot-camera footage (height, motion blur, lens). Validate on your robot before acting on the probabilities, and pick confidence thresholds on your own data.
• People must be marked in the image (box + ID); the model does not detect people itself.
• English prompts only. Calibration was fitted against the teacher, not against human labels.

Acknowledgements

The typed-decision design (one forward pass, option tokens, per-kind calibration, LoRA on lm_head) follows autotrust/JEV-27B-VL. Base and teacher models by the Qwen team.