VFM-4B

VFM-4B verifies natural-language criteria against a context and target text. It uses a Qwen3-4B backbone and a three-class head, pretrained on CriteriaBank for Runtime Verification of Multiple Natural Language Criteria for Agent Governance (NeurIPS 2026).

Each criterion receives YES (satisfied), NO (not satisfied), or N/A (inapplicable or insufficient evidence). Criteria are scored independently over a shared prefix, without text generation. Code and training recipes.

Use

pip install git+https://github.com/spitis/vfm
from vfm import VFM

vfm = VFM.from_pretrained("spitis/VFM-4B")
verdicts = vfm.verify(
    context="Describe today's weather.",
    target="It is sunny today.",
    criteria=["Does the response mention sunshine?", "Does the response mention rain?"],
)
for v in verdicts:
    print(v["criterion"], v["verdict"], v["p_yes"], v["p_no"], v["p_na"])

context accepts a string or chat messages; target is appended as the next turn. Outputs also include logits and p_yes_binary = p_yes / (p_yes + p_no). Use verify_batch for separate traces.

Keep criteria short and atomic. Prefixes are left-truncated at 4,096 tokens by default (max_length); criterion blocks at 64 tokens, including chat-template tokens. verify reuses the prefix cache across chunks of 64 criteria (chunk_size).

Training and results

Full-parameter pretraining uses 393,531 model-labelled CriteriaBank rows, cross-entropy plus KL to the frozen base model (ฮป = 1), AdamW, and batch size 128. The released checkpoint is the one saved at step 2,000 of the 3,075 pretraining steps.

Paper results below use the released weights for zero-shot evaluation; ternary benchmarks use their YES/NO subset.

Benchmark Zero-shot ROC-AUC
InFoBench 0.942
Synthetic governance 0.927
ContractNLI 0.899
INSURE-Dial 0.671
BeaverTails 0.779
IFEval 0.704
PRM800K 0.782
LLM-AggreFact 0.857

After benchmark finetuning (mean ยฑ standard deviation over three seeds):

Benchmark Metric Score
Synthetic governance Accuracy (%) 88.8 ยฑ 1.1
Governance transfer Accuracy (%) 77.7 ยฑ 0.8
INSURE-Dial Accuracy (%) 86.5 ยฑ 1.6
BeaverTails F1 (%) 82.6 ยฑ 0.4
IFEval Balanced accuracy (%) 95.1 ยฑ 0.6
PRM800K Balanced accuracy (%) 82.1 ยฑ 0.6
LLM-AggreFact Balanced accuracy (%) 76.3 ยฑ 0.8
ContractNLI Macro-F1 (%) 84.2 ยฑ 0.6
DynaBench Trace-level FAIL F1 0.885 ยฑ 0.011

Passing VFM's top three rules to a Gemma-4-26B-A4B-it judge raises DynaBench FAIL F1 to 0.946 ยฑ 0.005.

Finetuned results require separate benchmark training. Governance uses dev selection; INSURE-Dial uses five-fold checkpoint selection over real calls.

Limitations

Finetune and tune decision thresholds for your domain. Zero-shot calibration and N/A prediction are weak. Labels are model-generated; evaluations are English-only and text-only. Even the best finetuned verifier approves about 20% of gold violations on INSURE-Dial. Use for monitoring and escalation, not standalone enforcement.

License: Apache-2.0, including the Qwen3-4B base model. Tokenizer files are unchanged from Qwen3.

Citation

@inproceedings{pitis2026vfm,
  title     = {Runtime Verification of Multiple Natural Language Criteria for Agent Governance},
  author    = {Pitis, Silviu and Alamdari, Parand A. and Tang, Jessica and Klassen, Toryn Q. and McIlraith, Sheila A.},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}
Downloads last month
20
Safetensors
Model size
4B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for spitis/VFM-4B

Finetuned
Qwen/Qwen3-4B
Finetuned
(1,304)
this model

Dataset used to train spitis/VFM-4B