Instructions to use spitis/VFM-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use spitis/VFM-4B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="spitis/VFM-4B", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("spitis/VFM-4B", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
VFM-4B
VFM-4B verifies natural-language criteria against a context and target text. It uses a Qwen3-4B backbone and a three-class head, pretrained on CriteriaBank for Runtime Verification of Multiple Natural Language Criteria for Agent Governance (NeurIPS 2026).
Each criterion receives YES (satisfied), NO (not satisfied), or N/A (inapplicable or insufficient evidence). Criteria are scored independently over a shared prefix, without text generation. Code and training recipes.
Use
pip install git+https://github.com/spitis/vfm
from vfm import VFM
vfm = VFM.from_pretrained("spitis/VFM-4B")
verdicts = vfm.verify(
context="Describe today's weather.",
target="It is sunny today.",
criteria=["Does the response mention sunshine?", "Does the response mention rain?"],
)
for v in verdicts:
print(v["criterion"], v["verdict"], v["p_yes"], v["p_no"], v["p_na"])
context accepts a string or chat messages; target is appended as the next turn. Outputs also include logits and p_yes_binary = p_yes / (p_yes + p_no). Use verify_batch for separate traces.
Keep criteria short and atomic. Prefixes are left-truncated at 4,096 tokens by default (max_length); criterion blocks at 64 tokens, including chat-template tokens. verify reuses the prefix cache across chunks of 64 criteria (chunk_size).
Training and results
Full-parameter pretraining uses 393,531 model-labelled CriteriaBank rows, cross-entropy plus KL to the frozen base model (ฮป = 1), AdamW, and batch size 128. The released checkpoint is the one saved at step 2,000 of the 3,075 pretraining steps.
Paper results below use the released weights for zero-shot evaluation; ternary benchmarks use their YES/NO subset.
| Benchmark | Zero-shot ROC-AUC |
|---|---|
| InFoBench | 0.942 |
| Synthetic governance | 0.927 |
| ContractNLI | 0.899 |
| INSURE-Dial | 0.671 |
| BeaverTails | 0.779 |
| IFEval | 0.704 |
| PRM800K | 0.782 |
| LLM-AggreFact | 0.857 |
After benchmark finetuning (mean ยฑ standard deviation over three seeds):
| Benchmark | Metric | Score |
|---|---|---|
| Synthetic governance | Accuracy (%) | 88.8 ยฑ 1.1 |
| Governance transfer | Accuracy (%) | 77.7 ยฑ 0.8 |
| INSURE-Dial | Accuracy (%) | 86.5 ยฑ 1.6 |
| BeaverTails | F1 (%) | 82.6 ยฑ 0.4 |
| IFEval | Balanced accuracy (%) | 95.1 ยฑ 0.6 |
| PRM800K | Balanced accuracy (%) | 82.1 ยฑ 0.6 |
| LLM-AggreFact | Balanced accuracy (%) | 76.3 ยฑ 0.8 |
| ContractNLI | Macro-F1 (%) | 84.2 ยฑ 0.6 |
| DynaBench | Trace-level FAIL F1 | 0.885 ยฑ 0.011 |
Passing VFM's top three rules to a Gemma-4-26B-A4B-it judge raises DynaBench FAIL F1 to 0.946 ยฑ 0.005.
Finetuned results require separate benchmark training. Governance uses dev selection; INSURE-Dial uses five-fold checkpoint selection over real calls.
Limitations
Finetune and tune decision thresholds for your domain. Zero-shot calibration and N/A prediction are weak. Labels are model-generated; evaluations are English-only and text-only. Even the best finetuned verifier approves about 20% of gold violations on INSURE-Dial. Use for monitoring and escalation, not standalone enforcement.
License: Apache-2.0, including the Qwen3-4B base model. Tokenizer files are unchanged from Qwen3.
Citation
@inproceedings{pitis2026vfm,
title = {Runtime Verification of Multiple Natural Language Criteria for Agent Governance},
author = {Pitis, Silviu and Alamdari, Parand A. and Tang, Jessica and Klassen, Toryn Q. and McIlraith, Sheila A.},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026}
}
- Downloads last month
- 20