README.md

---
library_name: custom
base_model: Qwen/Qwen3.5-9B
datasets:
- NU-World-Model-Embodied-AI/phyground
pipeline_tag: video-text-to-text
language:
- en
license: other
license_name: phyjudge-adapter-terms-unspecified
tags:
- lora
- peft
- video-evaluation
- physical-ai
- world-model
- judge
- qwen3.5
- arxiv:2605.10806
---

# PhyJudge-9B: Physical Plausibility Judge for Generated Video

[Project page](https://phyground.github.io/) ·
[Paper](https://arxiv.org/abs/2605.10806) ·
[Evaluation code](https://github.com/NU-World-Model-Embodied-AI/PhyGround) ·
[PhyGround dataset](https://huggingface.co/datasets/NU-World-Model-Embodied-AI/phyground)

PhyJudge-9B is a PEFT/LoRA adapter for
[Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B), trained to evaluate
generated videos on three general dimensions and 13 observable physical laws.
It returns one criterion-specific score per call on a **1–5 scale**, where a
higher score indicates a more plausible or better-aligned video under that
criterion.

PhyJudge is an evaluation model. It does not generate video and it is not a
generic text-only language-model adapter.

## Evaluation dimensions

### General dimensions

- `SA`: semantic alignment with the prompt;
- `PTV`: physical temporal validity;
- `persistence`: stability of object existence, identity, and attributes.

### Physical-law dimensions

| Domain | Laws |
| --- | --- |
| Solid-body mechanics | `gravity`, `inertia`, `momentum`, `impenetrability`, `collision`, `material` |
| Fluid dynamics | `buoyancy`, `displacement`, `flow_dynamics`, `boundary_interaction`, `fluid_continuity` |
| Optics | `reflection`, `shadow` |

Each call evaluates either one general dimension or one applicable law. A full
PhyGround evaluation normally makes 5–7 calls per video: three general
dimensions plus 2–4 laws associated with the prompt.

## Files

| File | Purpose |
| --- | --- |
| `adapter_config.json` | PEFT/LoRA configuration and base-model identifier |
| `adapter_model.safetensors` | LoRA weights, approximately 174 MB |
| `additional_config.json` | Additional ms-swift training settings |
| `training_args.json` | Training hyperparameters |
| `subq+human.yaml` | Prompt and 1–5 scoring templates |
| `infer.py` | Standalone video-scoring script |

## Setup

```bash
pip install -U \
  "transformers>=5.2.0" \
  "peft>=0.19.1" accelerate pyyaml \
  "qwen-vl-utils[decord]" \
  huggingface_hub

hf download NU-World-Model-Embodied-AI/phyjudge-9B \
  --local-dir ./phyjudge-9B
```

The card's prompt/parser path was checked with Transformers 5.2.0 and PEFT
0.19.1. Loading the 9B base model in bf16 needs roughly 24 GB of GPU memory in the
authors' setup. Actual memory use depends on software versions, video length,
frame sampling, and resolution. Reduce `--fps` or `--max-pixels` if needed;
doing so can also change the score.

## Quickstart

Score semantic alignment:

```bash
python ./phyjudge-9B/infer.py \
  --adapter-dir ./phyjudge-9B \
  --video /path/to/video.mp4 \
  --caption "A ball rolls down a ramp and knocks over a block." \
  --metric SA
```

Score one physical law:

```bash
python ./phyjudge-9B/infer.py \
  --adapter-dir ./phyjudge-9B \
  --video /path/to/video.mp4 \
  --caption "A ball rolls down a ramp and knocks over a block." \
  --law gravity
```

The script loads the base model recorded in `adapter_config.json`, attaches the
LoRA adapter, samples the video at 2 FPS by default, and performs deterministic
decoding. Output is a JSON object:

```json
{
  "key": "gravity",
  "score": 4,
  "raw": "{\"gravity\": 4}"
}
```

Use `--print-prompt` to inspect the rendered prompt. Use `--criteria` only when
you intentionally want to override the released per-law criterion; modified
criteria are no longer directly comparable with the published protocol.

## Programmatic use

```python
from pathlib import Path
import torch

from infer import (
    build_messages,
    build_prompt,
    decode_generated,
    load_model,
    load_yaml,
    parse_score,
    prepare_inputs,
)

processor, model, adapter_dir = load_model(
    "NU-World-Model-Embodied-AI/phyjudge-9B",
    dtype=torch.bfloat16,
    device_map="auto",
)
cfg = load_yaml(adapter_dir / "subq+human.yaml")

system, user, key = build_prompt(
    cfg,
    caption="A ball rolls down a ramp and knocks over a block.",
    law="gravity",
)
messages = build_messages(system, user, Path("video.mp4"))
inputs = prepare_inputs(
    processor,
    messages,
    next(model.parameters()).device,
    fps=2.0,
    max_pixels=360 * 640,
)

with torch.inference_mode():
    generated = model.generate(**inputs, max_new_tokens=64, do_sample=False)

raw = decode_generated(processor, inputs, generated)
print({"key": key, "score": parse_score(raw, key), "raw": raw})
```

## Training details

- Base model: Qwen/Qwen3.5-9B.
- Method: LoRA via PEFT, rank 32, alpha 64, dropout 0.05.
- Vision encoder: frozen.
- Precision: bf16 with gradient checkpointing.
- Optimizer: AdamW, learning rate `1e-4`, cosine schedule.
- Duration: 1 epoch / 294 steps on the `subq+human` split.
- Framework versions recorded by the release: ms-swift 4.1.2, PEFT 0.19.1,
  DeepSpeed ZeRO-2.

See `training_args.json`, `additional_config.json`, and `adapter_config.json`
for the released configuration.

## Reported agreement with human rankings

Under the paper's model-ranking protocol, PhyJudge-9B has **3.3% aggregate
relative bias** against human rankings; Gemini 3.1 Pro has 16.6% under the same
protocol. This is a task-specific benchmark result, not a claim that
PhyJudge-9B is generally more capable than a frontier multimodal model.

## Intended use

PhyJudge-9B is intended for:

- research evaluation of generated videos under the released PhyGround rubric;
- per-law failure diagnosis and model comparison;
- reproducing or extending the PhyGround benchmark;
- screening results before targeted human review.

It should not be used as the sole authority for safety-critical, scientific,
legal, or real-world physical judgments.

## Limitations

- Scores inherit coverage limits from PhyGround's 13-law taxonomy and prompt
  distribution.
- The judge may miss brief events because it samples frames rather than
  continuously perceiving every instant.
- Scores can change with video decoding, sampling FPS, resolution, prompt
  wording, or criterion wording.
- The model can return malformed JSON or an incorrect score; `infer.py` reports
  `null` when parsing fails.
- A trained evaluator can reproduce annotation preferences and biases. Human
  audits remain appropriate for new models and out-of-distribution content.

## License

**Adapter-specific reuse terms have not yet been separately specified in this
repository.** The metadata therefore uses `license: other` rather than implying
a license grant that is not present. The Qwen3.5-9B base model is released
under Apache-2.0, but its license does not by itself determine the license of
this adapter. Maintainers should add a dedicated `LICENSE` file and replace the
metadata above once the intended adapter license is approved.

## Citation

```bibtex
@misc{lin2026phygroundbenchmarkingphysicalreasoning,
  title         = {PhyGround: Benchmarking Physical Reasoning in Generative World Models},
  author        = {Juyi Lin and Arash Akbari and Yumei He and Lin Zhao and Haichao Zhang and Arman Akbari and Xingchen Xu and Zoe Y. Lu and Enfu Nan and Hokin Deng and Edmund Yeh and Sarah Ostadabbas and Yun Fu and Jennifer Dy and Pu Zhao and Yanzhi Wang},
  year          = {2026},
  eprint        = {2605.10806},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2605.10806}
}
```