Spaces:
Sleeping
Land PR #15 surgical fixes (Windows safety + human-play hints)
Browse filesMost of PR #15 is BEHIND pr13-revised on the .py source โ Wave-12/13
+ narrative-rewrite + the Phase-1 Controller landed since the PR was
authored, so picking PR #15's agent.py / eval_core.py / prompt_v2.py
/ minimap.py / one_v_one.py wholesale would revert the new tools
(capture_actor / c4_detonate / infiltrate / fire_superweapon),
the RustEnvPool compat layer, and the controller abstraction we
already have (openra_bench/controller.py is byte-identical to PR
#15's version โ same Phase-1 design).
Picked surgically:
- openra_bench/playback.py โ Windows-illegal filename chars in cell
names get sanitised; data: URL prefix stripped from base64 PNGs
before write. Both purely additive.
- openra_bench/scenarios/loader.py โ explicit utf-8 encoding on
load_pack (Windows-locale safety). Dropped PR #15's reversion of
is_map_supported(dict) โ our generator-spec handling is newer.
- openra_bench/human_labeling.py โ added
_objective_regions_from_condition + _initial_type_by_id helpers
and wired them into InteractiveSession (objective-region overlay
on the Play tab, type-by-id inference for own units). PRESERVED
the fog_mode parameter (our addition; PR #15 dropped it).
Made _pb_frame robust: prompt_v2.minimap_b64 wrapped in
try/except with minimap.render_b64 fallback.
- .gitignore โ added node_modules/ (for the E2E test infra landed
in the previous commit) and playback/human-*/ (tester-specific
Playback fixtures).
- README.md โ taken from PR #19 (purely-additive Mission Player
section).
Skipped wholesale (PR #15 strictly behind us):
openra_bench/{agent,eval_core,minimap,prompt_v2,one_v_one,
mapgen,full_playback,handoff,human_study,playback_view,
providers,run_eval,game_knowledge}.py
PR #15's diff for these is mostly deletions of our newer code.
- .gitignore +6 -0
- README.md +43 -0
- openra_bench/human_labeling.py +142 -8
- openra_bench/playback.py +7 -1
- openra_bench/scenarios/loader.py +1 -1
|
@@ -16,3 +16,9 @@ data/runs/paper-v1-**/*.partial
|
|
| 16 |
data/runs/paper-v1-**/*.png
|
| 17 |
data/runs/paper-v1-**/.logs/
|
| 18 |
data/runs/_archive_*/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
data/runs/paper-v1-**/*.png
|
| 17 |
data/runs/paper-v1-**/.logs/
|
| 18 |
data/runs/_archive_*/
|
| 19 |
+
|
| 20 |
+
# Site E2E test infra (PR #19)
|
| 21 |
+
node_modules/
|
| 22 |
+
|
| 23 |
+
# Local Playback fixtures (tester-specific human runs)
|
| 24 |
+
playback/human-*/
|
|
@@ -21,6 +21,8 @@ Standardized benchmark and leaderboard for AI agents playing Red Alert through [
|
|
| 21 |
- **Evaluation harness**: Automated N-game benchmarking with metrics collection
|
| 22 |
- **OpenEnv rubrics**: Composable scoring (win/loss, military efficiency, economy)
|
| 23 |
- **Replay verification**: Replay files linked to leaderboard entries
|
|
|
|
|
|
|
| 24 |
|
| 25 |
## Quick Start
|
| 26 |
|
|
@@ -94,6 +96,47 @@ The Gradio app exposes these API endpoints (Gradio 5+ SSE protocol):
|
|
| 94 |
| `submit_with_replay` | Submit JSON + replay file |
|
| 95 |
| `filter_leaderboard` | Query/filter leaderboard data |
|
| 96 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 97 |
## Scoring
|
| 98 |
|
| 99 |
| Component | Weight | Description |
|
|
|
|
| 21 |
- **Evaluation harness**: Automated N-game benchmarking with metrics collection
|
| 22 |
- **OpenEnv rubrics**: Composable scoring (win/loss, military efficiency, economy)
|
| 23 |
- **Replay verification**: Replay files linked to leaderboard entries
|
| 24 |
+
- **Mission Player**: Static game-like website for browsing, annotating, and reviewing scenarios
|
| 25 |
+
- **Bilingual**: English and Chinese scenario instructions generated deterministically
|
| 26 |
|
| 27 |
## Quick Start
|
| 28 |
|
|
|
|
| 96 |
| `submit_with_replay` | Submit JSON + replay file |
|
| 97 |
| `filter_leaderboard` | Query/filter leaderboard data |
|
| 98 |
|
| 99 |
+
## Mission Player (Static Site)
|
| 100 |
+
|
| 101 |
+
A game-like mission selection and annotation website in `site/`. No framework, no build step -- a single HTML file deployable to GitHub Pages.
|
| 102 |
+
|
| 103 |
+
### For players / annotators
|
| 104 |
+
|
| 105 |
+
Open `site/index.html` via any HTTP server:
|
| 106 |
+
|
| 107 |
+
```bash
|
| 108 |
+
cd site && python3 -m http.server 8765
|
| 109 |
+
# Open http://localhost:8765/index.html
|
| 110 |
+
```
|
| 111 |
+
|
| 112 |
+
Workflow: browse scenario cards, pick a mission, read bilingual objectives (EN/ZH toggle), switch difficulty (easy/medium/hard), annotate the map with point/region tools, tag and add notes, mark complete, navigate to next mission, export annotations as JSON.
|
| 113 |
+
|
| 114 |
+
### For maintainers
|
| 115 |
+
|
| 116 |
+
Generate or refresh static data after scenario changes:
|
| 117 |
+
|
| 118 |
+
```bash
|
| 119 |
+
python site/generate.py # generate scenarios.json + map thumbnails
|
| 120 |
+
python site/generate.py --dry-run # print counts without writing
|
| 121 |
+
```
|
| 122 |
+
|
| 123 |
+
Map thumbnails require the Rust engine wheel (`openra_train`). Without it, the site works with a placeholder map area; annotations still work on the placeholder.
|
| 124 |
+
|
| 125 |
+
Deploy by copying `site/index.html` and `site/public/` to any static host.
|
| 126 |
+
|
| 127 |
+
See `docs/IMPLEMENTATION_NOTES.md` for full details.
|
| 128 |
+
|
| 129 |
+
### Running tests
|
| 130 |
+
|
| 131 |
+
```bash
|
| 132 |
+
# Data pipeline + coverage invariant tests (Python)
|
| 133 |
+
python -m pytest tests/test_site.py tests/test_app.py -v
|
| 134 |
+
|
| 135 |
+
# E2E DOM interaction tests (Node.js + jsdom)
|
| 136 |
+
npm install # first time only
|
| 137 |
+
node tests/test_site_e2e.mjs
|
| 138 |
+
```
|
| 139 |
+
|
| 140 |
## Scoring
|
| 141 |
|
| 142 |
| Component | Weight | Description |
|
|
@@ -47,6 +47,126 @@ TurnActions = "list[HumanAction]"
|
|
| 47 |
InputSource = "Callable[[dict], list[HumanAction]] | Sequence[list[HumanAction]]"
|
| 48 |
|
| 49 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
# โโ Pixel โ cell transforms โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
| 51 |
|
| 52 |
|
|
@@ -492,6 +612,13 @@ class InteractiveSession:
|
|
| 492 |
self._env = self._pool.acquire()
|
| 493 |
self._adapter = RustObsAdapter()
|
| 494 |
self._adapter.observe(self._env.reset(seed=seed))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 495 |
self._forbidden = {
|
| 496 |
str(t).lower() for t in (compiled.forbidden_tools or [])
|
| 497 |
}
|
|
@@ -571,7 +698,10 @@ class InteractiveSession:
|
|
| 571 |
def render_state(self) -> dict:
|
| 572 |
"""The current observation โ the SAME render_state an LLM agent
|
| 573 |
is shown for this scenario."""
|
| 574 |
-
|
|
|
|
|
|
|
|
|
|
| 575 |
|
| 576 |
def status(self) -> dict:
|
| 577 |
"""Turn / outcome / done summary for the UI."""
|
|
@@ -636,16 +766,20 @@ class InteractiveSession:
|
|
| 636 |
self._pb_terrain = terrain_png_for(
|
| 637 |
self.compiled.scenario.base_map
|
| 638 |
)
|
| 639 |
-
|
|
|
|
|
|
|
| 640 |
|
| 641 |
-
|
| 642 |
-
|
| 643 |
-
|
| 644 |
-
|
|
|
|
|
|
|
| 645 |
if png is None:
|
| 646 |
-
from .
|
| 647 |
|
| 648 |
-
png =
|
| 649 |
return png
|
| 650 |
except Exception: # noqa: BLE001
|
| 651 |
return None
|
|
|
|
| 47 |
InputSource = "Callable[[dict], list[HumanAction]] | Sequence[list[HumanAction]]"
|
| 48 |
|
| 49 |
|
| 50 |
+
# โโ Scenario hints for human play โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
| 51 |
+
|
| 52 |
+
_REGION_PREDICATES = {
|
| 53 |
+
"reach_region",
|
| 54 |
+
"units_in_region_gte",
|
| 55 |
+
"units_of_type_in_region_gte",
|
| 56 |
+
"all_units_in_region",
|
| 57 |
+
"building_in_region",
|
| 58 |
+
"enemy_key_buildings_destroyed_in_region",
|
| 59 |
+
}
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def _condition_node(cond: Any) -> dict:
|
| 63 |
+
if cond is None:
|
| 64 |
+
return {}
|
| 65 |
+
if isinstance(cond, dict):
|
| 66 |
+
return cond
|
| 67 |
+
return dict(getattr(cond, "__pydantic_extra__", None) or {})
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def _objective_regions_from_condition(cond: Any) -> list[dict]:
|
| 71 |
+
"""Extract authored objective regions from a win-condition tree."""
|
| 72 |
+
out: list[dict] = []
|
| 73 |
+
|
| 74 |
+
def walk(node: Any) -> None:
|
| 75 |
+
data = _condition_node(node)
|
| 76 |
+
if not data:
|
| 77 |
+
return
|
| 78 |
+
for child in data.get("all_of") or []:
|
| 79 |
+
walk(child)
|
| 80 |
+
for child in data.get("any_of") or []:
|
| 81 |
+
walk(child)
|
| 82 |
+
if "not" in data:
|
| 83 |
+
walk(data["not"])
|
| 84 |
+
then = data.get("then")
|
| 85 |
+
if isinstance(then, dict):
|
| 86 |
+
for child in then.get("clauses") or []:
|
| 87 |
+
walk(child)
|
| 88 |
+
seq = data.get("waypoint_sequence")
|
| 89 |
+
if isinstance(seq, dict):
|
| 90 |
+
default_r = seq.get("radius", 6)
|
| 91 |
+
for i, point in enumerate(seq.get("points") or [], start=1):
|
| 92 |
+
if not isinstance(point, dict):
|
| 93 |
+
continue
|
| 94 |
+
if "x" in point and "y" in point:
|
| 95 |
+
out.append({
|
| 96 |
+
"x": int(point["x"]),
|
| 97 |
+
"y": int(point["y"]),
|
| 98 |
+
"radius": float(point.get("radius", default_r)),
|
| 99 |
+
"label": str(point.get("label") or f"W{i}"),
|
| 100 |
+
})
|
| 101 |
+
for key in _REGION_PREDICATES:
|
| 102 |
+
v = data.get(key)
|
| 103 |
+
if not isinstance(v, dict) or "x" not in v or "y" not in v:
|
| 104 |
+
continue
|
| 105 |
+
label = v.get("label")
|
| 106 |
+
if not label:
|
| 107 |
+
if key == "units_in_region_gte":
|
| 108 |
+
label = f">={int(v.get('n', 1))} units"
|
| 109 |
+
elif key == "units_of_type_in_region_gte":
|
| 110 |
+
label = f">={int(v.get('n', 1))} {v.get('type')}"
|
| 111 |
+
elif key == "building_in_region":
|
| 112 |
+
label = str(v.get("type") or "building")
|
| 113 |
+
else:
|
| 114 |
+
label = key.replace("_", " ")
|
| 115 |
+
out.append({
|
| 116 |
+
"x": int(v["x"]),
|
| 117 |
+
"y": int(v["y"]),
|
| 118 |
+
"radius": float(v.get("radius", 3)),
|
| 119 |
+
"label": str(label),
|
| 120 |
+
})
|
| 121 |
+
|
| 122 |
+
walk(cond)
|
| 123 |
+
seen = set()
|
| 124 |
+
unique = []
|
| 125 |
+
for r in out:
|
| 126 |
+
key = (r["x"], r["y"], r["radius"], r["label"])
|
| 127 |
+
if key in seen:
|
| 128 |
+
continue
|
| 129 |
+
seen.add(key)
|
| 130 |
+
unique.append(r)
|
| 131 |
+
return unique
|
| 132 |
+
|
| 133 |
+
|
| 134 |
+
def _initial_type_by_id(scenario: Any, render_state: dict) -> dict[str, str]:
|
| 135 |
+
"""Infer own unit types from authored initial placements.
|
| 136 |
+
|
| 137 |
+
Rust currently omits own-unit actor types in some observations, but
|
| 138 |
+
it preserves deterministic actor ids and initial cells. This maps the
|
| 139 |
+
initial visible units back to the scenario placements so the human UI
|
| 140 |
+
can distinguish 1tnk/2tnk/etc. after the units move.
|
| 141 |
+
"""
|
| 142 |
+
expected: dict[tuple[int, int], list[str]] = {}
|
| 143 |
+
for actor in getattr(scenario, "actors", []) or []:
|
| 144 |
+
if str(getattr(actor, "owner", "")).lower() != "agent":
|
| 145 |
+
continue
|
| 146 |
+
pos = getattr(actor, "position", None)
|
| 147 |
+
if not pos or len(pos) < 2:
|
| 148 |
+
continue
|
| 149 |
+
cell = (int(pos[0]), int(pos[1]))
|
| 150 |
+
atype = str(getattr(actor, "type", "") or "").lower()
|
| 151 |
+
count = int(getattr(actor, "count", 1) or 1)
|
| 152 |
+
expected.setdefault(cell, []).extend([atype] * count)
|
| 153 |
+
|
| 154 |
+
observed: dict[tuple[int, int], list[str]] = {}
|
| 155 |
+
for unit in render_state.get("units_summary") or []:
|
| 156 |
+
if not isinstance(unit, dict) or unit.get("id") is None:
|
| 157 |
+
continue
|
| 158 |
+
cell = (int(unit.get("cell_x", 0)), int(unit.get("cell_y", 0)))
|
| 159 |
+
observed.setdefault(cell, []).append(str(unit["id"]))
|
| 160 |
+
|
| 161 |
+
out: dict[str, str] = {}
|
| 162 |
+
for cell, types in expected.items():
|
| 163 |
+
ids = sorted(observed.get(cell, []))
|
| 164 |
+
for uid, atype in zip(ids, types):
|
| 165 |
+
if atype:
|
| 166 |
+
out[uid] = atype
|
| 167 |
+
return out
|
| 168 |
+
|
| 169 |
+
|
| 170 |
# โโ Pixel โ cell transforms โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
| 171 |
|
| 172 |
|
|
|
|
| 612 |
self._env = self._pool.acquire()
|
| 613 |
self._adapter = RustObsAdapter()
|
| 614 |
self._adapter.observe(self._env.reset(seed=seed))
|
| 615 |
+
self._adapter.type_by_id.update(
|
| 616 |
+
_initial_type_by_id(compiled.scenario, self._adapter.render_state())
|
| 617 |
+
)
|
| 618 |
+
self._objective_regions = (
|
| 619 |
+
_objective_regions_from_condition(compiled.win_condition)
|
| 620 |
+
if compiled.objective_coords == "exact" else []
|
| 621 |
+
)
|
| 622 |
self._forbidden = {
|
| 623 |
str(t).lower() for t in (compiled.forbidden_tools or [])
|
| 624 |
}
|
|
|
|
| 698 |
def render_state(self) -> dict:
|
| 699 |
"""The current observation โ the SAME render_state an LLM agent
|
| 700 |
is shown for this scenario."""
|
| 701 |
+
rs = self._adapter.render_state()
|
| 702 |
+
if self._objective_regions:
|
| 703 |
+
rs["objective_regions"] = list(self._objective_regions)
|
| 704 |
+
return rs
|
| 705 |
|
| 706 |
def status(self) -> dict:
|
| 707 |
"""Turn / outcome / done summary for the UI."""
|
|
|
|
| 766 |
self._pb_terrain = terrain_png_for(
|
| 767 |
self.compiled.scenario.base_map
|
| 768 |
)
|
| 769 |
+
png = None
|
| 770 |
+
try:
|
| 771 |
+
from .prompt_v2 import minimap_b64 as _v2_mm
|
| 772 |
|
| 773 |
+
png = _v2_mm(
|
| 774 |
+
rs, self._pb_terrain, self._pb_explored,
|
| 775 |
+
constant_colors=self.compiled.level in ("easy", "medium"),
|
| 776 |
+
)
|
| 777 |
+
except Exception: # noqa: BLE001
|
| 778 |
+
pass
|
| 779 |
if png is None:
|
| 780 |
+
from .minimap import render_b64
|
| 781 |
|
| 782 |
+
png = render_b64(rs, self._pb_terrain)
|
| 783 |
return png
|
| 784 |
except Exception: # noqa: BLE001
|
| 785 |
return None
|
|
@@ -45,7 +45,11 @@ class Playback:
|
|
| 45 |
"""Per-episode recorder. Create one per (scenario, seed)."""
|
| 46 |
|
| 47 |
def __init__(self, root: str | Path, cell: str, seed: int):
|
| 48 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
self.dir.mkdir(parents=True, exist_ok=True)
|
| 50 |
self._turns_fh = open(self.dir / "turns.jsonl", "w")
|
| 51 |
self._n = 0
|
|
@@ -67,6 +71,8 @@ class Playback:
|
|
| 67 |
self._n += 1
|
| 68 |
if minimap_png_b64:
|
| 69 |
try:
|
|
|
|
|
|
|
| 70 |
(self.dir / f"minimap_turn{turn:03d}.png").write_bytes(
|
| 71 |
base64.b64decode(minimap_png_b64)
|
| 72 |
)
|
|
|
|
| 45 |
"""Per-episode recorder. Create one per (scenario, seed)."""
|
| 46 |
|
| 47 |
def __init__(self, root: str | Path, cell: str, seed: int):
|
| 48 |
+
safe_cell = "".join(
|
| 49 |
+
ch if ch not in '<>:"/\\|?*' and ord(ch) >= 32 else "_"
|
| 50 |
+
for ch in str(cell)
|
| 51 |
+
)
|
| 52 |
+
self.dir = Path(root) / safe_cell / f"seed{seed}"
|
| 53 |
self.dir.mkdir(parents=True, exist_ok=True)
|
| 54 |
self._turns_fh = open(self.dir / "turns.jsonl", "w")
|
| 55 |
self._n = 0
|
|
|
|
| 71 |
self._n += 1
|
| 72 |
if minimap_png_b64:
|
| 73 |
try:
|
| 74 |
+
if "," in minimap_png_b64 and minimap_png_b64.lstrip().startswith("data:"):
|
| 75 |
+
minimap_png_b64 = minimap_png_b64.split(",", 1)[1]
|
| 76 |
(self.dir / f"minimap_turn{turn:03d}.png").write_bytes(
|
| 77 |
base64.b64decode(minimap_png_b64)
|
| 78 |
)
|
|
@@ -50,7 +50,7 @@ def resolve_map_path(base_map: str) -> Path | None:
|
|
| 50 |
def load_pack(path: str | Path) -> ScenarioPack:
|
| 51 |
"""Parse and validate a single pack YAML."""
|
| 52 |
path = Path(path)
|
| 53 |
-
with open(path) as f:
|
| 54 |
data = yaml.safe_load(f)
|
| 55 |
try:
|
| 56 |
return ScenarioPack(**data)
|
|
|
|
| 50 |
def load_pack(path: str | Path) -> ScenarioPack:
|
| 51 |
"""Parse and validate a single pack YAML."""
|
| 52 |
path = Path(path)
|
| 53 |
+
with open(path, encoding="utf-8") as f:
|
| 54 |
data = yaml.safe_load(f)
|
| 55 |
try:
|
| 56 |
return ScenarioPack(**data)
|