yxc20098 commited on
Commit
a04d20a
ยท
1 Parent(s): 1c57350

Land PR #15 surgical fixes (Windows safety + human-play hints)

Browse files

Most of PR #15 is BEHIND pr13-revised on the .py source โ€” Wave-12/13
+ narrative-rewrite + the Phase-1 Controller landed since the PR was
authored, so picking PR #15's agent.py / eval_core.py / prompt_v2.py
/ minimap.py / one_v_one.py wholesale would revert the new tools
(capture_actor / c4_detonate / infiltrate / fire_superweapon),
the RustEnvPool compat layer, and the controller abstraction we
already have (openra_bench/controller.py is byte-identical to PR
#15's version โ€” same Phase-1 design).

Picked surgically:
- openra_bench/playback.py โ€” Windows-illegal filename chars in cell
names get sanitised; data: URL prefix stripped from base64 PNGs
before write. Both purely additive.
- openra_bench/scenarios/loader.py โ€” explicit utf-8 encoding on
load_pack (Windows-locale safety). Dropped PR #15's reversion of
is_map_supported(dict) โ€” our generator-spec handling is newer.
- openra_bench/human_labeling.py โ€” added
_objective_regions_from_condition + _initial_type_by_id helpers
and wired them into InteractiveSession (objective-region overlay
on the Play tab, type-by-id inference for own units). PRESERVED
the fog_mode parameter (our addition; PR #15 dropped it).
Made _pb_frame robust: prompt_v2.minimap_b64 wrapped in
try/except with minimap.render_b64 fallback.
- .gitignore โ€” added node_modules/ (for the E2E test infra landed
in the previous commit) and playback/human-*/ (tester-specific
Playback fixtures).
- README.md โ€” taken from PR #19 (purely-additive Mission Player
section).

Skipped wholesale (PR #15 strictly behind us):
openra_bench/{agent,eval_core,minimap,prompt_v2,one_v_one,
mapgen,full_playback,handoff,human_study,playback_view,
providers,run_eval,game_knowledge}.py
PR #15's diff for these is mostly deletions of our newer code.

.gitignore CHANGED
@@ -16,3 +16,9 @@ data/runs/paper-v1-**/*.partial
16
  data/runs/paper-v1-**/*.png
17
  data/runs/paper-v1-**/.logs/
18
  data/runs/_archive_*/
 
 
 
 
 
 
 
16
  data/runs/paper-v1-**/*.png
17
  data/runs/paper-v1-**/.logs/
18
  data/runs/_archive_*/
19
+
20
+ # Site E2E test infra (PR #19)
21
+ node_modules/
22
+
23
+ # Local Playback fixtures (tester-specific human runs)
24
+ playback/human-*/
README.md CHANGED
@@ -21,6 +21,8 @@ Standardized benchmark and leaderboard for AI agents playing Red Alert through [
21
  - **Evaluation harness**: Automated N-game benchmarking with metrics collection
22
  - **OpenEnv rubrics**: Composable scoring (win/loss, military efficiency, economy)
23
  - **Replay verification**: Replay files linked to leaderboard entries
 
 
24
 
25
  ## Quick Start
26
 
@@ -94,6 +96,47 @@ The Gradio app exposes these API endpoints (Gradio 5+ SSE protocol):
94
  | `submit_with_replay` | Submit JSON + replay file |
95
  | `filter_leaderboard` | Query/filter leaderboard data |
96
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
97
  ## Scoring
98
 
99
  | Component | Weight | Description |
 
21
  - **Evaluation harness**: Automated N-game benchmarking with metrics collection
22
  - **OpenEnv rubrics**: Composable scoring (win/loss, military efficiency, economy)
23
  - **Replay verification**: Replay files linked to leaderboard entries
24
+ - **Mission Player**: Static game-like website for browsing, annotating, and reviewing scenarios
25
+ - **Bilingual**: English and Chinese scenario instructions generated deterministically
26
 
27
  ## Quick Start
28
 
 
96
  | `submit_with_replay` | Submit JSON + replay file |
97
  | `filter_leaderboard` | Query/filter leaderboard data |
98
 
99
+ ## Mission Player (Static Site)
100
+
101
+ A game-like mission selection and annotation website in `site/`. No framework, no build step -- a single HTML file deployable to GitHub Pages.
102
+
103
+ ### For players / annotators
104
+
105
+ Open `site/index.html` via any HTTP server:
106
+
107
+ ```bash
108
+ cd site && python3 -m http.server 8765
109
+ # Open http://localhost:8765/index.html
110
+ ```
111
+
112
+ Workflow: browse scenario cards, pick a mission, read bilingual objectives (EN/ZH toggle), switch difficulty (easy/medium/hard), annotate the map with point/region tools, tag and add notes, mark complete, navigate to next mission, export annotations as JSON.
113
+
114
+ ### For maintainers
115
+
116
+ Generate or refresh static data after scenario changes:
117
+
118
+ ```bash
119
+ python site/generate.py # generate scenarios.json + map thumbnails
120
+ python site/generate.py --dry-run # print counts without writing
121
+ ```
122
+
123
+ Map thumbnails require the Rust engine wheel (`openra_train`). Without it, the site works with a placeholder map area; annotations still work on the placeholder.
124
+
125
+ Deploy by copying `site/index.html` and `site/public/` to any static host.
126
+
127
+ See `docs/IMPLEMENTATION_NOTES.md` for full details.
128
+
129
+ ### Running tests
130
+
131
+ ```bash
132
+ # Data pipeline + coverage invariant tests (Python)
133
+ python -m pytest tests/test_site.py tests/test_app.py -v
134
+
135
+ # E2E DOM interaction tests (Node.js + jsdom)
136
+ npm install # first time only
137
+ node tests/test_site_e2e.mjs
138
+ ```
139
+
140
  ## Scoring
141
 
142
  | Component | Weight | Description |
openra_bench/human_labeling.py CHANGED
@@ -47,6 +47,126 @@ TurnActions = "list[HumanAction]"
47
  InputSource = "Callable[[dict], list[HumanAction]] | Sequence[list[HumanAction]]"
48
 
49
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
50
  # โ”€โ”€ Pixel โ‡„ cell transforms โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
51
 
52
 
@@ -492,6 +612,13 @@ class InteractiveSession:
492
  self._env = self._pool.acquire()
493
  self._adapter = RustObsAdapter()
494
  self._adapter.observe(self._env.reset(seed=seed))
 
 
 
 
 
 
 
495
  self._forbidden = {
496
  str(t).lower() for t in (compiled.forbidden_tools or [])
497
  }
@@ -571,7 +698,10 @@ class InteractiveSession:
571
  def render_state(self) -> dict:
572
  """The current observation โ€” the SAME render_state an LLM agent
573
  is shown for this scenario."""
574
- return self._adapter.render_state()
 
 
 
575
 
576
  def status(self) -> dict:
577
  """Turn / outcome / done summary for the UI."""
@@ -636,16 +766,20 @@ class InteractiveSession:
636
  self._pb_terrain = terrain_png_for(
637
  self.compiled.scenario.base_map
638
  )
639
- from .prompt_v2 import minimap_b64 as _v2_mm
 
 
640
 
641
- png = _v2_mm(
642
- rs, self._pb_terrain, self._pb_explored,
643
- constant_colors=self.compiled.level in ("easy", "medium"),
644
- )
 
 
645
  if png is None:
646
- from .agent import _render_minimap_b64
647
 
648
- png = _render_minimap_b64(rs, self._pb_terrain)
649
  return png
650
  except Exception: # noqa: BLE001
651
  return None
 
47
  InputSource = "Callable[[dict], list[HumanAction]] | Sequence[list[HumanAction]]"
48
 
49
 
50
+ # โ”€โ”€ Scenario hints for human play โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
51
+
52
+ _REGION_PREDICATES = {
53
+ "reach_region",
54
+ "units_in_region_gte",
55
+ "units_of_type_in_region_gte",
56
+ "all_units_in_region",
57
+ "building_in_region",
58
+ "enemy_key_buildings_destroyed_in_region",
59
+ }
60
+
61
+
62
+ def _condition_node(cond: Any) -> dict:
63
+ if cond is None:
64
+ return {}
65
+ if isinstance(cond, dict):
66
+ return cond
67
+ return dict(getattr(cond, "__pydantic_extra__", None) or {})
68
+
69
+
70
+ def _objective_regions_from_condition(cond: Any) -> list[dict]:
71
+ """Extract authored objective regions from a win-condition tree."""
72
+ out: list[dict] = []
73
+
74
+ def walk(node: Any) -> None:
75
+ data = _condition_node(node)
76
+ if not data:
77
+ return
78
+ for child in data.get("all_of") or []:
79
+ walk(child)
80
+ for child in data.get("any_of") or []:
81
+ walk(child)
82
+ if "not" in data:
83
+ walk(data["not"])
84
+ then = data.get("then")
85
+ if isinstance(then, dict):
86
+ for child in then.get("clauses") or []:
87
+ walk(child)
88
+ seq = data.get("waypoint_sequence")
89
+ if isinstance(seq, dict):
90
+ default_r = seq.get("radius", 6)
91
+ for i, point in enumerate(seq.get("points") or [], start=1):
92
+ if not isinstance(point, dict):
93
+ continue
94
+ if "x" in point and "y" in point:
95
+ out.append({
96
+ "x": int(point["x"]),
97
+ "y": int(point["y"]),
98
+ "radius": float(point.get("radius", default_r)),
99
+ "label": str(point.get("label") or f"W{i}"),
100
+ })
101
+ for key in _REGION_PREDICATES:
102
+ v = data.get(key)
103
+ if not isinstance(v, dict) or "x" not in v or "y" not in v:
104
+ continue
105
+ label = v.get("label")
106
+ if not label:
107
+ if key == "units_in_region_gte":
108
+ label = f">={int(v.get('n', 1))} units"
109
+ elif key == "units_of_type_in_region_gte":
110
+ label = f">={int(v.get('n', 1))} {v.get('type')}"
111
+ elif key == "building_in_region":
112
+ label = str(v.get("type") or "building")
113
+ else:
114
+ label = key.replace("_", " ")
115
+ out.append({
116
+ "x": int(v["x"]),
117
+ "y": int(v["y"]),
118
+ "radius": float(v.get("radius", 3)),
119
+ "label": str(label),
120
+ })
121
+
122
+ walk(cond)
123
+ seen = set()
124
+ unique = []
125
+ for r in out:
126
+ key = (r["x"], r["y"], r["radius"], r["label"])
127
+ if key in seen:
128
+ continue
129
+ seen.add(key)
130
+ unique.append(r)
131
+ return unique
132
+
133
+
134
+ def _initial_type_by_id(scenario: Any, render_state: dict) -> dict[str, str]:
135
+ """Infer own unit types from authored initial placements.
136
+
137
+ Rust currently omits own-unit actor types in some observations, but
138
+ it preserves deterministic actor ids and initial cells. This maps the
139
+ initial visible units back to the scenario placements so the human UI
140
+ can distinguish 1tnk/2tnk/etc. after the units move.
141
+ """
142
+ expected: dict[tuple[int, int], list[str]] = {}
143
+ for actor in getattr(scenario, "actors", []) or []:
144
+ if str(getattr(actor, "owner", "")).lower() != "agent":
145
+ continue
146
+ pos = getattr(actor, "position", None)
147
+ if not pos or len(pos) < 2:
148
+ continue
149
+ cell = (int(pos[0]), int(pos[1]))
150
+ atype = str(getattr(actor, "type", "") or "").lower()
151
+ count = int(getattr(actor, "count", 1) or 1)
152
+ expected.setdefault(cell, []).extend([atype] * count)
153
+
154
+ observed: dict[tuple[int, int], list[str]] = {}
155
+ for unit in render_state.get("units_summary") or []:
156
+ if not isinstance(unit, dict) or unit.get("id") is None:
157
+ continue
158
+ cell = (int(unit.get("cell_x", 0)), int(unit.get("cell_y", 0)))
159
+ observed.setdefault(cell, []).append(str(unit["id"]))
160
+
161
+ out: dict[str, str] = {}
162
+ for cell, types in expected.items():
163
+ ids = sorted(observed.get(cell, []))
164
+ for uid, atype in zip(ids, types):
165
+ if atype:
166
+ out[uid] = atype
167
+ return out
168
+
169
+
170
  # โ”€โ”€ Pixel โ‡„ cell transforms โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
171
 
172
 
 
612
  self._env = self._pool.acquire()
613
  self._adapter = RustObsAdapter()
614
  self._adapter.observe(self._env.reset(seed=seed))
615
+ self._adapter.type_by_id.update(
616
+ _initial_type_by_id(compiled.scenario, self._adapter.render_state())
617
+ )
618
+ self._objective_regions = (
619
+ _objective_regions_from_condition(compiled.win_condition)
620
+ if compiled.objective_coords == "exact" else []
621
+ )
622
  self._forbidden = {
623
  str(t).lower() for t in (compiled.forbidden_tools or [])
624
  }
 
698
  def render_state(self) -> dict:
699
  """The current observation โ€” the SAME render_state an LLM agent
700
  is shown for this scenario."""
701
+ rs = self._adapter.render_state()
702
+ if self._objective_regions:
703
+ rs["objective_regions"] = list(self._objective_regions)
704
+ return rs
705
 
706
  def status(self) -> dict:
707
  """Turn / outcome / done summary for the UI."""
 
766
  self._pb_terrain = terrain_png_for(
767
  self.compiled.scenario.base_map
768
  )
769
+ png = None
770
+ try:
771
+ from .prompt_v2 import minimap_b64 as _v2_mm
772
 
773
+ png = _v2_mm(
774
+ rs, self._pb_terrain, self._pb_explored,
775
+ constant_colors=self.compiled.level in ("easy", "medium"),
776
+ )
777
+ except Exception: # noqa: BLE001
778
+ pass
779
  if png is None:
780
+ from .minimap import render_b64
781
 
782
+ png = render_b64(rs, self._pb_terrain)
783
  return png
784
  except Exception: # noqa: BLE001
785
  return None
openra_bench/playback.py CHANGED
@@ -45,7 +45,11 @@ class Playback:
45
  """Per-episode recorder. Create one per (scenario, seed)."""
46
 
47
  def __init__(self, root: str | Path, cell: str, seed: int):
48
- self.dir = Path(root) / cell.replace("/", "_") / f"seed{seed}"
 
 
 
 
49
  self.dir.mkdir(parents=True, exist_ok=True)
50
  self._turns_fh = open(self.dir / "turns.jsonl", "w")
51
  self._n = 0
@@ -67,6 +71,8 @@ class Playback:
67
  self._n += 1
68
  if minimap_png_b64:
69
  try:
 
 
70
  (self.dir / f"minimap_turn{turn:03d}.png").write_bytes(
71
  base64.b64decode(minimap_png_b64)
72
  )
 
45
  """Per-episode recorder. Create one per (scenario, seed)."""
46
 
47
  def __init__(self, root: str | Path, cell: str, seed: int):
48
+ safe_cell = "".join(
49
+ ch if ch not in '<>:"/\\|?*' and ord(ch) >= 32 else "_"
50
+ for ch in str(cell)
51
+ )
52
+ self.dir = Path(root) / safe_cell / f"seed{seed}"
53
  self.dir.mkdir(parents=True, exist_ok=True)
54
  self._turns_fh = open(self.dir / "turns.jsonl", "w")
55
  self._n = 0
 
71
  self._n += 1
72
  if minimap_png_b64:
73
  try:
74
+ if "," in minimap_png_b64 and minimap_png_b64.lstrip().startswith("data:"):
75
+ minimap_png_b64 = minimap_png_b64.split(",", 1)[1]
76
  (self.dir / f"minimap_turn{turn:03d}.png").write_bytes(
77
  base64.b64decode(minimap_png_b64)
78
  )
openra_bench/scenarios/loader.py CHANGED
@@ -50,7 +50,7 @@ def resolve_map_path(base_map: str) -> Path | None:
50
  def load_pack(path: str | Path) -> ScenarioPack:
51
  """Parse and validate a single pack YAML."""
52
  path = Path(path)
53
- with open(path) as f:
54
  data = yaml.safe_load(f)
55
  try:
56
  return ScenarioPack(**data)
 
50
  def load_pack(path: str | Path) -> ScenarioPack:
51
  """Parse and validate a single pack YAML."""
52
  path = Path(path)
53
+ with open(path, encoding="utf-8") as f:
54
  data = yaml.safe_load(f)
55
  try:
56
  return ScenarioPack(**data)