Pelican SVG Environment

Blind vector drawing, scored. The model is asked for an SVG of an animal riding a vehicle and never sees the result, so it has to hold the spatial arrangement in its head and emit coordinates for it. This is the check Simon Willison popularised as “generate an SVG of a pelican riding a bicycle”, turned into something you can run repeatedly, at scale, and train against.

What this is, and what it is not

Not a model ranking. Simon Willison, whose prompt this is, is explicit that “the correlation between pelican performance and actual model quality has been mostly severed now”, and ends with “don’t go using pelicans to compare models” (Kimi K3, and what we can still learn from the pelican benchmark). Our own numbers agree with him: on the canonical task, across 139 scored samples from 7 frontier models, 136 scored a perfect 1.000 on the structural layer and 47 scored a perfect 1.000 overall. It is saturated.

Not a contamination study either. Dylan Castillo already ran that, across an 8 by 6 grid and 1008 SVGs from 7 frontier models, and found little evidence that labs optimise for the pelican. The extra animals and vehicles here are ours and serve a smaller purpose: checking that the scorer measures drawing ability rather than one memorised picture. Only pelican_bicycle is Simon’s, and it is the default.

What it is is a reproducible, executable target. The same code that scores a frontier model over an API scores a 0.6B model mid-training, the reward is dense enough to run GRPO against, and the whole thing deploys as a Space. That is the part a prompt cannot do:

• examples/pelican_svg_eval.py scores any set of models through HF Inference Providers.
• examples/pelican_svg_grpo.py trains against it with TRL, runnable on HF Jobs.

The interesting question stops being “which model draws the best pelican” and becomes “can a small model be taught to, and does it learn to draw or to game the scorer”.

The task catalogue

6 animals by 5 vehicles, so 30 tasks. All three scoring layers adapt to whichever one is served, so nothing about the pipeline is special to the pelican:

• The judge’s checklist is generated per task from the features below. A pelican is asked about a throat pouch, a capybara about a blunt snout, an octopus about curling tentacles.
• The geometry layer reads wheels from the vehicle, and drops the wheel-pair checks entirely for a single-wheeled vehicle rather than failing them.
• The anti-cheat terms are derived from the task, so axolotl_scooter forbids axolotl, salamander, amphibian, scooter, kick scooter and not the pelican’s words.

Animals

Vehicles

How much of the grid has actually been run

All 30 combinations have been scored end to end by a live model. None crashed and none was rejected by the gate. The counts below are how much traffic each cell has actually seen, which is a different question from whether it works:

Read a 1 as “the plumbing works”, not as a result. It says the task builds, the prompt renders, the gate admits the reply, the geometry resolves against that vehicle’s wheel count and the judge returns a checklist. It says nothing about how well the scoring behaves on that combination, and it cannot: one sample has no variance to report.

Across the 30-task sweep, reward ranged 0.362 to 1.000 with a median of 0.838, structure averaged 0.905 and the judged semantic score 0.785. The weakest cell was hedgehog_tandem-bicycle at 0.362, where the judge would not accept the animal. Three of the 30 were corrected upward by the <use> fix described below, capybara_unicycle most of all, from 0.333 to 1.000 on structure.

pelican_bicycle is Simon Willison’s original prompt and the default. It is also the only task the GRPO example trains on: examples/pelican_svg_grpo.py pins it deliberately, since that script is a small demonstration that the environment is trainable rather than an attempt to teach a model the whole grid.

Scoring

Three layers, ordered by cost. Each earns the right to run the next, so a run against a model producing garbage stays cheap.

1. Gate, free, pass or fail

Rejects anything that is not an honest attempt to draw. A rejected submission scores zero and never reaches the judge.

The two cheats worth knowing about:

• Embedded raster. A base64 PNG inside an <image> renders as a completely convincing picture. A vision judge scores it highly. It is only visible in the source, which is why source inspection runs before anything looks at pixels.
• Text label. <text>a pelican riding a bicycle</text> also renders, and also fools a careless judge.

2. Structure, free, weight 0.35

Geometry over the parsed document with transforms applied and coordinates normalised by the viewBox, so nothing depends on the units the model picked. Seven checks: wheel count, wheels similar in size, level, sensibly far apart, something spanning between them, a rider present, and the rider proportionate to the wheelbase.

Wheels are found as a row of similar-sized round shapes at similar height, with the lowest row preferred. Not by picking the biggest circle: see the correction below for what that cost.

Deliberately shape-level. It cannot tell a pelican from a capybara and does not try. What it can do is answer questions no judge should be trusted with because the answer is arithmetic.

3. Vision judge, one call pair per sample, weight 0.65

• A blind caption. The judge is shown the picture and asked what it is, with no mention of the task. If it says “a pelican on a bicycle” unprompted, that is the strongest available evidence, and a leading question cannot inflate it. In testing this mattered: asked “is this a pelican riding a bicycle?” the judge said yes; asked to describe the same image cold, it said “a cartoon duck”.
• A binary feature checklist. Individual yes-or-no questions rather than a 0-10 rating, because individual features are reproducible and a global rating drifts.

Failing the riding posture costs 75% of the semantic score rather than all of it. Posture is a spatial relation and the least reliable item on the checklist, so hanging the whole component on that one boolean gives the judge’s worst call the loudest vote.

What training against it revealed

Four GRPO runs through TRL on HF Jobs, 80 steps each. Three with the judge disabled, so the reward is deterministic arithmetic, and one with --judge-in-reward. The training reward climbed in every one of them, which turns out to say very little.

What says more is re-scoring each checkpoint on 24 fresh samples with the judge on, and comparing against the untrained model. Every run measures that first, so there are five independent measurements of the starting point:

The judge-off runs finish 3.8 standard deviations above the untrained structural mean. The judge-on run finishes inside the untrained band, 0.8 below the mean, and its semantic score went down. Turning the judge on drops the weight on structure from 1.0 to 0.35, so it dilutes the only term that teaches anything and replaces it with one that teaches nothing: across all 48 probe drawings, before and after, the judge never once answered yes to the riding question.

It learned two real things: to emit well-formed SVG, and to build bicycle-like geometry, both on samples it never trained on. It did not learn to draw a bird. Twice, independently, which is why this is stated as a finding rather than an anecdote.

Looking at the probe drawings rather than the table sharpens it. Between before and after, samples with no detectable structure at all fell from 9 of 24 to 2 of 24 and the median structural score rose from 0.43 to 0.71, so something real was learned. But the single best semantic score in the probe went down, 0.045 to 0.028, and the word “bird” appears in one blind caption before training and none after. The output got more measurable and slightly less bird-like. Weights and all 48 probe drawings are public for both the judge-off run, pelican-svg-grpo-Qwen3-1.7B, and the judge-on run, pelican-svg-grpo-Qwen3-1.7B-judged, with both training curves at pelican-svg-grpo-curves.

Read that carefully, because the obvious label is the wrong one. The judge was switched off during training, so the semantic score was never part of the reward. The policy optimised exactly what it was paid for. This is not a model subverting its objective, it is a demonstration that the deterministic layer alone is an insufficient proxy for the task: two round shapes of equal size at equal height, a bar between them and any blob above the axle line scores 1.000 on all seven checks. Four primitives and no animal, and it is the exact figure the trained policy converged on.

Whether a model would game a reward the judge is part of was the obvious follow-up, and the answer is that at this size the question does not arise: the judged term is almost always zero, so there is nothing to game. In 24 fresh samples the untrained 1.7B scored non-zero on the semantic component twice, and GRPO takes its gradient from the variance inside a group of completions, so a term that is zero for every member of a group contributes nothing however heavily it is weighted. The same judge scores non-zero on 138 of 139 frontier samples. Before spending GPU time on a judged reward, measure how often it is non-zero on the base model you are about to train.

examples/pelican_svg_grpo.py ships the probe that produced all of this: it scores fresh samples with the judge before and after, and flags a run where the structural gain outpaces the judged gain. That ratio was 23 to 1 on the first 1.7B run and 45 to 1 on the second.

Usage

from envs.pelican_svg_env import PelicanSvgAction, PelicanSvgEnv

with PelicanSvgEnv(base_url="http://localhost:8000") as env:
    observation = env.reset(task_id="capybara_unicycle").observation
    result = env.step(PelicanSvgAction(response=my_model(observation.prompt)))
    print(result.reward, result.observation.feedback)

Pin the task with task_id for any benchmark run. Without it every reset draws a fresh task and two models are never asked the same question.

Run the server locally:

PYTHONPATH=src:envs uv run uvicorn pelican_svg_env.server.app:app --port 8000

Configuration

Environment variables: HF_TOKEN for the judge, PELICAN_SVG_JUDGE_MODEL to change it from Qwen/Qwen2.5-VL-72B-Instruct, PELICAN_SVG_DISABLE_JUDGE=1 to force offline scoring.

With no judge configured, structure carries the full weight. A judge that is configured but fails is treated differently: the semantic component stays at zero and judged is set false on the observation, so a harness can drop or retry the sample. Renormalising there would mean a submission scores higher precisely when nobody could look at it, which makes breaking the judge a winning strategy.

Two failure modes worth knowing about

Both bugs the scorer has had took the same shape: a correct drawing expressed in a less common way was read as a wrong drawing. Wheel detection anchored on the largest round shape, so a rider drawn bigger than the wheels became the anchor and the real wheels were discarded. And the shape walker descended into <defs> while ignoring <use>, so a model that wrote its wheel once and placed it twice had the template counted at the origin and the instances skipped.

Neither was visible in the aggregate numbers. Both were obvious the moment the low-scoring drawings were rendered and looked at, which is what --save-svgs on examples/pelican_svg_eval.py is for. fixtures/wheels_as_paths.svg and fixtures/wheels_via_use.svg are the regression cases.

Known limitations

• The judge model matters more than anything else. On one identical drawing, Qwen2.5-VL-72B scored the semantic component 0.639 and Qwen3-VL-30B scored it 0.875. That 0.24 spread dwarfs every other source of noise, so a published number is meaningless without naming the judge. Within a fixed judge the scoring is stable: 8 repeat runs of the same drawing through the deployed Space returned sd 0.0000 and a character-identical caption. Across longer spans one checklist item has been seen to flip, worth about 0.04 of final reward.
• Structure cannot identify a species. By design, but it means a model that draws an excellent generic blob on an excellent bicycle collects the full structural weight.

Rendering

resvg via resvg-py, pinned. It ships self-contained wheels for manylinux, musllinux, macOS and Windows, so the container needs no system libraries. It is deterministic across runs, and it refuses to resolve external entities or fetch remote hrefs, which matters when the input is untrusted model output.

Development

PYTHONPATH=src:envs uv run pytest tests/envs/test_pelican_svg_env.py -v

The tests need no network: the judge is exercised through a stub client, which is also the only way to cover the failure paths that matter.

Test fixtures live in fixtures/, a graded corpus from a complete scene down to the two cheats. Thresholds in the gate and the structural layer were calibrated against it and then re-checked against real model output, which caught two cases the hand-written corpus could not: an edge-density check that rejected legitimate filled-shape drawings, and single-wheeled vehicles being scored against bicycle geometry.