When the agent passes its own test: nine trials on Mercury 2.5.
In these trials, an agent-written smoke check passed after saving a PNG without rendering the requested 3D scene. We changed the validators and runner after examining that failure and the later trial logs.
The last two game runs passed the framebuffer probe but did not implement movement or collision.
These nine runs are iterative engineering trials where we changed the verification rules between attempts, not a blind benchmark. Even after the final two runs passed the external framebuffer probe in 23 and 41 productive turns, neither finished player movement or collision within a single session.
A non-rendering image stub, an empty source scope, and an untimed command.
A check that requires only exit code 0 and an image file cannot distinguish a rendered scene from a stub that writes a PNG. When we ran Mercury 2.5 Preview against a deliberately broken Python voxel-game baseline and a CLI task, three specific failures appeared in the logs.
1. The non-rendering image and the output-only scope.
1788823943886-game-9e9a8d: apparent verification with an empty source fingerprint.In Trial 2 (9e9a8d), the candidate’s smoke test exited 0 and saved an image to disk without rendering a 3D scene. The model also declared only artifacts/ as the contract scope. Because the output folder contained no source files, its source fingerprint was empty.
2. The 600-second find /.
1788827018813-game-83c5b1: one un-timed shell command starved the entire run.In Trial 6 (bb7c52), the external renderer validator sampled only 2 colors: a gray background and one oversized white line. The agent attributed the failure to the environment. The run also recorded 13 provider connection errors against its 90-turn cap. On the next run (Trial 7, 83c5b1), the prompt gave the exact absolute path to verify_ursina.py, but the model ran find / -name verify_ursina.py with no timeout and spent the entire 600-second wall clock in two turns.
All nine trials, including the dead ends.
We kept every trial directory, event stream, and failure log. The table lists the trials in order, including the runner and verification changes between them.
| Trial ID | Task & condition | Turns / time | Observed outcome |
|---|---|---|---|
7817b8 | CLI: baseline vs initial workflow | 35 turns (custom) / >45 (baseline) | Both passed 7/7 external CLI checks. Custom wasted calls when parallel check invocations collided. |
9e9a8d | Game: initial workflow | Completed | Invalid apparent pass. Smoke exited 0 and wrote a PNG without real rendering; contract scoped only artifacts/. |
cc97dc | CLI: queued checks + full fingerprint | 24 turns (base) / 29 turns (custom) | Baseline passed 7/7; custom passed 6/7 after crashing on non-ASCII Unicode output under Windows cp1252. |
d4a443 | Game: stricter check rules | >90 turns (base) / conn errors | Both failed the external framebuffer probe. Neither produced a playable build. |
8a4aab | CLI: required external validators | 42 turns | Passed 7/7 external behavioral checks, including non-Latin Unicode output in real subprocesses. |
bb7c52 | Game: required renderer validator | 91 turns (13 conn errors) | Failed graphics probe: real framebuffer had only 2 sampled colors (gray background + white line). |
83c5b1 | Game: revised runner | 2 turns / 600s | Starved on an untimed find / -name verify_ursina.py despite the absolute path being in the prompt. |
6bbbae | Game: 120s shell cap + productive turns + staged smoke | 23 productive turns / 100s ($0.022) | First external-probe pass: 106 sampled colors at 1280×720, procedural textures, save/load. Missing WASD/collision declared in handoff. |
9d5ccd | Game: + scope-reconciliation rule | 41 productive turns / 186s ($0.050) | External-probe pass: cleared the 3-color floor (3 colored cubes), 15 world tests, atomic saves. Missing movement/collision declared in handoff. |
Provider-reported costs ($0.022 and $0.050) are recorded estimates from the trial logs, not invoice totals. Because the verification rules changed between runs, these trials are engineering regression evidence rather than a fixed-condition benchmark.
Changes to source tracking, command timeouts, and required checks.
The earlier check rules allowed a non-rendering image stub and an output-only source scope. We changed the runner and verification rules before the later trials:
Source scope
Project-wide fingerprint
Output-only scopes are rejected, and any undeclared source edit invalidates earlier check passes.
Shell command cap
120s max per call
The runner was configured to stop each shell command after 120 seconds.
Framebuffer variety
1280×720 buffer
Coarse blank-frame gate (minimum 3 colors), not a visual quality score.
- Required external validators. In these trials, required check IDs configured in
.pi/delivery.jsonwere injected into the contract so the experiment’s rules required those check IDs for a verified finish. - FIFO check queue. When the model emits several check calls at once, queuing them serially (or running
id="all") prevents parallel subprocesses from fighting over the same build artifacts. - Productive-turn accounting. Transient upstream connection errors (13 of 91 turns in
bb7c52) are recorded separately so provider hiccups do not consume the agent’s turn budget. - Quoting the failing tail on repair. When a check fails, the repair prompt includes its exit code and stdout/stderr tail so the next attempt has the recorded failure output.
The final handoffs listed missing movement and collision.
After adding the 120-second bash cap, productive-turn accounting, and the framebuffer probe, Mercury 2.5 passed the external renderer probe in 23 productive turns (100 seconds, $0.022) and 41 productive turns (186 seconds, $0.050).
Neither of those two runs produced a complete game. Both stopped before implementing player WASD movement and collision, and the second run cleared the probe’s coarse three-color threshold by rendering three colored cubes alongside its 15 world tests and atomic save system.
In Trial 2 (9e9a8d), the smoke command returned exit code 0 despite using a non-rendering stub. Trials 8 and 9 passed the external framebuffer probe and listed movement and collision as missing. Trial 9 also included the scope-reconciliation rule.
What to take into your own repo.
Review whether a smoke test exercises the requested behavior or only checks for an exit code and an output file. Test observable behavior, such as framebuffer contents and Unicode output under the target OS encoding. Set per-command timeouts, keep required checks in user-owned configuration, and verify how the workflow handles edits to that configuration.
Research record: nine sequential development trials on Windows with Python 3.11, Ursina 7.0.0, Panda3D 1.10.16, Node 23.9.0, and Mercury 2.5 Preview. Documented in B-icy/Carthagent (docs/evaluation.md). These runs are iterative regression evidence, not a comparative model ranking.