Two coding agents built a voxel world. Neither was ready to play.
In this run, CTG finished sooner, used fewer tokens, and cost less. Independent movement checks found gameplay defects in both submissions.
Two completed builds. Zero playable acceptance passes.
This is a single paired experiment, not a general ranking. Both submissions contain real work. Both also contain bugs serious enough to break the requested experience.
One task, two fresh starts.
We gave CTG and plain Pi identical copies of an existing Python game-reference project and the same prompt: build a small, native Minecraft-style world using Panda3D. The task required a native application that rendered the world at runtime.
| World | Approximately 64 × 64 blocks; seeded grass, dirt, stone and trees. |
|---|---|
| Play | Mouse-look, WASD, gravity, jump, collisions, safe spawn, block placement and removal. |
| Efficiency | Chunked meshes; omit hidden faces; rebuild only affected chunks. |
| Evidence | Tests, launch instructions, a screenshot and at least 300 rendered smoke-test frames. |
| Run conditions | Fresh sessions, high thinking, concurrent starts, 30-minute limit, no corrective steering. |
CTG used its separate delivery workflow. Pi ran without optional extensions or skills. The comparison tests these complete workflows on one task, not the model against itself under perfectly interchangeable conditions.
CTG reached its declared finish with lower recorded time, cost, and token usage.
Neither submission passed playable-game acceptance.
Coding time
25.2% less time
Wall-clock time, including tool execution.
Verified usage cost
31.9% lower cost
USD for these coding runs; excludes evaluation.
Cumulative tokens
83.6% fewer tokens
Includes cached input repeated across requests.
Each chart starts at zero and uses its own scale. Differences in execution conditions and pricing mean these observations do not isolate a causal advantage for Carthagent’s workflow. One attempt per tool is not enough to establish repeatability.
The token and work breakdown
| Measure | CTG | Pi |
|---|---|---|
| Uncached input | 254,304 | 349,691 |
| Cached input | 1,673,088 | 11,592,064 |
| Output, including reasoning | 48,623 | 108,304 |
| Reasoning subset | 15,767 | 64,274 |
| Total tokens | 1,976,015 | 12,050,059 |
| Model responses | 41 | 127 |
| Tool calls | 57 | 136 |
| Flagged tool errors | 9 | 3 |
Reasoning is already included in output. Cached input counts repeated context, not newly generated content. Flagged errors do not capture every failure concealed inside a successful shell command.
Geometry recorded from independent rendering runs.
Both built deterministic terrain, chunked meshes, and a native renderer. Below are the exact triangle and mesh counts recorded from fresh independent runs (seed 42, 600 reported frames each).
Conclusions below come from reproducible code and coordinate checks across six seeds and five camera headings. Pi also reported intermittent blank framebuffer captures during development.
Rendering, physics, and movement used inconsistent coordinates.
Terrain generation passed. Collision math passed. The missing checks were at the boundaries: world coordinates to rendered coordinates, and camera direction to player movement.
CTG: rendered height had the opposite sign from collision height.
The terrain’s rotation turns logical height +15 into rendered height −15. The camera uses positive height. The visible ground and the ground the player collides with are therefore in different places.
Independent checks also found reversed camera-relative movement at some headings, two wrongly wound face directions, and placement that could put a block inside the player. In that placement fixture, 60 ticks of movement and jumping left the player stuck.
Pi: keyboard movement never changed world Z.
Pi computes a horizontal camera vector, zeroes its vertical component, then reads that zero as the player’s other horizontal axis. World Z never changes. At the initial heading, pressing W produces no displacement.
Its rendering conversion and placement guards were otherwise more coherent. It rejected player-overlapping blocks and provided a solid boundary and floor. Those strengths cannot compensate for broken traversal.
Both test suites passed; both submissions failed playable acceptance.
| Check | CTG | Pi |
|---|---|---|
| Python test suite | 45 passed | 46 passed |
| Data/document tests | 15 passed | 15 passed |
| Fresh 600-frame runs | 3 completed | 3 completed |
| Seeded terrain / supported spawn | Passed on 6 seeds | Passed on 6 seeds |
| Hidden-face oracle | Exact match | Exact match |
| Basic gravity, jump, collision | Passed independent fixtures | Passed independent fixtures |
| Movement / camera integration | Failed | Failed |
| Player-safe block placement | Failed | Passed tested fixture |
| Playable acceptance | Failed | Failed |
CTG’s pure physics module was easier to test and had meaningful gameplay unit tests. Pi’s tests concentrated on storage, terrain, and meshing; movement and interaction were under-covered. Neither suite checked the coordinate conversions that connected those parts.
The independent review exercised production calculations without OS mouse input, checked six generation seeds, tested placement overlap and reach limits, and compared emitted faces against a separate oracle. It was not a manual play session. No source repairs were applied before scoring.
Why we did not publish an FPS winner
We ran three sequential 600-frame smoke tests per implementation after the coding runs ended. CTG’s reported p95 intervals ranged from 5.47–5.78 ms, but its first interval included seconds of setup and smoke mode skipped physics. Pi stepped the application and explicitly rendered again, so its reported iterations were not equivalent to CTG’s frames.
The world heights, geometry, and rendering workloads also differed. Both completing a rendering loop is useful evidence. Calling one the faster game from those timings would overstate what was measured.
Friction worth recording.
- Setup was not frictionless. Both had to work around missing Python environment bootstrapping support.
- CTG spent calls negotiating its workflow. Planning guards, tool-argument errors, and a review rejection account for part of its nine flagged failures.
- Pi debugged a real meshing error. A tuple/integer indexing mismatch appeared during development. It also reported intermittent blank screenshots.
- Existing tests were not an oracle. The original ore generator passed its suite but changed output across Python hash seeds. Both retained this pre-existing defect; it is separate from their new terrain generators.
- The final claims needed checking. CTG described 45 tests as existing regression coverage; the total was 24 original plus 21 added. Pi changed a cleanup-test exclusion for its virtual environment. We recorded both rather than taking completion summaries at face value.
Add checks across rendering, movement, and collision coordinates.
The missing checks concern coordinate agreement, camera-relative movement, placement overlap, and triangle winding.
| Add this check | What it catches |
|---|---|
| Transform a known block and compare its rendered height with collision height. | CTG’s inverted vertical axis. |
| Hold W and D at five camera headings; assert both horizontal axes are reachable. | Pi’s lost axis and CTG’s direction mismatch. |
| Cast through the crosshair and verify the hit corresponds to the camera’s forward vector. | Disagreement between targeting and what the player sees. |
| Try placing a block into the player’s bounding box. | Self-trapping placement. |
| Check triangle winding against outward normals. | Wrong-sided faces hidden by count-only tests. |
CTG reached its declared finish with lower recorded time, usage cost, token totals, and tool-call counts in this run. However, CTG’s required checks passed without detecting the coordinate and placement defects, just as Pi’s tests did.
A stronger follow-up would repeat paired trials, normalize execution limits, and score input-driven gameplay before comparing cost per successful delivery. In this experiment, there were no successful playable deliveries to put in that denominator.
Neither submission passed playable acceptance.
This attempt recorded lower time, cost, and token totals for CTG and exposed missing integration checks in both submissions. Neither “the agent finished” nor “the tests passed” answered the original question: can someone play the world?
Research date: October 13, 2026. One concurrent coding run per tool, high thinking, 30-minute limit. Independent rendering repeated three times per submission. Cost figures are verified usage charges, excluding research orchestration and evaluation. The figures on this page are geometry summaries and explanatory diagrams, not game screenshots.
All research notes