Forge Bench: Why AI Efficiency Is a Systems Problem

AI model pricing is usually presented as a rate card: input tokens cost one amount, output tokens another, and cached tokens something else. That is useful, but a coding agent is not a single API call. It is a system. The model repeatedly reads context, calls tools, edits files, checks results, and continues until the task is done.

Forge Bench is our attempt to measure that system rather than guess from the price of one token. The project uses pinned experimental designs, run records, raw traces, and reproducible analysis so claims about agent efficiency can be inspected, repeated, and challenged.

Open research: Forge Bench repository ↗ · released study package ↗

The first surprise: the newer baseline was already economical

The focused comparison begins with DeepSeek V4 Flash and its current-generation DeepSeek V4.1 Flash successor. In this workload, DeepSeek V4.1 Flash's non-cached token prices were higher, but its cache reads were much cheaper. Because the coding-agent traces were heavily cache dominated, the practical baseline cost for DeepSeek V4.1 Flash was lower than the older DeepSeek V4 Flash baseline despite the higher headline non-cached prices.

The token-use bars make the point especially clearly. DeepSeek V4.1 Flash baseline and DeepSeek V4 Flash baseline used roughly similar total token volumes in the pooled comparison—both close to one million estimated mixed tokens—but most of that volume was cached. Pricing the workload from headline input and output rates alone would therefore miss the part of the bill that dominates these runs.

The second surprise: Ponytail helps DeepSeek V4 Flash, but not DeepSeek V4.1 Flash in the same way

Ponytail is an agent-efficiency intervention that encourages reuse, native features, and the smallest solution that meets the task. With DeepSeek V4 Flash, the effect was substantial in the focused comparison: the DeepSeek V4 Flash + Ponytail configuration cost about 48% less and took about 32% less wall time than the DeepSeek V4 Flash baseline. It also used substantially fewer tokens.

That result would be easy to turn into a generic recommendation: add the tool and make the agent cheaper. The full experiment argues against doing that. DeepSeek V4.1 Flash did not show the same improvement with Ponytail. The tool changed behavior differently depending on the model, and combining Caveman + Ponytail did not produce a reliable additive gain.

That is the more important result. Agent optimizations are not independent accessories. A prompt intervention, skill, harness rule, cache policy, provider, and model can interact. Something that reduces work for one model can be neutral—or move behavior in the opposite direction—for another.

What we actually measured

The public release combines two complementary five-task SWE-bench Verified studies. Each study tested four harness conditions—Baseline Hermes, Caveman, Ponytail, and Caveman + Ponytail—while retaining model, provider, treatment, task, study identity, run records, and raw merged traces. The released package contains 80 joined run records plus the analysis code and derived tables used for the figures.

For the interaction and focused comparisons, the analysis uses log1p mixed-effects models with study and task random intercepts and transforms estimates back to the original units. That is a more useful framing than treating every agent turn as an independent replicate: the tasks and study blocks are part of the experimental structure.

Why we are being careful with the conclusion

This is a research release, not a universal leaderboard. There is one run per model × treatment × task cell in each study. There are only two study blocks, so study-level variation cannot be estimated precisely. DeepSeek V4 Flash used Relace in both studies, while DeepSeek V4.1 Flash used Relace in the earlier study and DeepSeek in the later study, which means the pooled analysis cannot cleanly separate the DeepSeek V4.1 Flash provider change from the study change.

The intervals around some pooled estimates are wide. The useful claim is therefore not that one configuration will always be faster or cheaper everywhere. It is that the observed interactions are large enough to justify testing them directly, with provider × model and provider × tool effects as obvious next experiments.

Why this matters beyond one benchmark

Most discussion of AI efficiency still starts with the model. In deployed agentic systems, the real unit is closer to a system of systems: model + provider + cache + context + harness + tools + task + execution environment. The system decides how many calls happen, how much context is reused, how often tools are invoked, and how long the loop remains active.

That changes the engineering question. Instead of asking, “Which model has the cheapest tokens?” we want to ask, “Which configuration completes this kind of work with the least cost and time while preserving the capability we need?” Those are experimental questions.

Forge Bench will expand in that direction: more models, providers, tasks, harnesses, and execution environments, with raw traces and reproducible analyses published wherever practical. We are especially interested in whether the current model × tool interaction reproduces under pinned providers and in what the traces reveal about why the behavior changes.

Explore Forge Bench on GitHub ↗ · See AgenticForge Labs research ↗

For new experiments and practical notes, follow AgenticForge Labs updates on LinkedIn.