Forge Bench: Why AI Efficiency Is a Systems Problem

AI model pricing is usually presented as a rate card: input tokens cost one amount, output tokens another, and cached tokens something else. That is useful, but a coding agent is not a single API call. It is a system. The model repeatedly reads context, calls tools, edits files, checks results, and continues until the task is done.

Forge Bench is our attempt to measure that system rather than guess from the price of one token. The project uses pinned experimental designs, run records, raw traces, and reproducible analysis so claims about agent efficiency can be inspected, repeated, and challenged.

Open research: Forge Bench repository ↗ · released study package ↗

The first surprise: the newer baseline was already economical

The focused comparison begins with DeepSeek V4 Flash and its current-generation DeepSeek V4.1 Flash successor. In this workload, DeepSeek V4.1 Flash's non-cached token prices were higher, but its cache reads were much cheaper. Because the coding-agent traces were heavily cache dominated, the practical baseline cost for DeepSeek V4.1 Flash was lower than the older DeepSeek V4 Flash baseline despite the higher headline non-cached prices.

The token-use bars make the point especially clearly. DeepSeek V4.1 Flash baseline and DeepSeek V4 Flash baseline used roughly similar total token volumes in the pooled comparison—both close to one million estimated mixed tokens—but most of that volume was cached. Pricing the workload from headline input and output rates alone would therefore miss the part of the bill that dominates these runs.

The second surprise: Ponytail helps DeepSeek V4 Flash, but not DeepSeek V4.1 Flash in the same way

Ponytail is an agent-efficiency intervention that encourages reuse, native features, and the smallest solution that meets the task. With DeepSeek V4 Flash, the effect was substantial in the focused comparison: the DeepSeek V4 Flash + Ponytail configuration cost about 48% less and took about 32% less wall time than the DeepSeek V4 Flash baseline. It also used substantially fewer tokens.

That result would be easy to turn into a generic recommendation: add the tool and make the agent cheaper. The full experiment argues against doing that. DeepSeek V4.1 Flash did not show the same improvement with Ponytail. The tool changed behavior differently depending on the model, and combining Caveman + Ponytail did not produce a reliable additive gain.

That is the more important result. Agent optimizations are not independent accessories. A prompt intervention, skill, harness rule, cache policy, provider, and model can interact. Something that reduces work for one model can be neutral—or move behavior in the opposite direction—for another.

What we actually measured

The public release combines two complementary five-task SWE-bench Verified studies. Each study tested four harness conditions—Baseline Hermes, Caveman, Ponytail, and Caveman + Ponytail—while retaining model, provider, treatment, task, study identity, run records, and raw merged traces. The released package contains 80 joined run records plus the analysis code and derived tables used for the figures.

For the interaction and focused comparisons, the analysis uses log1p mixed-effects models with study and task random intercepts and transforms estimates back to the original units. That is a more useful framing than treating every agent turn as an independent replicate: the tasks and study blocks are part of the experimental structure.

From benchmarking to agentic science

Forge Bench is designed to go beyond one-off benchmarks and simple model comparisons. We are treating agent performance as an experimental-science problem: define the factors, use designed comparisons and replication at the appropriate levels, preserve the underlying traces, and apply statistical models that can separate main effects from interactions.

That means studying variation across tasks, models, tools, harness configurations, providers, repeated studies, and eventually hardware and deployment environments. Our first release already shows why that structure matters. Ponytail changed DeepSeek V4 Flash behavior substantially but did not affect DeepSeek V4.1 Flash in the same way, while the provider change across studies is a reminder that provider and cache economics also belong in the experimental design.

Those sources of variation are not simply noise to discard; they are part of the agent system we want to measure. Future Forge Bench experiments can explicitly cross model × tool × provider, add replication at the run, task, and study levels, and test whether observed effects reproduce across workloads and infrastructure.

The broader goal is the formalization of agentic science: moving from anecdotes and leaderboard scores toward designed experiments, reproducible traces, statistical inference, and cumulative evidence about how agent systems actually behave.

The system is the experimental unit

Most discussion of AI efficiency still starts with the model. In deployed agentic systems, the real unit is closer to a system of systems: model + provider + cache + context + harness + tools + task + execution environment. The system decides how many calls happen, how much context is reused, how often tools are invoked, and how long the loop remains active.

That changes the engineering question. Instead of asking, “Which model has the cheapest tokens?” we want to ask, “Which configuration completes this kind of work with the least cost and time while preserving the capability we need?” Those are experimental questions.

Forge Bench will expand in that direction: more models, providers, tasks, harnesses, and execution environments, with raw traces and reproducible analyses published wherever practical. We are especially interested in whether the current model × tool interaction reproduces under pinned providers and in what the traces reveal about why the behavior changes.

Explore Forge Bench on GitHub ↗ · See AgenticForge Labs research ↗

For new experiments and practical notes, follow AgenticForge Labs updates on LinkedIn.