Token efficiency
Measure the whole trace
Count the work consumed across reasoning, context, tool use, retries, and repeated turns so apparent model efficiency can be compared with actual agent workload.
Research · benchmarks · open methods
Measure what makes agentic systems useful.
AgenticForge Labs studies the efficiency, reliability, and practical behavior of the same digital, generative, and physical systems we build. The goal is evidence that helps us choose architectures and models deliberately—not just impressive demos.
TokensWork per taskTotal usage, reasoning, and tool-loop overhead
LatencyTime to useful workModel, provider, and long-running trace behavior
CacheReuse that mattersHow persistent context changes repeated-work economics
CostCapability per dollarEnd-to-end workflow efficiency, not API price alone
Forge Bench
Efficiency is an engineering constraint.
Forge Bench examines agentic workloads as complete computational systems. We track more than a single prompt and response: token use across long-running traces, latency, cache behavior, provider differences, tool calls, task completion, and cost. Our first public release combines two complementary five-task studies and publishes the run records, raw traces, analysis code, and figures needed to inspect the result.
Define→Run→Trace→Measure→Compare→Repeat
Latency & caching
Speed depends on the workflow
Track response time, provider behavior, and cache reuse across persistent tasks where repeated context can materially change both speed and cost.
Cost & capability
Optimize useful work, not the cheapest token
Relate total workflow cost to task completion and quality so model routing can be designed around the work that needs to happen.
Reproducibility
Treat agent evaluation like an experiment
Use controlled tasks, repeated runs, explicit factors, stored traces, and transparent analysis so conclusions can be challenged and reproduced.
Research release · September 2026
AI efficiency depends on the system around the model.
The first public Forge Bench release combines two five-task SWE-bench Verified studies across four Hermes harness conditions. The pooled package contains 80 joined run records plus raw traces, derived tables, analysis code, and figures. The result is not a universal model ranking; it is evidence that cache economics and model × tool interactions can materially change real coding-agent cost and speed.
Headline token prices were not the bill
DeepSeek V4.1 Flash used roughly similar total mixed-token volume to the DeepSeek V4 Flash baseline, but much cheaper cache reads kept its estimated baseline cost lower in this cache-heavy workload.
DeepSeek V4 Flash + Ponytail
Relative to the DeepSeek V4 Flash baseline, Ponytail substantially reduced token use and produced lower pooled estimates for both cost and time.
Model × tool interaction matters
Ponytail did not help DeepSeek V4.1 Flash in the same way, and Caveman + Ponytail did not show a reliable additive gain. Agent optimizations need to be tested with the models and providers that actually run them.
Across the Forge
One measurement discipline, three axes of work.
Forge Bench begins with model and agent efficiency, then extends the same experimental approach across the primary technical directions of AgenticForge Labs.
Digital agency
Agents & orchestration
Context, memory, routing, tool use, long-running work, reliability, and the compute required to complete useful tasks.
Generative systems
Persistent creation
Continuity, provenance, iteration cost, model handoffs, media-pipeline throughput, and how creative state survives across generations.
Physical agency
Robotics & fabrication
Perception-to-action latency, task success, recovery, hardware utilization, validation, and safe handoffs between probabilistic reasoning and deterministic control.
Open where we can be
Research is more useful when it can be inspected.
The persistent AgenticForge runtime remains proprietary core technology. Around it, we intend to publish benchmark methods, selected traces and datasets, interfaces, SDKs, adapters, reference implementations, and analysis where doing so is practical, safe, and compatible with licensing and partner obligations.
Open principle
Methods before marketing.
We want public results to include enough context to understand how they were produced: task definitions, relevant configurations, measurement assumptions, and limitations.
How the open ecosystem fits the runtime →Public code
Early software from the lab.
Public research software is released early enough to be useful and inspected. Status labels matter: these projects are evolving, and hardware-facing code should be tested conservatively.
Public repository
ResearchForge Bench
A reproducible framework for randomized studies of AI coding-agent harnesses and models, with complete long-running traces, SWE-bench outcomes, token and cost accounting, timing decomposition, tool-use analysis, multivariate analysis, and auditable study packages.
Public repository
BetaSO-ARM101 Motion SDK
A focused Python motion SDK for the five-axis SO-ARM101 follower arm and stock gripper, with joint motion, forward/inverse kinematics, Cartesian movement, tools/TCPs, diagnostics, workspace checks, simulation, and an optional desktop controller. ROS and LeRobot are not runtime dependencies.
Progressive release
More to comeOpen components as they are ready.
Only repositories that are actually public are listed here. Additional robot interfaces and reusable components will appear as they are prepared for public use.
Build, measure, publish, improve.
Research connects the systems we build with the evidence needed to make them more efficient, reproducible, and dependable.
Across AgenticForge Labs
Research and creative worlds.
Two brands apply our engineering work to different kinds of discovery and making.
Scientific imaging & robotics
Phenotyper
Imaging instruments and agent-ready robotic systems for plant and microbial research.
Explore Phenotyper ↗
Creative worlds & production
AgenticForge Studio
AI-expanded worlds and generative production, with Forge Born as its original story world.
Explore Studio · Forge Born →