Token efficiency
Measure the whole trace
Count the work consumed across reasoning, context, tool use, retries, and repeated turns so apparent model efficiency can be compared with actual agent workload.
Research · benchmarks · open methods
Measure what makes agentic systems useful.
AgenticForge Labs studies the efficiency, reliability, and practical behavior of the same digital, generative, and physical systems we build. The goal is evidence that helps us choose architectures and models deliberately—not just impressive demos.
TokensWork per taskTotal usage, reasoning, and tool-loop overhead
LatencyTime to useful workModel, provider, and long-running trace behavior
CacheReuse that mattersHow persistent context changes repeated-work economics
CostCapability per dollarEnd-to-end workflow efficiency, not API price alone
Forge Bench
Efficiency is an engineering constraint.
Our current Forge Bench work examines agentic workloads as complete computational systems. We track more than a single prompt and response: token use across long-running traces, latency, cache behavior, provider differences, tool calls, task completion, and cost. Early runs are exploratory measurements to reproduce and refine—not universal model rankings.
Define→Run→Trace→Measure→Compare→Repeat
Latency & caching
Speed depends on the workflow
Track response time, provider behavior, and cache reuse across persistent tasks where repeated context can materially change both speed and cost.
Cost & capability
Optimize useful work, not the cheapest token
Relate total workflow cost to task completion and quality so model routing can be designed around the work that needs to happen.
Reproducibility
Treat agent evaluation like an experiment
Use controlled tasks, repeated runs, explicit factors, stored traces, and transparent analysis so conclusions can be challenged and reproduced.
Preliminary results · September 2026
The same harness intervention can help one model and hurt another.
Our first completed Forge Bench model-comparison study used one randomized complete block: 2 DeepSeek model configurations × 4 Hermes harness treatments × 5 SWE-bench Verified tasks = 40 planned runs, with both models pinned to the same Relace OpenRouter upstream. Some runs were invalid, so these are exploratory signals from five task units—not universal model rankings.
DeepSeek V4 Flash + Ponytail
Mean total tokens fell from about 1.07M at baseline to about 0.56M with Ponytail in this study.
DeepSeek V4.1 Flash + Ponytail
The same intervention moved in the opposite direction: about 0.93M baseline versus about 3.52M with Ponytail.
Tool-use trajectories diverged
V4.1 + Ponytail accumulated many more tool calls than the other V4.1 conditions, showing why long-running traces reveal behavior that aggregate benchmark scores can hide.
Across the Forge
One measurement discipline, three axes of work.
Forge Bench begins with model and agent efficiency, then extends the same experimental approach across the primary technical directions of AgenticForge Labs.
Digital agency
Agents & orchestration
Context, memory, routing, tool use, long-running work, reliability, and the compute required to complete useful tasks.
Generative systems
Persistent creation
Continuity, provenance, iteration cost, model handoffs, media-pipeline throughput, and how creative state survives across generations.
Physical agency
Robotics & fabrication
Perception-to-action latency, task success, recovery, hardware utilization, validation, and safe handoffs between probabilistic reasoning and deterministic control.
Open where we can be
Research is more useful when it can be inspected.
The persistent AgenticForge runtime remains proprietary core technology. Around it, we intend to publish benchmark methods, selected traces and datasets, interfaces, SDKs, adapters, reference implementations, and analysis where doing so is practical, safe, and compatible with licensing and partner obligations.
Open principle
Methods before marketing.
We want public results to include enough context to understand how they were produced: task definitions, relevant configurations, measurement assumptions, and limitations.
How the open ecosystem fits the runtime →Public code
Early software from the lab.
Public research software is released early enough to be useful and inspected. Status labels matter: these projects are evolving, and hardware-facing code should be tested conservatively.
Public repository
ResearchForge Bench
A reproducible framework for randomized studies of AI coding-agent harnesses and models, with complete long-running traces, SWE-bench outcomes, token and cost accounting, timing decomposition, tool-use analysis, multivariate analysis, and auditable study packages.
Public repository
BetaSO-ARM101 Motion SDK
A focused Python motion SDK for the five-axis SO-ARM101 follower arm and stock gripper, with joint motion, forward/inverse kinematics, Cartesian movement, tools/TCPs, diagnostics, workspace checks, simulation, and an optional desktop controller. ROS and LeRobot are not runtime dependencies.
Progressive release
More to comeOpen components as they are ready.
Only repositories that are actually public are listed here. Additional robot interfaces and reusable components will appear as they are prepared for public use.
Build, measure, publish, improve.
Research connects the systems we build with the evidence needed to make them more efficient, reproducible, and dependable.