Live Mid-Run Steering Cuts Agent Token Costs by 47% While Lifting Accuracy 14 Points

The standard self-improvement loop for agents follows a clean sequence: run the task, collect the trajectory, extract lessons, update the system before the next run. That sequence has a fatal flaw. In long-horizon tasks, early mistakes do not wait for the post-mortem. They compound inside the active run, consuming tokens, derailing subtasks, and producing outputs the system cannot retroactively fix. Waiting until execution ends to apply what the agent just learned is not a timing inconvenience. It is the wrong architecture.

PILOT (Proactive In-Loop Orchestration Technique) separates the agent that executes from the agent that watches, and gives the watcher real authority. A dedicated supervisor monitors the active worker continuously. When the supervisor detects a failing trajectory, it does not log the failure for later. It steers or aborts the worker mid-run, then immediately distils the failure mode into reusable skills and memory that persist for future runs. Two mechanisms run in parallel: live steering, which redirects the active worker during execution, and live self-evolution, which writes the lessons into the persistent harness before the run closes. The distinction from subagent delegation is precise: most delegation architectures hand off tasks but cannot recall or redirect a worker once it has started. PILOT's supervisor holds that authority throughout.

The design also sidesteps the context-pollution problem that plagues single-agent self-correction. When one model handles both task execution and trajectory assessment in the same context, the assessment is contaminated by the same reasoning errors that produced the bad trajectory. A separate supervisor context is not a redundancy. It is a clean read on what the worker is actually doing.

Across two frozen backbones, GLM-5.1 and Kimi-K2.6, and three benchmarks, PILOT ranks first in five of six configurations. On Terminal-Bench 2.0, it outperforms counterpart harnesses by up to 9.8 percentage points. In the self-improvement setting, it gains 14.6 points with GLM-5.1 and 12.4 points with Kimi-K2.6. Mean output tokens fall by 42.9% and 47.4% respectively, and successful evaluations per million output tokens rise by 110.3% and 134.0%. The token efficiency numbers are not incidental. They are the mechanism working: fewer tokens wasted on compounding errors, more tokens spent on productive execution. For teams building long-horizon agents on metered inference budgets, the takeaway is direct: mid-run steering is not a quality-of-life feature, it is a cost control mechanism.

We're thinking: We read PILOT as a direct challenge to the assumption that batch-then-retrain is merely slow. For short tasks, the delay is annoying but recoverable. For long-horizon runs where step 3's error reshapes steps 4 through 40, the delay is structural damage. PILOT's efficiency gains, nearly halving token output while more than doubling successful evaluations per million tokens, suggest that the dominant cost in current long-horizon agent deployments is not model capability but error propagation time. That reframes the build decision: before scaling model size or expanding context windows, teams should ask whether a lightweight supervisor with abort authority would recover more value per dollar than any other upgrade on the roadmap.

Key takeaways:

• PILOT decouples execution from assessment by running a persistent supervisor alongside the active worker, giving it authority to steer or abort mid-run and immediately encode failure modes as reusable skills, rather than deferring all learning to post-execution.
• PILOT ranks first in 5 of 6 benchmark configurations, with gains of up to 14.6 accuracy points and token efficiency improvements exceeding 110% on successful evaluations per million output tokens; caveat is that results are on two specific frozen backbones and may not transfer directly to instruction-tuned models with different self-correction tendencies.
• Teams running long-horizon agents on cloud inference should instrument a supervisor layer with mid-run abort authority before expanding context windows or model size, since PILOT's results indicate error propagation is a larger cost driver than raw capability limits.

Source: PILOT in the Loop: Live Self-Improvement for Long-Horizon Agents