EnvACE Trains Agents Without Real Environments, and It Works

The standard assumption in agentic reinforcement learning is that the agent must interact with something external: a real API, a sandboxed executor, a simulator. That interaction loop is what generates the training signal. Building, verifying, and maintaining those environments is expensive enough that serious agentic RL has remained largely confined to teams with significant infrastructure. EnvACE breaks that assumption directly.

The method works by folding the environment into the policy itself. During training, the agent alternates between two roles: it generates a tool call, then immediately plays the environment, producing the response that action would have induced. That rehearsed response feeds back into the agent's context, and the next action is conditioned on it. Both roles, acting and environment simulation, are optimized jointly end-to-end using task-success rewards. No external executor is required.

This is not a distillation trick or a data augmentation patch. The policy is learning two things simultaneously: how to act, and what the world does in response to actions. The relationship between actions and their consequences gets encoded directly in the model's parameters, which is what the paper calls an "agent world model." Think of it as the difference between a chess player who must physically move pieces to test a line versus one who can run the continuation internally before committing. The internal simulation is imperfect, but it is fast, cheap, and improves with training.

At test time, that internalized world model becomes an additional capability. Before committing to an action in a live environment, the agent can privately rehearse candidate actions, evaluate the simulated outcomes, and then execute. This inference-time rehearsal requires no additional external calls.

Across BFCL-v4, tau-bench, VitaBench, and FinMCP-Bench, four distinct benchmarks covering tool use, financial reasoning, and multi-step agent tasks, EnvACE outperforms environment-scaling baselines in overall evaluation. Controlled studies confirm that world rehearsal consistently improves policy learning across model scales, meaning the benefit is not an artifact of a particular model size. Inference-time private rehearsal yields further gains under a moderate rehearsal budget. For teams building agentic systems on a constrained infrastructure budget, the takeaway is direct: world rehearsal is a viable substitute for real environment rollouts during training, not just a cheaper approximation.

We're thinking: We read EnvACE as a structural cost argument, not just a performance result. If world rehearsal can substitute for real environment rollouts during training, the cost of building and verifying executable environments drops to near zero, which removes the single biggest barrier to agentic RL outside well-resourced labs. The more pointed implication is competitive: teams that previously could not afford to run agentic RL at all because they lacked executable environment infrastructure now have a credible path. The caveat worth watching is distribution coverage. The internalized world model is only as good as the action-response relationships the policy has seen. In domains with rare or high-consequence tool behaviors, rehearsal quality may degrade in ways that are hard to detect without ground-truth execution. That is the failure mode to instrument before trusting private rehearsal in production.

Key takeaways:

• EnvACE folds environment simulation into the policy itself, training both the acting and environment-response roles jointly via task-success rewards, eliminating the need for external executors during training.
• Across four agentic benchmarks (BFCL-v4, tau-bench, VitaBench, FinMCP-Bench), EnvACE outperforms environment-scaling baselines overall; gains hold across model scales, though rehearsal quality in out-of-distribution tool domains remains an open caveat.
• Teams building agentic RL pipelines who are blocked by the cost of constructing and verifying executable environments should evaluate EnvACE's world rehearsal approach as a primary training strategy, with ground-truth execution reserved for evaluation rather than training.

Source: EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic RL