SPADE Breaks the Static-Environment Ceiling in Agent RL

Most agent RL research treats the training environment as a fixed input: curate tasks, freeze the distribution, train until convergence. That assumption quietly caps how far any model can go. Once a capable model exhausts a fixed task pool, there are no harder problems to pull signal from, and performance plateaus regardless of compute added.

SPADE (Self-Play in Adaptive Synthetic Executable Environments) attacks this differently. A single LLM plays two roles simultaneously: an Environment Designer that writes complete training environments as executable Python code, and a Reasoning Agent that learns to solve them. The Designer produces full OpenAI Gym-style reset()/step() interfaces, including state transitions, reward functions, and verification logic, so one unified interface covers both structured reasoning problems and multi-step agentic tool use. The key design insight is that difficulty is not fixed in advance. It is estimated continuously using regret: the gap between the Reasoning Agent's reward with and without privileged hints. The Designer optimizes toward environments where that gap is large but nonzero, meaning the task is at the edge of the agent's current capability without being unsolvable.

Two components turn out to be critical in practice. The Designer is grounded on documents sampled from a large pretraining corpus, giving it raw material to construct semantically meaningful environments rather than syntactically valid but hollow ones. It also maintains an accumulated environment memory across training, so it can track which environments have already been mastered and avoid regenerating them. Together these prevent the Designer from either recycling solved problems or generating tasks so novel they are incoherent.

At 30B parameters, SPADE beats the strongest fixed-environment baseline by +5.3 on average across eight held-out benchmarks spanning math, science, code, and general reasoning. The tool-use gains are sharper: +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent. In the games setting, the margin over the strongest baseline grows as model scale increases, which is the opposite of what happens with fixed environments, where larger models hit the ceiling faster. For teams training agents past current benchmark ceilings, the takeaway is direct: the environment distribution is a trainable variable, and leaving it static is leaving capability on the table.

We're thinking: We find the regret-based difficulty signal the most underappreciated piece here. Most curriculum methods rely on heuristics: task length, human difficulty ratings, or held-out accuracy. SPADE's regret signal is endogenous: it measures how much the agent benefits from knowing the answer, which is a direct proxy for the agent's current knowledge boundary. That makes the curriculum self-correcting without any human labeling. The practical implication is that teams who have hit a plateau with fixed synthetic benchmarks should look at this before adding more curated data. The ceiling they are hitting is architectural, not a data volume problem, and SPADE's framing suggests the fix is dynamic environment generation rather than larger static corpora.

Key takeaways:

• SPADE makes environment design a learned component: one LLM generates executable training environments and adapts their difficulty using the agent's hint-conditioned regret as a live signal.
• At 30B parameters, SPADE gains +5.3 average across eight held-out benchmarks and +13.9 on ACEBench-Agent over fixed-environment baselines; gains in the games setting grow with model scale, though results are currently limited to 30B and below.
• Teams training agents on fixed synthetic task pools should treat the environment distribution as a design variable: implementing regret-based adaptive generation is more likely to break a performance plateau than expanding a static dataset.

Source: SPADE: Self-Play in Adaptive Synthetic Executable Environments