Crop-and-Zoom Rarely Changes the Answer: A Causal Audit of Visual Tool-Use

Multimodal LLMs that call crop-and-zoom operations appear to benefit from them. Aggregate accuracy numbers often tick upward. That appearance is the illusion: a new causal audit across six models and five benchmarks shows that the returned visual observations frequently have no causal effect on the final answer at all.

The audit frames visual tool-use as a causal graph and separates two paths: the observation-mediated path, where a returned image crop actually shifts the model's reasoning, and the action-induced shortcut, where the model reaches the same answer it would have reached anyway. To distinguish these paths, the audit applies interventions at three levels. At the policy level, tool-use is compared against direct inference. At the trajectory level, all observations during a rollout are corrupted to see whether accuracy collapses. At the step level, individual observations are counterfactually replaced under a fixed prefix, isolating each crop's contribution through an estimand called Visual Evidence Gain. That step-level measure is the diagnostic that matters most: it catches cases where the trajectory-level numbers look acceptable but individual observations are doing nothing.

Two failure modes surface repeatedly. In Calling Without Looking, the model invokes a visual tool, receives an observation, and then ignores it entirely: the answer is causally identical to what direct inference would produce, at higher token cost. In Looking Without Planning, the returned observation is genuinely informative but the call schedule is incoherent, so the model crops irrelevant regions before or instead of the regions that would help. Effective tool-use, where the observation causally shifts the answer in the right direction, is concentrated in a Calibrated minority of rollouts. The aggregate accuracy gain that product teams see is real but narrow: it comes from that minority, while the majority of tool calls are expensive and causally inert.

Across six models and five fine-grained perception benchmarks, the trajectory-level diagnostic confirms that corrupting all observations during rollout often fails to collapse accuracy, which is precisely the wrong behavior for a system that is supposed to be grounding its answers in visual evidence. For teams building vision-augmented inference pipelines, the takeaway is direct: token budgets allocated to visual tool-use calls are not buying proportional reasoning improvement, and the failure is structural, not a tuning problem.

We're thinking: We find the Calling Without Looking failure mode particularly damaging for product assumptions, because it is invisible to standard eval. Aggregate accuracy can improve while the majority of individual tool calls are causally disconnected from the answer, meaning teams shipping crop-and-zoom features may be measuring the Calibrated minority and pricing the product on it. The deeper problem is that current training and prompting pipelines give models no explicit pressure to actually condition on returned observations. Until call scheduling and observation integration are trained jointly and audited causally rather than by accuracy alone, visual tool-use is closer to a learned ritual than a reasoning operation.

Key takeaways:

• Visual tool-use in multimodal LLMs splits into two failure modes, Calling Without Looking and Looking Without Planning, with causally effective use concentrated in a minority of rollouts; Visual Evidence Gain at the step level is the diagnostic that separates them.
• Across six models and five fine-grained perception benchmarks, corrupting all visual observations during rollout often fails to degrade accuracy, confirming that aggregate gains mask widespread causal inertness; the audit methodology is open-source at github.com/OpenCausaLab/CauAudit, though results are currently limited to the tested model and benchmark set.
• Teams building or evaluating vision-augmented LLM pipelines should run trajectory-level corruption tests before attributing accuracy gains to visual grounding, and should treat per-call token costs as real until step-level causal analysis confirms the observations are actually being used.

Source: The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images