Agent Memory Scores 100% on Recall, 14% on Actual Use

Long-term memory for AI agents rests on one assumption so obvious it almost never gets examined: the memory you need will look like the query that needs it. InMind, a 125-task expert-verified benchmark, tests what happens when that assumption fails. The results are not a marginal miss. They expose a structural gap that existing evaluations have not been measuring.

The failure mode is called the implicit-association blind spot. A user mentions a tree-nut allergy. Later, they ask for a macaron recipe. The correct answer changes because macarons use almond flour, but "macaron" and "tree-nut allergy" share no surface tokens, no semantic overlap a retriever can detect. The memory is stored. The model knows the bridging fact. The retriever simply never surfaces it, because the query does not resemble the memory. This is not a retrieval quality problem in the conventional sense. It is a retrieval interface problem: the entire architecture assumes queries and relevant memories occupy nearby points in embedding space, and world-knowledge associations routinely violate that assumption.

InMind separates three explanations that prior evaluations conflate. First: the fact was never stored. Second: the model lacks the world knowledge to connect the fact to the query. Third: the fact was stored and the model has the knowledge, but the retriever never surfaced it. By running paired controls, the benchmark isolates each failure mode cleanly. With the decisive memory placed directly in context, backbone models answer 84.0 percent of indirect queries correctly. That number establishes the ceiling: the knowledge is there, the connection is possible. When the same memory must be retrieved through six vector, graph, and agentic memory systems, accuracy drops to at most 14.4 percent, even though those same systems recall the stored facts on demand at up to 100 percent.

The dimensionality experiment makes the structural point sharper. Scaling to embeddings with eight times the dimensionality improves answer-blind target recall for every system tested. The gap between stored-and-retrieved versus stored-and-visible remains essentially intact. More expressive embeddings help the retriever find what it is looking for. They do not help the retriever know what to look for when the query does not resemble the answer.

A minimal diagnostic probe closes most of the gap by keeping relevant memories visible before the query arrives, rather than surfacing them in response to it. That single intervention relocates the failure precisely: the problem is not storage, not world knowledge, not embedding quality. It is the query-conditioned interface itself. The benchmark is built to score routing, the problem of deciding which facts must stay visible regardless of what the user asks next. 84.0 percent with memory in context versus 14.4 percent maximum through retrieval. For teams building production memory layers for agents, the takeaway is direct: any architecture that retrieves only in response to queries will silently fail on allergy-style implicit associations, and no embedding upgrade fixes that.

We're thinking: We read InMind as a safety and correctness indictment of the current default architecture, not just a benchmark contribution. Dense retrieval on query similarity is the industry standard for agent memory, and it has a failure mode that does not show up in standard recall metrics because those metrics test whether you can retrieve a fact when you ask for it directly. The real failure happens when you never ask for it directly, which is exactly when it matters most. A user's dietary restriction, a financial constraint mentioned in passing, a prior commitment that rules out an option: these are the memories that change answers without resembling queries. Teams shipping memory-augmented agents should treat this as a production gap, not a research problem, and start auditing for implicit-association failures before a user does it for them.

Key takeaways:

• Dense retrieval architectures fail on implicit associations because query-conditioned interfaces cannot surface memories that share no surface similarity with the incoming query, even when the model has the world knowledge to connect them.
• With memory in context, backbone models answer 84.0% of indirect queries correctly; six retrieval systems tested reach at most 14.4%, and scaling embedding dimensionality 8x does not close the gap. Caveat: the benchmark covers 125 tasks across ten life domains, so coverage of edge-case associations is necessarily finite.
• Teams building long-term memory for agents should audit their retrieval layer for implicit-association failures and evaluate proactive routing strategies that keep high-stakes user facts visible across sessions rather than relying solely on query-triggered retrieval.

Source: Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory