A 410M Model Beats a 12B Model: The Case for Separate Memory Parameters

The standard assumption in language model scaling is that knowledge and reasoning live together in the same weights, and that growing capacity means growing everything uniformly. That assumption leads directly to the monolithic decoder: one parameter set to rule both retrieval and inference. Memory Decoder at Scale finds that assumption is not just imprecise, it is costly.

The core architectural bet is separation. Long-term memory, the part of a model that stores factual associations and domain knowledge, gets its own parametric module, pretrained independently and attached to a smaller base model at inference. Think of it as splitting a library from the reading room: the base model handles active reasoning while the memory module handles what is known. The base model stays lean; the memory module scales on its own axis. At the engineering level, this required solving a concrete bottleneck: at 300B training tokens, standard Faiss indexing and retrieval pipelines become infeasible in both time and cost. The team replaced the single-machine Faiss pipeline with a distributed indexing and retrieval system, combined with sparse, batch-wise loading of kNN distributions, so only the nearest-neighbor distributions actually needed for a given batch are materialized in memory.

The results across 17 benchmarks are direct. Pairing a 6.9B general memory module with Pythia-410M lifts its average score from 29.86 to 37.34. Pythia-12B scores 37.24. The combined system uses 39% fewer total parameters. For Qwen3 Base models ranging from 0.6B to 14B, attaching a 1.7B domain-specific memory module improves average scores across three domains by more than 9 points at every scale tested. For ML engineers deciding where to allocate the next billion parameters, the takeaway is direct: memory capacity and reasoning capacity can be scaled independently, and scaling memory first is often the more parameter-efficient move.

We're thinking: We find the engineering result as significant as the benchmark numbers. The distributed Faiss pipeline is not a footnote; it is what makes the architecture viable at production pretraining scale, and it is the part most teams will need to reproduce before any of the benchmark gains become relevant to them. The deeper implication is that the monolithic decoder has been treated as a given partly because no one had shown the alternative holds at scale. Now that it has, the question shifts from "is separation viable" to "what is the right ratio of memory parameters to reasoning parameters for a given task distribution," which is a much more tractable engineering question than rearchitecting from scratch.

Key takeaways:

• Memory Decoder separates long-term knowledge into a standalone parametric module pretrained independently, allowing memory capacity to scale without growing the base model's reasoning weights.
• Pairing a 6.9B memory module with Pythia-410M surpasses Pythia-12B on 17 benchmarks with 39% fewer total parameters; domain memory adds more than 9 points across Qwen3 scales from 0.6B to 14B, though results depend on the distributed Faiss infrastructure the team built specifically for this scale.
• Teams scaling language models for knowledge-intensive tasks should evaluate allocating additional parameters to a separate memory module before defaulting to a larger base model, and should budget for distributed retrieval infrastructure if pretraining beyond roughly 100B tokens.

Source: Memory Decoder at Scale