LLMs Fabricate 42% of Personalization Claims, and Self-Monitoring Makes It Worse

Memory-augmented LLMs are shipping in production under the assumption that models can be trusted to infer user attributes faithfully, and that asking models to audit their own inferences adds a safety margin. Both assumptions are wrong.

MirageBench stress-tests this directly. The benchmark covers 150 personas balanced across stereotypical, counter-stereotypical, and neutral profiles, six personalization tasks arranged along an "imagination gradient" from low-inference to high-inference, and a four-way faithfulness taxonomy judged by an independent model validated against blind human annotation on 400 claims. That validation is tight: Cohen's kappa of 0.863 on four-class and 0.900 on binary classification. The result is 143,616 judged claims across 12 models from 7 families, making this the most systematic audit of personalization faithfulness published to date.

The core finding: over-inference is not an edge case or a model-specific failure. Every model in the evaluation fabricates attributes beyond what the evidence supports. The range is 35% to 49% of claims per model, with a cross-model mean of 41.6%. No model escapes it. On neutral profiles, where stereotypical shortcuts are unavailable, over-inference still appears at measurable rates, which rules out the explanation that models are simply pattern-matching to demographic priors.

The more disorienting finding is what MirageBench calls the Self-Monitoring Inversion. When models are asked to audit their own inferences, the models that report the lowest over-inference rates are actually the ones flagged as fabricating the most by the external judge. The rank correlation between self-assessed over-inference and judge-measured over-inference is negative: rho = -0.60, p = 0.044. The paper flags this as exploratory, with a wide bootstrap confidence interval of [-0.90, +0.06] across only 12 models, so the directional finding is worth taking seriously while the precise magnitude stays uncertain.

The inversion has a structural explanation. A model that is confidently wrong about user attributes is also likely to be confidently wrong about whether it is being confidently wrong. Self-auditing draws on the same internal representations that produced the fabrication in the first place. Think of it as asking a hallucinating witness to evaluate their own testimony: the mechanism generating the error is the same mechanism generating the confidence. Within a single model, self-audit still ranks individual claims moderately well, with AUROC between 0.58 and 0.83. The problem is cross-model comparison: self-report cannot be used to select the least-fabricating model from a pool, because the ranking it produces is inverted relative to ground truth.

Task dependency adds another layer. Over-inference rates vary from 27% to 59% depending on task type, which means the imagination gradient is real. Low-inference tasks like preference recall stay closer to grounded claims. High-inference tasks like personality projection push models toward fabrication. In a multi-turn pilot, inferred attributes accumulate approximately linearly across turns with little revision, meaning errors do not self-correct as conversations extend. They compound.

Over-inference reaches 41.6% mean across 12 models on 143,616 judged claims, with no model below 35%. For teams shipping memory-augmented personalization, the takeaway is direct: self-monitoring pipelines cannot substitute for external verification, and the models you would select based on self-reported confidence are likely the worst performers on actual faithfulness.

We're thinking: We find the Self-Monitoring Inversion more alarming than the base over-inference rate. The 42% fabrication figure is bad, but it is at least measurable and addressable with external judges. The inversion means that the correction mechanism most teams would reach for, asking the model to flag its own uncertain inferences, actively misleads model selection. Products shipping a "confidence-gated memory" feature are not reducing fabrication risk. They may be selecting for the models most prone to it. The practical implication is that any personalization system treating self-reported confidence as a trust signal needs an independent faithfulness layer, not as a future improvement, but as a prerequisite for the current architecture to work as advertised.

Key takeaways:

• Over-inference is a structural property of current LLMs on personalization tasks: every model tested fabricates user attributes, with self-monitoring negatively rank-correlated to actual faithfulness at the model-selection level (rho = -0.60), making internal audits misleading for comparing models.
• 41.6% mean over-inference across 143,616 judged claims, 12 models, 7 families, with per-model range of 35-49%; the cross-model inversion finding is exploratory with a wide bootstrap CI and should be replicated at larger model counts before being treated as a stable effect.
• Teams building memory-augmented personalization should replace or supplement self-monitoring confidence gates with an independent faithfulness judge before deploying persistent user profiles in production.

Source: The Personalization Mirage