Hard Prompt Compression Retains the Answer, Deletes the Entity That Defines It

Hard prompt compression is supposed to preserve what matters. The assumption built into every token-scoring, sentence-ranking, or chunk-selecting compressor is that relevance and completeness travel together: keep the high-scoring units, discard the noise, and the answer survives. That assumption is structurally wrong.

The failure mode has a name: referential dangling. It happens when independent scoring splits a dependent evidence pair, retaining the chunk that contains the answer while deleting the chunk that defines the entity the answer refers to. The compressed context is not corrupted in any way a standard metric catches. It reads fluently. The answer token is present. But the reference that gives that answer meaning is gone, and the model is left interpreting a pronoun with no antecedent, a number with no unit, an entity label with no definition.

The mechanism is straightforward. Most hard compressors score units independently: a sentence gets a relevance score, a chunk gets an embedding similarity score, and the top-scoring units under a token budget are retained. Dependent evidence pairs, the kind that appear in multi-hop reasoning where one passage establishes what an entity is and a second passage states what that entity did, are not scored as a unit. They are scored separately. When one half of the pair scores high and the other scores low, the compressor splits them. The retained half is relevant in isolation. It is incomplete in context.

Beaver, a chunk-level compressor that ranks units using Qwen3-0.6B embeddings, leaves the answer path incomplete in 34-54% of bridge examples across three multi-hop QA datasets at a 0.30 compression ratio. Across six hard compressors tested on a shared HotpotQA bridge set, dangling rates reach 60%. Every document in LongBench-v2 Single-Document QA contains at least one dangling reference. The scale is not a corner case. It is the default condition.

The evidence on accuracy impact is direct. On dangling examples evaluated with Qwen3-8B, reinserting the missing supporting paragraph while removing non-supporting paragraphs to hold the token budget constant improves accuracy by 29-34 percentage points (p < 0.0001), recovering at least 88% of the gap to contexts that retain both supporting paragraphs. Stronger models do not absorb the loss: GPT-5.5 on MuSiQue scores 8.8 points lower on compressed contexts than on contexts retaining both supporting paragraphs. The failure is not a small-model artifact. A compact classifier trained to rank omitted sentences by whether they are needed to interpret retained text, reinserted without support annotations at inference, improves HotpotQA accuracy by 4.7 points while shifting the compression ratio only from 0.30 to 0.31. For teams running production RAG pipelines with hard compression, the takeaway is direct: your compressor's relevance score is not a completeness score, and the gap between those two things is silently degrading answer quality on a large fraction of your queries.

We're thinking: The part that should concern production teams most is not the accuracy drop itself but the failure mode's invisibility. A compressed context with referential dangling produces confident, fluent model outputs because the answer token is present. Standard evaluation metrics, which compare output tokens against a gold answer, will not flag this. Retrieval metrics will not flag it either: the retrieved passage was relevant. Only the dependency between retained and deleted text was broken, and no current pipeline component checks for that. We think this means teams that have tuned their compression ratios against benchmark accuracy are likely underestimating degradation on multi-hop and bridge-type queries specifically, which tend to be the harder, higher-value queries in production workloads.

Key takeaways:

• Hard compressors optimize relevance per unit independently, which structurally cannot detect or preserve cross-unit evidence dependencies; referential dangling is a direct consequence of that design, not a tuning failure.
• At a 0.30 compression ratio, dangling rates range from 34% to 60% across six compressors and three datasets, and GPT-5.5 loses 8.8 accuracy points on compressed versus uncompressed contexts, confirming the loss is model-agnostic; the caveat is that the classifier-based restoration adds only 0.01 to the compression ratio, so the fix is cheap but requires a trained reranker.
• Teams running RAG pipelines with hard prompt compression should audit their compression stage for referential completeness, not just relevance scores, and treat multi-hop or bridge queries as a separate failure category requiring dependency-aware retention or post-hoc restoration.

Source: Relevant but Incomplete: Referential Dangling in Hard Prompt Compression