Context Regeneration Filters Irrelevance
System 2 Attention (S2A) addresses a fundamental problem: soft attention in Transformers incorporates everything in context, including irrelevant or misleading information.
The solution: regenerate the input context using the model’s own reasoning, keeping only relevant portions.
Two-step process:
- Ask the model to extract and rewrite only the relevant context
- Generate the response using this filtered context
Results:
- Factual QA accuracy: 80.3%
- Long-form generation objectivity: 3.82/5.0
The pattern generalizes: when context includes noise that might mislead attention, explicit filtering (even if imperfect) beats hoping attention learns to ignore it.
This is preprocessing as a prompting strategy, shaping the input before reasoning begins.