Lost in the middle: test context position, not just size
Long-context benchmarks show that relevant-information position can change accuracy, but the effect is task- and model-dependent rather than a universal attention curve.
A model accepting 200,000 tokens does not prove that it uses every position equally well. Input capacity is a limit; retrieval and reasoning accuracy are behaviors that require separate measurement.
The 2023 “Lost in the Middle” study demonstrates the gap. Moving the same relevant information within a long context changes model performance on tested retrieval and question-answering tasks. The result supports position-aware evaluation, not a universal formula for every model and workload.
| Resource | Link |
|---|---|
| Lost in the Middle (Liu et al., 2023) | arxiv.org/abs/2307.03172 |
| Effective context engineering for AI agents | anthropic.com/engineering/effective-context-engineering-for-ai-agents |
What the study measures
Liu and colleagues evaluate language models on two tasks that require finding relevant information in supplied context:
- Key-value retrieval: recover a value associated with a key from many key-value pairs.
- Multi-document question answering: answer a question using one relevant document among distractors.
The experiments move the relevant item to different positions while holding the underlying task constant. Performance is often highest when the item appears near the beginning or end and degrades when it appears in the middle. Explicitly long-context models also show position sensitivity in the evaluated settings.
The paper measures task performance. It does not establish that every transformer has one fixed U-shaped internal attention curve, that the middle 30% of every prompt is a dead zone, or that information placed there disappears from model memory.
Position is one variable in context quality
A context window combines several pressures:
| Variable | Example |
|---|---|
| Position | Relevant document first, middle, or last |
| Context length | 10 documents versus 100 |
| Distractor similarity | Unrelated text versus plausible competing evidence |
| Task shape | Exact retrieval versus synthesis across sources |
| Query placement | Question before or after the evidence |
| Model and version | Different training and long-context behavior |
| Output criterion | Exact match, citation accuracy, or graded usefulness |
A larger window can hold more evidence and more distraction at the same time. Adding tokens therefore changes both capacity and the retrieval problem.
Anthropic describes context as a finite resource with diminishing marginal returns. Its practical recommendation is to find the smallest high-signal set of tokens that supports the desired behavior, then retrieve additional material when the task needs it.
Measure a position sweep
The useful artifact is not a diagram of a generic curve. It is a repeatable evaluation for the actual workload.
Start with a fixture set large enough to report uncertainty. Each answer should be grounded in one known source. For every fixture:
- Build a fixed distractor set.
- Place the relevant source at 0%, 25%, 50%, 75%, and 100% of the source sequence.
- Keep the question, decoding settings, output schema, and total context constant.
- Record answer accuracy, citation accuracy, latency, and input tokens.
- Repeat the sweep at multiple context lengths.
| Field | Purpose |
|---|---|
fixture_id |
Join every permutation of the same task |
model_version |
Detect behavior changes across releases |
context_tokens |
Separate length effects from position effects |
relevant_position |
Compare the controlled placement |
answer_correct |
Measure task success |
citation_correct |
Verify that the right source supports the answer |
latency_ms |
Capture the operational cost |
Report the per-position rate with a confidence interval. A five-point difference across 20 examples is not the same evidence as the same difference across 2,000 examples.
Prefer curation over positional folklore
Moving one paragraph to the end can improve one benchmark and fail another. More durable context strategies reduce the amount of irrelevant competition:
- Retrieve the smallest evidence set that preserves recall.
- Give documents stable identifiers and clear boundaries.
- Remove duplicate tool output and superseded state.
- Load detailed material just in time through tools.
- Compact long histories while preserving decisions, failures, and unresolved work.
- Persist durable task state outside the active context, then retrieve it deliberately.
These techniques change the information set, not just its order.
Query placement should also be evaluated rather than asserted. Putting the question after a source bundle makes the current task recent; putting it before can frame how the evidence is read. The position sweep reveals which arrangement works for the target model and task.
Separate retrieval from synthesis
A wrong answer can come from at least two failures:
- The model does not recover the relevant evidence.
- The model recovers it but reasons or formats the answer incorrectly.
Measure those stages independently. Require the model to cite the source identifier or return the extracted evidence span, then grade the final answer. A citation miss indicates retrieval trouble; a correct citation with a wrong conclusion indicates synthesis trouble.
That distinction determines the fix. Retrieval failures call for better selection, indexing, chunking, or context layout. Synthesis failures call for clearer task instructions, examples, tools, or reasoning checks.
Re-run the test when the stack changes
Position behavior can change with a model release, a new retrieval policy, a different system prompt, or a revised document format. Keep the sweep in the evaluation suite and run it against the assembled production request.
Do not copy a percentage from a 2023 benchmark into a product requirement. Record the model identifier, fixture set, context lengths, and request contract that produced the current result.
Takeaways
Capacity is not utilization
A supported context length says how much input fits, not how reliably every position informs the answer.
Qualify the benchmark
Lost in the Middle finds position sensitivity on specific retrieval and question-answering tasks, not a universal dead zone.
Run paired position sweeps
Hold fixtures, distractors, and token count constant while moving the relevant evidence through the context.
Curate before rearranging
Retrieval, compaction, and deliberate state management reduce irrelevant context instead of relying on a favored position.