“We Need to Get Rid of Significance in A/B Testing, Seriously!” Published in ACM interactions

It’s already been a little while, but I don’t wanna miss the chance to mention here that I’ve published my very first article in the ACM interactions magazine, in their section Blog@iX. 🎉

You can read my text either on the interactions website or in the ACM Digital Library. Additionally, it has been cross-posted by the UX Collective on Medium.

TL;DR: The way A/B testing is done in many—if not most—companies today, p-values and significance are pretty much meaningless. Still, decision models often only allow treatments to be implemented if they achieved a “significant uplift”. One better alternative is to employ a Bayesian approach which allows to report probabilities for minimum uplifts and to calculate ROI on a case-by-case basis.