“We Need to Get Rid of Significance in A/B Testing, Seriously!” Published in ACM interactions

It’s already been a little while, but I don’t wanna miss the chance to mention here that I’ve published my very first article in the ACM interactions magazine, in their section Blog@iX. 🎉

You can read my text either on the interactions website or in the ACM Digital Library. Additionally, it has been cross-posted by the UX Collective on Medium.

TL;DR: The way A/B testing is done in many—if not most—companies today, p-values and significance are pretty much meaningless. Still, decision models often only allow treatments to be implemented if they achieved a “significant uplift”. One better alternative is to employ a Bayesian approach which allows to report probabilities for minimum uplifts and to calculate ROI on a case-by-case basis.


☕☕☕

I love coffee, and if you enjoy what I write, you can spend me one if you like, or subscribe to my newsletter.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.