It’s already been a little while, but I don’t wanna miss the chance to mention here that I’ve published my very first article in the ACM interactions magazine, in their section Blog@iX. 🎉
TL;DR: The way A/B testing is done in many—if not most—companies today, p-values and significance are pretty much meaningless. Still, decision models often only allow treatments to be implemented if they achieved a “significant uplift”. One better alternative is to employ a Bayesian approach which allows to report probabilities for minimum uplifts and to calculate ROI on a case-by-case basis.