Analysing e-Commerce A/B Tests with Dependent Data: Empirical Evidence on Measurement Uncertainty in Average Basket Value and Other e-Commerce KPIs

Digital technology organizations often use A/B tests to guide their product and business decisions. In e-commerce, it is a common pitfall to ignore dependent transaction/item value and size that arises when one measure changes to key performance indices such as Average Basket Value (ABV), Average Basket Size (ABS), and Average Selling Price (ASP). We present empirical evidence on dependent transaction value/size, its impact on measurement uncertainty, and practical implications on A/B test outcomes if left unmitigated. By making the evidence available, we hope to drive awareness of the pitfall among experimenters in e-commerce and hence the adoption of many established mitigation approaches.