Are You Sure You're Sure? -- Effects of Visual Representation on the Cliff Effect in Statistical Inference

Common reporting styles for statistical results, such as $p$-values and confidence intervals (CI), have been reported to be prone to dichotomous interpretations, especially with respect to null hypothesis testing frameworks. For example, when the $p$-value is small enough or the CIs of the mean effects of a studied drug and a placebo are not overlapping, scientists tend to claim significant differences while often disregarding the magnitudes and absolute differences in the effect sizes. Techniques relying on the visual estimation of the strength of evidence have been recommended to reduce such dichotomous interpretations but their effectiveness has also been challenged. We ran two experiments to compare several alternative representations of confidence intervals and used Bayesian multilevel models to estimate the effects of the representation styles on differences in subjective confidence in the results. We also asked the respondents' opinions and preferences in representation styles. Our results suggest that adding visual information to classic CI representation can decrease the tendency towards dichotomous interpretations $-$ measured as the "cliff effect": the sudden drop in confidence around $p$-value 0.05 $-$ compared with classic CI visualization and textual representation of the CI with $p$-values. As a contribution to open science, our data and all analyses are publicly available at https://github.com/helske/statvis .