Elementary methods provide more replicable results in microbial differential abundance analysis

Differential abundance analysis is a key component of microbiome studies. It focuses on the task of assessing the magnitude and statistical significance of differences in microbial abundances between conditions. While dozens of methods for differential abundance analysis exist, they have been reported to produce remarkably discordant results. Currently, there is no consensus on the preferred methods. While correctness of results in differential abundance analysis is an ambiguous concept that cannot be evaluated without employing simulated data, we argue that consistency of results across datasets should be considered as an essential quality of a well-performing method. We compared the performance of 13 differential abundance analysis methods employing datasets from multiple (N = 54) taxonomic profiling studies based on 16S rRNA gene or shotgun sequencing. For each method, we examined how the results replicated between random partitions of each dataset and between datasets from independent studies. While certain methods showed good consistency, some widely used methods were observed to make a substantial number of conflicting findings. Overall, the highest consistency without unnecessary reduction in sensitivity was attained by analyzing total sum scaling (TSS) normalized counts with a non-parametric method (Wilcoxon test or ordinal regression model) or linear regression (MaAsLin2). Comparable performance was also attained by analyzing presence/absence of taxa with logistic regression. In conclusion, while numerous sophisticated methods for differential abundance analysis have been developed, elementary methods seem to provide more consistent results without unnecessarily compromising sensitivity. We therefore suggest that the elementary methods should be preferred in microbial differential abundance analysis when replicability needs to be emphasized.