Consequences of relying on statistical significance: Some illustrations.
Van Calster B., Steyerberg EW., Collins GS., Smits T.
BACKGROUND: Despite regular criticisms of null hypothesis significance testing (NHST), a focus on testing persists, sometimes in the belief to get published and sometimes encouraged by journal reviewers. This paper aims to demonstrate known key limitations of NHST using simple nontechnical illustrations. DESIGN: The first illustration is based on simulated data of 20 000 studies that compare two groups for an outcome event. The true effect size (difference in event rates) and sample size (20-100 per group) were varied. The second illustration used real data from a meta-analysis on alpha-blockers for the treatment of ureteric stones. RESULTS: The simulations demonstrated the large between-study variability in P-values (range between