Ecological Simulations Need Effect Sizes, Not Significance Tests

Ecologists are misusing statistical significance tests when analyzing computer simulation models, producing meaningless p-values that can be manipulated by simply running more simulations. The researchers argue scientists should focus on the actual size of differences between model scenarios instead of statistical significance. The authors reviewed recent ecological literature to identify cases where researchers used statistical tests (like ANOVA) on simulation model outputs, then analyzed why this practice is problematic using statistical theory and specific published examples. Statistical power in simulations can be arbitrarily high since researchers can run unlimited replications, making p-values meaningless. Null hypotheses in model comparisons are known to be false before testing begins, invalidating the premise of the statistical test. With 24,000 simulation runs, researchers can produce extremely small p-values regardless of biological effect size. Focus should shift to quantifying effect sizes and biological significance rather than statistical significance. The research addresses a fundamental methodological problem in ecological modeling that could lead to incorrect conclusions about ecosystem dynamics and management decisions if researchers continue misinterpreting simulation results.

Citation

White, J. Wilson; Rassweiler, Andrew; Samhouri, Jameal F.; Stier, Adrian C.; White, Crow (2014). Ecologists should not use statistical significance tests to interpret simulation model results. Oikos.

Read the full paper

This paper is Open Access.