Replication Success under Questionable Research Practices – A Simulation Study
Increasing evidence suggests that the reproducibility and replicability of scientific findings is threatened by researchers employing questionable research practices (QRP) in order to achieve publishable, positive and significant results. Numerous metrics have been developed to determine replication success but it has not yet been established how well those metrics perform in the presence of QRPs. This paper aims to compare the performance of different metrics quantifying replication success in the presence of four different types of QRPs: cherry picking, questionable interim analyses, questionable inclusion of covariates, and questionable subgroup analyses. Our results show that the metric based on the golden sceptical p-value does better in maintaining low values of overall type-I error rate, but often needs larger replication sample sizes, especially when severe QRPs are employed.