TY - JOUR
T1 - Why there are so many contradicted or exaggerated findings in highly-cited clinical research?
AU - Lu, Mengyi
AU - Liu, Suyu
AU - Yuan, Ying
N1 - Publisher Copyright:
© 2022 Elsevier Inc.
PY - 2022/7
Y1 - 2022/7
N2 - It is not uncommon that clinical studies of the same intervention contradicted with each other, e.g., one study produced positive results, while the other produced negative results. Ioanndis (2005a) found that among 49 highly-cited original clinical research studies, published in New England Journal of Medicine, Journal of the American Medical Association, Lancet or in a high-impact medical specialty journal, 32% of them were contradicted in subsequent large-scale studies, or were shown to have potentially overestimated the efficacy of the experimental intervention. This finding is disturbing and of serious concern given the widespread impact of these highly-cited studies and the rigorous standards used to design and conduct the studies. We perform Bayesian analysis of these highly-cited clinical studies based on Bayesian factor. We identified one cause of the issue: p values strongly overstated the experimental evidence. For the highly-cited studies, when the p value was 0.05, there was a 74.4% percentage chance that the null hypothesis was true. The use of a p value of 0.05 as the criterion for significance caused many researchers to mistakenly draw conclusions of positive findings, which were then contradicted by subsequent large-scale studies.
AB - It is not uncommon that clinical studies of the same intervention contradicted with each other, e.g., one study produced positive results, while the other produced negative results. Ioanndis (2005a) found that among 49 highly-cited original clinical research studies, published in New England Journal of Medicine, Journal of the American Medical Association, Lancet or in a high-impact medical specialty journal, 32% of them were contradicted in subsequent large-scale studies, or were shown to have potentially overestimated the efficacy of the experimental intervention. This finding is disturbing and of serious concern given the widespread impact of these highly-cited studies and the rigorous standards used to design and conduct the studies. We perform Bayesian analysis of these highly-cited clinical studies based on Bayesian factor. We identified one cause of the issue: p values strongly overstated the experimental evidence. For the highly-cited studies, when the p value was 0.05, there was a 74.4% percentage chance that the null hypothesis was true. The use of a p value of 0.05 as the criterion for significance caused many researchers to mistakenly draw conclusions of positive findings, which were then contradicted by subsequent large-scale studies.
KW - Contradicted
KW - Highly-cited
KW - p values
KW - Test statistics
UR - http://www.scopus.com/inward/record.url?scp=85130404094&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85130404094&partnerID=8YFLogxK
U2 - 10.1016/j.cct.2022.106782
DO - 10.1016/j.cct.2022.106782
M3 - Article
C2 - 35525531
AN - SCOPUS:85130404094
SN - 1551-7144
VL - 118
JO - Contemporary Clinical Trials
JF - Contemporary Clinical Trials
M1 - 106782
ER -