Advertisement
ReviewVolume 126p154-159October 2020

The quality of evidence for medical interventions does not improve or worsen: a metaepidemiological study of Cochrane reviews

Jeremy Howicka Send email to [email protected] ∙ Despina Koletsib,1 ∙ Nikolaos Pandisc ∙ … ∙ Padhraig S. Flemingd ∙ Martin Loefe ∙ Harald Walache,f ∙ Stefan Schmidtg ∙ John P.A. Ioannidish … Show more
Affiliations & Notes
aFaculty of Philosophy, University of Oxford, Oxford, OX2 6GG, UK
bClinic of Orthodontics and Pediatric Dentistry, Center of Dental Medicine, University of Zurich, Zurich, Switzerland
cDepartment of Orthodontics and Dentofacial Orthopedics, School of Dental Medicine, Medical Faculty, University of Bern, Bern, Switzerland
dInstitute of Dentistry, Queen Mary, University of London, London, United Kingdom
eCHS-Institute, Berlin, Germany
fDepartment of Pediatric Gastroenterlogy, Poznan University of the Medical Sciences, Poznan, Poland
gDepartment of Psychosomatic Medicine and Psychotherapy, Medical Center, University of Freiburg, Freiburg, Germany
hDepartments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA
1
Joint first author.
Article Info
Publication History:
Accepted August 5, 2020; Published online September 2, 2020
Footnotes:
Support: The writing of this protocol was not independently funded.
Declaration of interest: None of the authors have any conflicts of interests related to this study.
Registration: Open Science Framework: Howick, J., Koletsi, D., Fleming, P., Schmidt, S., Loef, M., Walach, H., … Ioannidis, J. (2020, March 30). Has the Quality of Evidence for Medical Interventions Improved? Protocol for a Metaepidemiological Study. Retrieved from osf.io/bw7ky.
DOI: 10.1016/j.jclinepi.2020.08.005 External LinkAlso available on ScienceDirect External Link
Copyright: © 2020 Elsevier Inc. All rights reserved.
Linked Articles
Cover Image - Journal of Clinical Epidemiology, Volume 126, Issue undefined

Abstract

Objectives

The objective of the study was to determine the change in quality of evidence in updates of Cochrane reviews that were initially published between January 1, 2013 and June 30, 2014. We used the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) system to document evidence quality.

Study Design and Setting

We searched the Cochrane Database of Systematic Reviews on March 20, 2020 to identify which of the reviews from the initial (2013/14) sample had been updated. Using the same methods to determine the quality of evidence in the previous analysis, we assessed the quality of evidence for the first-listed primary outcomes in the updated reviews.

Results

Of the 608 reviews in the original sample, 154 had been updated with and 151 contained available data for both original and updated systematic reviews (24.8%). The updated reviews included: 15 (9.9%) with high-quality evidence, 56 (37.1%) with moderate-quality evidence, 47 (31.1%) with low-quality evidence, and 33 (21.9%) with very low-quality evidence. No change in the GRADE quality of evidence was found for most (103, 68.2%) of the updated reviews. The quality of evidence rating was downgraded in 28 reviews (58.3%) and upgraded in 20 (41.7%), although only six reviews were promoted to high quality.

Conclusion

Updated systematic reviews continued to suggest that only a minority of outcomes for health care interventions are supported by high-quality evidence. The quality of the evidence did not consistently improve or worsen in updated reviews.

Keywords

  1. Systematic review
  2. Evidence
  3. Quality score
  4. Meta-analysis
  5. Effectiveness
  6. GRADE

Get full text access

Log in, subscribe or purchase for full access.

References

1.
Ezzo, J. ∙ Bausell, B. ∙ Moerman, D.E. ...
Reviewing the reviews. How strong is the evidence? How clear are the conclusions?
Int J Technol Assess Health Care. 2001; 17(4):457-466
Crossref
PubMed
Google Scholar
2.
Garrow, J.S.
What to do about CAM: how much of orthodox medicine is evidence based?
BMJ. 2007; 335:951
Crossref
Scopus (28)
PubMed
Google Scholar
3.
El Dib, R.P. ∙ Atallah, A.N. ∙ Andriolo, R.B.
Mapping the Cochrane evidence for decision making in health care
J Eval Clin Pract. 2007; 13:689-692
Crossref
Scopus (75)
PubMed
Google Scholar
4.
Balshem, H. ∙ Helfand, M. ∙ Schunemann, H.J. ...
GRADE guidelines: 3. Rating the quality of evidence
J Clin Epidemiol. 2011; 64:401-406
Full Text
Full Text (PDF)
Scopus (5487)
PubMed
Google Scholar
5.
Fleming, P.S. ∙ Koletsi, D. ∙ Ioannidis, J.P. ...
High quality of the evidence for medical and other health-related interventions was uncommon in Cochrane systematic reviews
J Clin Epidemiol. 2016; 78:34-42
Full Text
Full Text (PDF)
Scopus (50)
PubMed
Google Scholar
6.
Chalmers, I. ∙ Haynes, B.
Reporting, updating, and correcting systematic reviews of the effects of health care
BMJ. 1994; 309:862-865
Crossref
Scopus (132)
PubMed
Google Scholar
7.
Garritty, C. ∙ Tsertsvadze, A. ∙ Tricco, A.C. ...
Updating systematic reviews: an international survey
PLoS One. 2010; 5:e9914
Crossref
Scopus (44)
PubMed
Google Scholar
8.
Moher, D. ∙ Tsertsvadze, A.
Systematic reviews: when is an update an update?
Lancet. 2006; 367(9514):881-883
Full Text
Full Text (PDF)
Scopus (89)
PubMed
Google Scholar
9.
French, S.D. ∙ McDonald, S. ∙ McKenzie, J.E. ...
Investing in updating: how do conclusions change when Cochrane systematic reviews are updated?
BMC Med Res Methodol. 2005; 5:33
Crossref
Scopus (47)
PubMed
Google Scholar
10.
Higgins, J.J. ∙ Thomas, J.C. ∙ Chandler, J. ...
Cochrane Handbook for Systematic Reviews of Interventions version 6.0
The Cochrane Collaboration, Chichester, 2019
Crossref
Scopus (34091)
Google Scholar
11.
Shojania, K.G. ∙ Sampson, M. ∙ Ansari, M.T. ...
How quickly do systematic reviews go out of date? A survival analysis
Ann Intern Med. 2007; 147:224-233
Crossref
Scopus (606)
PubMed
Google Scholar
12.
Mattick, R.P. ∙ Breen, C. ∙ Kimber, J. ...
Buprenorphine maintenance versus placebo or methadone maintenance for opioid dependence
Cochrane Database Syst Rev. 2014; CD002207
PubMed
Google Scholar
13.
Higgins, J.J. ∙ Green, S.
The Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]
The Cochrane Collaboration, Chichester, 2011
Google Scholar
14.
Berkman, N.D. ∙ Lohr, K.N. ∙ Morgan, L.C. ...
Interrater reliability of grading strength of evidence varies with the complexity of the evidence in systematic reviews
J Clin Epidemiol. 2013; 66:1105-1107.e1
Full Text
Full Text (PDF)
Scopus (22)
PubMed
Google Scholar
15.
Hartling, L. ∙ Hamm, M.P. ∙ Milne, A. ...
Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs
J Clin Epidemiol. 2013; 66:973-981
Full Text
Full Text (PDF)
Scopus (128)
PubMed
Google Scholar
16.
Boardman, H.M. ∙ Hartley, L. ∙ Eisinga, A. ...
Hormone therapy for preventing cardiovascular disease in post-menopausal women
Cochrane Database Syst Rev. 2015; CD002229
PubMed
Google Scholar
17.
Hakoum, M.B. ∙ Kahale, L.A. ∙ Tsolakian, I.G. ...
Anticoagulation for the initial treatment of venous thromboembolism in people with cancer
Cochrane Database Syst Rev. 2018; 1:CD006649
PubMed
Google Scholar

Article metrics

The content on this site is intended for healthcare professionals.


All content on this site: Copyright © 2026 Elsevier Inc., its licensors, and contributors.
All rights are reserved, including those for text and data mining, AI training, and similar technologies.
For all open access content, the relevant licensing terms apply.

Privacy Policy      Terms and Conditions   Accessibility   Help & Contact      Responsible AI


RELX