Reliability of Chalmers' scale to assess quality in meta-analyses on pharmacological treatments for osteoporosis

Anick Bérard, Nathalie Andreu, Jean Pierre Tétrault, Théophile Niyonsenga, Daniel Myhal

Research output: Contribution to journalArticle

11 Citations (Scopus)

Abstract

PURPOSE: This study estimates the inter-rater and test-retest reliability of Chalmers' quality score scale in the context of bone mass loss and fracture rate in postmenopausal women. METHODS: An exhaustive literature search was performed on Medline to locate clinical trials studying the effect of medication use on bone mass loss and fracture rate in postmenopausal women. Twenty articles were randomly selected and four raters independently assessed the quality of each article with Chalmers' scale. Among the 20 articles, 10 were blinded on authors' names, journal, year of publication and source of funding. Raters were also asked to assess all 20 articles one more time, two months after the first evaluation. Intraclass (ICC) and test-retest correlation coefficients were calculated. RESULTS: The overall inter-rater ICC was 0.66 [0.55, 0.79](95%). The overall test-retest reliability of Chalmers' scale was 0.81 [0.67, 0.98](95%). When ratings were stratified according to articles' blinding status, blinded assessments generated a smaller inter-rater ICC than non-blinded assessments: 0.30 [0.17, 0.53](95%) vs. 0.80 [0.71, 0.90](95%). In addition, analyzing sub-scales separately generated different estimates of reliability. CONCLUSIONS: This study shows that the reliability of the quality scale developed by Chalmers substantially varies between sub-scales, and is highly dependent on articles' blinding status. The possibility of bias in rating non-blinded articles can not be ruled out. The reliability of the scale can also be dependent on the outcome studied. (C) 2000 Elsevier Science Inc.

Original languageEnglish
Pages (from-to)498-503
Number of pages6
JournalAnnals of Epidemiology
Volume10
Issue number8
DOIs
Publication statusPublished - 23 Dec 2000
Externally publishedYes

Fingerprint

Reproducibility of Results
Osteoporosis
Meta-Analysis
Pharmacology
Bone and Bones
Names
Publications
Clinical Trials
Therapeutics

Cite this

Bérard, Anick ; Andreu, Nathalie ; Tétrault, Jean Pierre ; Niyonsenga, Théophile ; Myhal, Daniel. / Reliability of Chalmers' scale to assess quality in meta-analyses on pharmacological treatments for osteoporosis. In: Annals of Epidemiology. 2000 ; Vol. 10, No. 8. pp. 498-503.
@article{59b2eb17af9246e6abb9f3d25fcdfe1f,
title = "Reliability of Chalmers' scale to assess quality in meta-analyses on pharmacological treatments for osteoporosis",
abstract = "PURPOSE: This study estimates the inter-rater and test-retest reliability of Chalmers' quality score scale in the context of bone mass loss and fracture rate in postmenopausal women. METHODS: An exhaustive literature search was performed on Medline to locate clinical trials studying the effect of medication use on bone mass loss and fracture rate in postmenopausal women. Twenty articles were randomly selected and four raters independently assessed the quality of each article with Chalmers' scale. Among the 20 articles, 10 were blinded on authors' names, journal, year of publication and source of funding. Raters were also asked to assess all 20 articles one more time, two months after the first evaluation. Intraclass (ICC) and test-retest correlation coefficients were calculated. RESULTS: The overall inter-rater ICC was 0.66 [0.55, 0.79](95{\%}). The overall test-retest reliability of Chalmers' scale was 0.81 [0.67, 0.98](95{\%}). When ratings were stratified according to articles' blinding status, blinded assessments generated a smaller inter-rater ICC than non-blinded assessments: 0.30 [0.17, 0.53](95{\%}) vs. 0.80 [0.71, 0.90](95{\%}). In addition, analyzing sub-scales separately generated different estimates of reliability. CONCLUSIONS: This study shows that the reliability of the quality scale developed by Chalmers substantially varies between sub-scales, and is highly dependent on articles' blinding status. The possibility of bias in rating non-blinded articles can not be ruled out. The reliability of the scale can also be dependent on the outcome studied. (C) 2000 Elsevier Science Inc.",
keywords = "Chalmers' Quality Scale, Inter-Rater Reliability, Meta-Analysis, Quality, Score, Test-Retest Reliability",
author = "Anick B{\'e}rard and Nathalie Andreu and T{\'e}trault, {Jean Pierre} and Th{\'e}ophile Niyonsenga and Daniel Myhal",
year = "2000",
month = "12",
day = "23",
doi = "10.1016/S1047-2797(00)00069-7",
language = "English",
volume = "10",
pages = "498--503",
journal = "Annals of Epidemiology",
issn = "1047-2797",
publisher = "Elsevier Inc.",
number = "8",

}

Reliability of Chalmers' scale to assess quality in meta-analyses on pharmacological treatments for osteoporosis. / Bérard, Anick; Andreu, Nathalie; Tétrault, Jean Pierre; Niyonsenga, Théophile; Myhal, Daniel.

In: Annals of Epidemiology, Vol. 10, No. 8, 23.12.2000, p. 498-503.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Reliability of Chalmers' scale to assess quality in meta-analyses on pharmacological treatments for osteoporosis

AU - Bérard, Anick

AU - Andreu, Nathalie

AU - Tétrault, Jean Pierre

AU - Niyonsenga, Théophile

AU - Myhal, Daniel

PY - 2000/12/23

Y1 - 2000/12/23

N2 - PURPOSE: This study estimates the inter-rater and test-retest reliability of Chalmers' quality score scale in the context of bone mass loss and fracture rate in postmenopausal women. METHODS: An exhaustive literature search was performed on Medline to locate clinical trials studying the effect of medication use on bone mass loss and fracture rate in postmenopausal women. Twenty articles were randomly selected and four raters independently assessed the quality of each article with Chalmers' scale. Among the 20 articles, 10 were blinded on authors' names, journal, year of publication and source of funding. Raters were also asked to assess all 20 articles one more time, two months after the first evaluation. Intraclass (ICC) and test-retest correlation coefficients were calculated. RESULTS: The overall inter-rater ICC was 0.66 [0.55, 0.79](95%). The overall test-retest reliability of Chalmers' scale was 0.81 [0.67, 0.98](95%). When ratings were stratified according to articles' blinding status, blinded assessments generated a smaller inter-rater ICC than non-blinded assessments: 0.30 [0.17, 0.53](95%) vs. 0.80 [0.71, 0.90](95%). In addition, analyzing sub-scales separately generated different estimates of reliability. CONCLUSIONS: This study shows that the reliability of the quality scale developed by Chalmers substantially varies between sub-scales, and is highly dependent on articles' blinding status. The possibility of bias in rating non-blinded articles can not be ruled out. The reliability of the scale can also be dependent on the outcome studied. (C) 2000 Elsevier Science Inc.

AB - PURPOSE: This study estimates the inter-rater and test-retest reliability of Chalmers' quality score scale in the context of bone mass loss and fracture rate in postmenopausal women. METHODS: An exhaustive literature search was performed on Medline to locate clinical trials studying the effect of medication use on bone mass loss and fracture rate in postmenopausal women. Twenty articles were randomly selected and four raters independently assessed the quality of each article with Chalmers' scale. Among the 20 articles, 10 were blinded on authors' names, journal, year of publication and source of funding. Raters were also asked to assess all 20 articles one more time, two months after the first evaluation. Intraclass (ICC) and test-retest correlation coefficients were calculated. RESULTS: The overall inter-rater ICC was 0.66 [0.55, 0.79](95%). The overall test-retest reliability of Chalmers' scale was 0.81 [0.67, 0.98](95%). When ratings were stratified according to articles' blinding status, blinded assessments generated a smaller inter-rater ICC than non-blinded assessments: 0.30 [0.17, 0.53](95%) vs. 0.80 [0.71, 0.90](95%). In addition, analyzing sub-scales separately generated different estimates of reliability. CONCLUSIONS: This study shows that the reliability of the quality scale developed by Chalmers substantially varies between sub-scales, and is highly dependent on articles' blinding status. The possibility of bias in rating non-blinded articles can not be ruled out. The reliability of the scale can also be dependent on the outcome studied. (C) 2000 Elsevier Science Inc.

KW - Chalmers' Quality Scale

KW - Inter-Rater Reliability

KW - Meta-Analysis

KW - Quality

KW - Score

KW - Test-Retest Reliability

UR - http://www.scopus.com/inward/record.url?scp=0033665166&partnerID=8YFLogxK

U2 - 10.1016/S1047-2797(00)00069-7

DO - 10.1016/S1047-2797(00)00069-7

M3 - Article

VL - 10

SP - 498

EP - 503

JO - Annals of Epidemiology

JF - Annals of Epidemiology

SN - 1047-2797

IS - 8

ER -