Development, inter-rater reliability and feasibility of a checklist to assess implementation (Ch-IMP) in systematic reviews

The case of provider-based prevention and treatment programs targeting children and youth

Margaret Cargo, Ivana Stankov, James Thomas, Michael Saini, Patricia Rogers, Evan Mayo-Wilson, Karin Hannes

Research output: Contribution to journalReview article

9 Citations (Scopus)

Abstract

Background: Several papers report deficiencies in the reporting of information about the implementation of interventions in clinical trials. Information about implementation is also required in systematic reviews of complex interventions to facilitate the translation and uptake of evidence of provider-based prevention and treatment programs. To capture whether and how implementation is assessed within systematic effectiveness reviews, we developed a checklist for implementation (Ch-IMP) and piloted it in a cohort of reviews on provider-based prevention and treatment interventions for children and young people. This paper reports on the inter-rater reliability, feasibility and reasons for discrepant ratings. Methods: Checklist domains were informed by a framework for program theory; items within domains were generated from a literature review. The checklist was pilot-tested on a cohort of 27 effectiveness reviews targeting children and youth. Two raters independently extracted information on 47 items. Inter-rater reliability was evaluated using percentage agreement and unweighted kappa coefficients. Reasons for discrepant ratings were content analysed. Results: Kappa coefficients ranged from 0.37 to 1.00 and were not influenced by one-sided bias. Most kappa values were classified as excellent (n = 20) or good (n = 17) with a few items categorised as fair (n = 7) or poor (n = 1). Prevalence-adjusted kappa coefficients indicate good or excellent agreement for all but one item. Four areas contributed to scoring discrepancies: 1) clarity or sufficiency of information provided in the review; 2) information missed in the review; 3) issues encountered with the tool; and 4) issues encountered at the review level. Use of the tool demands time investment and it requires adjustment to improve its feasibility for wider use. Conclusions: The case of provider-based prevention and treatment interventions showed relevancy in developing and piloting the Ch-IMP as a useful tool for assessing the extent to which systematic reviews assess the quality of implementation. The checklist could be used by authors and editors to improve the quality of systematic reviews, and shows promise as a pedagogical tool to facilitate the extraction and reporting of implementation characteristics.

Original languageEnglish
Article number73
Pages (from-to)1-18
Number of pages18
JournalBMC Medical Research Methodology
Volume15
Issue number1
DOIs
Publication statusPublished - 7 Sep 2015
Externally publishedYes

Fingerprint

Checklist
Therapeutics
Clinical Trials

Cite this

@article{8608475b5ecb457696acb05bc5b4ad5a,
title = "Development, inter-rater reliability and feasibility of a checklist to assess implementation (Ch-IMP) in systematic reviews: The case of provider-based prevention and treatment programs targeting children and youth",
abstract = "Background: Several papers report deficiencies in the reporting of information about the implementation of interventions in clinical trials. Information about implementation is also required in systematic reviews of complex interventions to facilitate the translation and uptake of evidence of provider-based prevention and treatment programs. To capture whether and how implementation is assessed within systematic effectiveness reviews, we developed a checklist for implementation (Ch-IMP) and piloted it in a cohort of reviews on provider-based prevention and treatment interventions for children and young people. This paper reports on the inter-rater reliability, feasibility and reasons for discrepant ratings. Methods: Checklist domains were informed by a framework for program theory; items within domains were generated from a literature review. The checklist was pilot-tested on a cohort of 27 effectiveness reviews targeting children and youth. Two raters independently extracted information on 47 items. Inter-rater reliability was evaluated using percentage agreement and unweighted kappa coefficients. Reasons for discrepant ratings were content analysed. Results: Kappa coefficients ranged from 0.37 to 1.00 and were not influenced by one-sided bias. Most kappa values were classified as excellent (n = 20) or good (n = 17) with a few items categorised as fair (n = 7) or poor (n = 1). Prevalence-adjusted kappa coefficients indicate good or excellent agreement for all but one item. Four areas contributed to scoring discrepancies: 1) clarity or sufficiency of information provided in the review; 2) information missed in the review; 3) issues encountered with the tool; and 4) issues encountered at the review level. Use of the tool demands time investment and it requires adjustment to improve its feasibility for wider use. Conclusions: The case of provider-based prevention and treatment interventions showed relevancy in developing and piloting the Ch-IMP as a useful tool for assessing the extent to which systematic reviews assess the quality of implementation. The checklist could be used by authors and editors to improve the quality of systematic reviews, and shows promise as a pedagogical tool to facilitate the extraction and reporting of implementation characteristics.",
keywords = "Checklist development, Children and youth, Complex interventions, Implementation, Process evaluation, Provider-based interventions, Systematic reviews, Theory-driven reviews",
author = "Margaret Cargo and Ivana Stankov and James Thomas and Michael Saini and Patricia Rogers and Evan Mayo-Wilson and Karin Hannes",
year = "2015",
month = "9",
day = "7",
doi = "10.1186/s12874-015-0037-7",
language = "English",
volume = "15",
pages = "1--18",
journal = "BMC Medical Research Methodology",
issn = "1471-2288",
publisher = "BioMed Central",
number = "1",

}

Development, inter-rater reliability and feasibility of a checklist to assess implementation (Ch-IMP) in systematic reviews : The case of provider-based prevention and treatment programs targeting children and youth. / Cargo, Margaret; Stankov, Ivana; Thomas, James; Saini, Michael; Rogers, Patricia; Mayo-Wilson, Evan; Hannes, Karin.

In: BMC Medical Research Methodology, Vol. 15, No. 1, 73, 07.09.2015, p. 1-18.

Research output: Contribution to journalReview article

TY - JOUR

T1 - Development, inter-rater reliability and feasibility of a checklist to assess implementation (Ch-IMP) in systematic reviews

T2 - The case of provider-based prevention and treatment programs targeting children and youth

AU - Cargo, Margaret

AU - Stankov, Ivana

AU - Thomas, James

AU - Saini, Michael

AU - Rogers, Patricia

AU - Mayo-Wilson, Evan

AU - Hannes, Karin

PY - 2015/9/7

Y1 - 2015/9/7

N2 - Background: Several papers report deficiencies in the reporting of information about the implementation of interventions in clinical trials. Information about implementation is also required in systematic reviews of complex interventions to facilitate the translation and uptake of evidence of provider-based prevention and treatment programs. To capture whether and how implementation is assessed within systematic effectiveness reviews, we developed a checklist for implementation (Ch-IMP) and piloted it in a cohort of reviews on provider-based prevention and treatment interventions for children and young people. This paper reports on the inter-rater reliability, feasibility and reasons for discrepant ratings. Methods: Checklist domains were informed by a framework for program theory; items within domains were generated from a literature review. The checklist was pilot-tested on a cohort of 27 effectiveness reviews targeting children and youth. Two raters independently extracted information on 47 items. Inter-rater reliability was evaluated using percentage agreement and unweighted kappa coefficients. Reasons for discrepant ratings were content analysed. Results: Kappa coefficients ranged from 0.37 to 1.00 and were not influenced by one-sided bias. Most kappa values were classified as excellent (n = 20) or good (n = 17) with a few items categorised as fair (n = 7) or poor (n = 1). Prevalence-adjusted kappa coefficients indicate good or excellent agreement for all but one item. Four areas contributed to scoring discrepancies: 1) clarity or sufficiency of information provided in the review; 2) information missed in the review; 3) issues encountered with the tool; and 4) issues encountered at the review level. Use of the tool demands time investment and it requires adjustment to improve its feasibility for wider use. Conclusions: The case of provider-based prevention and treatment interventions showed relevancy in developing and piloting the Ch-IMP as a useful tool for assessing the extent to which systematic reviews assess the quality of implementation. The checklist could be used by authors and editors to improve the quality of systematic reviews, and shows promise as a pedagogical tool to facilitate the extraction and reporting of implementation characteristics.

AB - Background: Several papers report deficiencies in the reporting of information about the implementation of interventions in clinical trials. Information about implementation is also required in systematic reviews of complex interventions to facilitate the translation and uptake of evidence of provider-based prevention and treatment programs. To capture whether and how implementation is assessed within systematic effectiveness reviews, we developed a checklist for implementation (Ch-IMP) and piloted it in a cohort of reviews on provider-based prevention and treatment interventions for children and young people. This paper reports on the inter-rater reliability, feasibility and reasons for discrepant ratings. Methods: Checklist domains were informed by a framework for program theory; items within domains were generated from a literature review. The checklist was pilot-tested on a cohort of 27 effectiveness reviews targeting children and youth. Two raters independently extracted information on 47 items. Inter-rater reliability was evaluated using percentage agreement and unweighted kappa coefficients. Reasons for discrepant ratings were content analysed. Results: Kappa coefficients ranged from 0.37 to 1.00 and were not influenced by one-sided bias. Most kappa values were classified as excellent (n = 20) or good (n = 17) with a few items categorised as fair (n = 7) or poor (n = 1). Prevalence-adjusted kappa coefficients indicate good or excellent agreement for all but one item. Four areas contributed to scoring discrepancies: 1) clarity or sufficiency of information provided in the review; 2) information missed in the review; 3) issues encountered with the tool; and 4) issues encountered at the review level. Use of the tool demands time investment and it requires adjustment to improve its feasibility for wider use. Conclusions: The case of provider-based prevention and treatment interventions showed relevancy in developing and piloting the Ch-IMP as a useful tool for assessing the extent to which systematic reviews assess the quality of implementation. The checklist could be used by authors and editors to improve the quality of systematic reviews, and shows promise as a pedagogical tool to facilitate the extraction and reporting of implementation characteristics.

KW - Checklist development

KW - Children and youth

KW - Complex interventions

KW - Implementation

KW - Process evaluation

KW - Provider-based interventions

KW - Systematic reviews

KW - Theory-driven reviews

UR - http://www.scopus.com/inward/record.url?scp=84940862312&partnerID=8YFLogxK

U2 - 10.1186/s12874-015-0037-7

DO - 10.1186/s12874-015-0037-7

M3 - Review article

VL - 15

SP - 1

EP - 18

JO - BMC Medical Research Methodology

JF - BMC Medical Research Methodology

SN - 1471-2288

IS - 1

M1 - 73

ER -