Adaptation of the technology readiness levels for impact assessment in implementation sciences: The TRL-IS checklist

    Research output: Contribution to journalArticlepeer-review

    Abstract

    Background: Intervention development is a critical process in implementation research. There are key stages involved in the process to design, pilot, demonstrate and release a technology or an intervention. The Technology Readiness Level (TRL) is a globally accepted instrument for assessing the maturity of research development. However, the original levels do not fit all, and
    some adjustments are required for its applicability in implementation sciences.
    Aims: This study aimed to gather the prior knowledge base on TRL in public and population health research; to develop a standard definition of readiness, and to adapt and validate the TRL to an implementation science context (TRL-IS).

    Materials and methods: A Mixed methods approach has been followed in this study. A scoping review using the PRISMA extension (PRISMA-ScR) informed a nominal expert panel for developing a standard definition of readiness and to modify the TRL following an ontoterminology approach. Then the maturity of six practical case study examples were rated by ten researchers using the modified TRL to estimate inter-rater reliability, and a group of experts provided final
    content and face validity and feasibility. This mixed methods study included 1) a scoping review to examine the current literature and develop a knowledge base, identify knowledge gaps and to clarify concepts; 2) the development of a standard definition of ‘Readiness’ and related terms; and 3) adaptation of the TRL to implementation science and development of a checklist to rate the maturity of applications. A standard definition of readiness and related terms was produced by the core team, and an international nominal group (n = 30) was conducted to discuss and validate the definition and
    terms, and the location of ‘Readiness’ in the initiation and early development phases of implementation.
    Following feedback from the nominal group, the development of the TRL-IS was finalised and a TRL-IS rating checklist was developed to rate the maturity of applications. The TRL-IS checklist was tested using six cases based on real world studies on implementation research. The inter-rater reliability of the TRL-IS was evaluated by ten raters and finally six raters evaluated the content and face validity, and feasibility, of the TRL-IS checklist using the System Usability
    Scale (SUS).
    Results: Few papers (n = 11) utilised the TRL to evaluate the readiness of readiness of health and social science implementation research. The main changes in the adaption of the TRL-IS included the removal of laboratory testing, limiting the use of “operational” environment and a clearer distinction between level 6 (pilot in a relevant environment) and 7 (demonstration in the real
    world prior to release). The adapted version was considered relevant by the expert panel. The TRL-IS checklist showed evidence of good inter-rater reliability (ICC = 0.90 with 95 % confident interval = 0.74–0.98, p < .001) and provides a consistent metric.
    Conclusions: In spite of recommendations made by national and international research funding agencies, few health and social science implementation studies include the TRL as part of their evaluation protocol. The TRL-IS offers a high degree of conceptual clarity between scientific maturity phases or readiness levels, and good reliability among raters of varying experience. This study highlights that adoption of the TRL-IS framework in implementation sciences will bolster the scientific robustness and comparability of research maturity in this domain.
    research; to develop a standard definition of readiness, and to adapt and validate the TRL to an
    implementation science context (TRL-IS).
    Materials and methods: A Mixed methods approach has been followed in this study. A scoping
    review using the PRISMA extension (PRISMA-ScR) informed a nominal expert panel for developing
    a standard definition of readiness and to modify the TRL following an ontoterminology
    approach. Then the maturity of six practical case study examples were rated by ten researchers
    using the modified TRL to estimate inter-rater reliability, and a group of experts provided final
    content and face validity and feasibility.
    This mixed methods study included 1) a scoping review to examine the current literature and
    develop a knowledge base, identify knowledge gaps and to clarify concepts; 2) the development
    of a standard definition of ‘Readiness’ and related terms; and 3) adaptation of the TRL to
    implementation science and development of a checklist to rate the maturity of applications.
    A standard definition of readiness and related terms was produced by the core team, and an
    international nominal group (n = 30) was conducted to discuss and validate the definition and
    terms, and the location of ‘Readiness’ in the initiation and early development phases of
    implementation.
    Following feedback from the nominal group, the development of the TRL-IS was finalised and a
    TRL-IS rating checklist was developed to rate the maturity of applications. The TRL-IS checklist
    was tested using six cases based on real world studies on implementation research.
    The inter-rater reliability of the TRL-IS was evaluated by ten raters and finally six raters evaluated
    the content and face validity, and feasibility, of the TRL-IS checklist using the System Usability
    Scale (SUS).
    Results: Few papers (n = 11) utilised the TRL to evaluate the readiness of readiness of health and
    social science implementation research. The main changes in the adaption of the TRL-IS included
    Original languageEnglish
    Article numbere29930
    Pages (from-to)1-16
    Number of pages16
    JournalHeliyon
    Volume10
    Issue number9
    DOIs
    Publication statusE-pub ahead of print - 15 May 2024

    Fingerprint

    Dive into the research topics of 'Adaptation of the technology readiness levels for impact assessment in implementation sciences: The TRL-IS checklist'. Together they form a unique fingerprint.

    Cite this