Adaptation of the technology readiness levels for impact assessment in implementation sciences: The TRL-IS checklist

Background: Intervention development is a critical process in implementation research. There are key stages involved in the process to design, pilot, demonstrate and release a technology or an intervention. The Technology Readiness Level (TRL) is a globally accepted instrument for assessing the maturity of research development. However, the original levels do not fit all, and
some adjustments are required for its applicability in implementation sciences.
Aims: This study aimed to gather the prior knowledge base on TRL in public and population health research; to develop a standard definition of readiness, and to adapt and validate the TRL to an implementation science context (TRL-IS).

Materials and methods: A Mixed methods approach has been followed in this study. A scoping review using the PRISMA extension (PRISMA-ScR) informed a nominal expert panel for developing a standard definition of readiness and to modify the TRL following an ontoterminology approach. Then the maturity of six practical case study examples were rated by ten researchers using the modified TRL to estimate inter-rater reliability, and a group of experts provided final
content and face validity and feasibility. This mixed methods study included 1) a scoping review to examine the current literature and develop a knowledge base, identify knowledge gaps and to clarify concepts; 2) the development of a standard definition of ‘Readiness’ and related terms; and 3) adaptation of the TRL to implementation science and development of a checklist to rate the maturity of applications. A standard definition of readiness and related terms was produced by the core team, and an international nominal group (n = 30) was conducted to discuss and validate the definition and
terms, and the location of ‘Readiness’ in the initiation and early development phases of implementation.
Following feedback from the nominal group, the development of the TRL-IS was finalised and a TRL-IS rating checklist was developed to rate the maturity of applications. The TRL-IS checklist was tested using six cases based on real world studies on implementation research. The inter-rater reliability of the TRL-IS was evaluated by ten raters and finally six raters evaluated the content and face validity, and feasibility, of the TRL-IS checklist using the System Usability
Scale (SUS).
Results: Few papers (n = 11) utilised the TRL to evaluate the readiness of readiness of health and social science implementation research. The main changes in the adaption of the TRL-IS included the removal of laboratory testing, limiting the use of “operational” environment and a clearer distinction between level 6 (pilot in a relevant environment) and 7 (demonstration in the real
world prior to release). The adapted version was considered relevant by the expert panel. The TRL-IS checklist showed evidence of good inter-rater reliability (ICC = 0.90 with 95 % confident interval = 0.74–0.98, p < .001) and provides a consistent metric.
Conclusions: In spite of recommendations made by national and international research funding agencies, few health and social science implementation studies include the TRL as part of their evaluation protocol. The TRL-IS offers a high degree of conceptual clarity between scientific maturity phases or readiness levels, and good reliability among raters of varying experience. This study highlights that adoption of the TRL-IS framework in implementation sciences will bolster the scientific robustness and comparability of research maturity in this domain.
research; to develop a standard definition of readiness, and to adapt and validate the TRL to an
implementation science context (TRL-IS).
Materials and methods: A Mixed methods approach has been followed in this study. A scoping
review using the PRISMA extension (PRISMA-ScR) informed a nominal expert panel for developing
a standard definition of readiness and to modify the TRL following an ontoterminology
approach. Then the maturity of six practical case study examples were rated by ten researchers
using the modified TRL to estimate inter-rater reliability, and a group of experts provided final
content and face validity and feasibility.
This mixed methods study included 1) a scoping review to examine the current literature and
develop a knowledge base, identify knowledge gaps and to clarify concepts; 2) the development
of a standard definition of ‘Readiness’ and related terms; and 3) adaptation of the TRL to
implementation science and development of a checklist to rate the maturity of applications.
A standard definition of readiness and related terms was produced by the core team, and an
international nominal group (n = 30) was conducted to discuss and validate the definition and
terms, and the location of ‘Readiness’ in the initiation and early development phases of
implementation.
Following feedback from the nominal group, the development of the TRL-IS was finalised and a
TRL-IS rating checklist was developed to rate the maturity of applications. The TRL-IS checklist
was tested using six cases based on real world studies on implementation research.
The inter-rater reliability of the TRL-IS was evaluated by ten raters and finally six raters evaluated
the content and face validity, and feasibility, of the TRL-IS checklist using the System Usability
Scale (SUS).
Results: Few papers (n = 11) utilised the TRL to evaluate the readiness of readiness of health and
social science implementation research. The main changes in the adaption of the TRL-IS included