Methodologies for assessing telemedicine: a systematic review of reviews

Background and objectives: Previous reviews have expressed concerns about the quality of telemedicine studies. There is debate about shortcomings and appropriate methodologies. The aim of this review of systematic reviews of telemedicine is to summarize methodologies used in telemedicine research, discuss knowledge gaps and recommendations and suggest methodological approaches for further research.

Methods: We conducted a review of systematic reviews of telemedicine according to a protocol listing explicit methods, selection criteria, data collection and quality assessment procedures. We included reviews where authors explicitly addressed and made recommendations for assessment methodologies. We did a qualitative analysis of the reviews included, sensitized by two broad methodological positions; positivist and naturalistic approaches. The analysis focused on methodologies used in the primary studies included in the reviews as reported by the review authors, and methodological recommendations made by the review authors.

Results: We identified 1593 titles/abstracts. We included 50 reviews that explicitly addressed assessment methodologies. One group of reviews recommended larger and more rigorously designed controlled studies to assess the impacts of telemedicine; a second group proposed standardisation of populations, and/or interventions and outcome measures to reduce heterogeneity and facilitate meta-analysis; a third group recommended combining quantitative and qualitative research methods; and others applying different naturalistic approaches including methodologies addressing mutual adaptations of services and users; politically driven action research and formative research aimed at collaboration to ensure capacity for improvement of services in natural settings.

Conclusions: Larger and more rigorous studies are crucial for the production of evidence of effectiveness of unambiguous telemedicine services for pre defined outcome measures. Summative methodologies acknowledging telemedicine as complex innovations and outcomes as partly contingent on values, meanings and contexts are also important. So are formative, naturalistic methodologies that acknowledge telemedicine as ongoing collaborative achievements and engage with stakeholders, including patients to produce and conceptualise new and effective telemedicine innovations.