Article Text
PDF
Abstract
Introduction Digital health technologies (DHTs) such as electronic health records, clinical decision support systems and electronic prescribing systems are widely used in healthcare. While adoption of DHTs can improve healthcare delivery, information quality (IQ) problems associated with DHTs can compromise quality and safety of care. The clinical information quality (CLIQ) framework for digital health is a novel approach to assessing the quality of clinical information from DHTs. This study aims to appraise the CLIQ framework by exploring clinicians’ perspectives on the relevance, definition and assessment of IQ dimensions as defined in the framework. This study will adapt the CLIQ framework to the needs of clinical information users—the clinicians. The contextualised CLIQ framework will offer a pragmatic approach to assessing clinical information from DHTs and may help to forestall IQ problems that can compromise quality and safety of care.
Methods and analysis The electronic Delphi (eDelphi) approach will be used to engage a heterogeneous group of clinicians with patient-facing and/or information governance roles recruited through purposive and snowball sampling techniques. A semi-structured online questionnaire will be used to explore clinicians’ perspectives on relevance, definition and assessment of IQ dimensions in the CLIQ framework. Survey responses on the relevance of dimensions will be summarised using descriptive statistics to inform decisions on retention of dimensions and termination of the study, based on pre-specified rules. Analysis of the free-text responses will be used to revise definition and assessment of dimensions.
Ethics and dissemination Ethics approval has been obtained from the Imperial College Research Governance and Integrity Team (Imperial College Research Ethics Committee (ICREC) Reference number: 20IC6396). The results of the study will be published in a peer-reviewed journal and presented at scientific conferences.
- Health informatics
- Information management
- Quality in health care
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
A systematic, practical, affordable and transparent eDelphi approach will be used to engage clinicians on information quality (IQ) of digital health technologies (DHTs).
Heterogeneity of the expert panel, with panellists drawn from multiple clinical professions and countries, will enrich the findings and enhance the external validity of the clinical information quality (CLIQ) framework.
This study will simultaneously take advantage of the clinical experience and information governance expertise of clinicians.
Contextualising the CLIQ framework to the needs of the clinicians will result in a pragmatic approach to assessing IQ of DHTs in clinical practice.
Validation based on expert panel approach is limited to face and content validity with further assessment required for appraising the construct validity and applicability of the CLIQ framework in clinical practice.
Introduction
Digital health Technologies (DHTs) such as electronic health records, clinical decision support systems and electronic prescribing systems are widely used in healthcare.1 While widespread adoption of DHTs can improve healthcare delivery, information quality (IQ) problems associated with DHTs can compromise quality and safety of care.2 Patient safety incidents, relating to delayed, missing, partial or wrong information and resulting in patient harm or deaths, have been reported in the literature.3–6 For example, a patient had seizures due to incorrect mapping of different formulations of an epilepsy medication in the electronic prescription system.3
Although the negative impact of poor IQ of DHTs is well documented in the literature, not much is known about how to assess the quality of clinical information from DHTs. A systematic review published in 2021 identified 10 IQ frameworks that are relevant to assessment of clinical information from DHTs.7 Although these frameworks define fundamental dimensions that describe specific aspects of information, none offered a pragmatic approach to assessing information in clinical practice. Drawing on the findings of this systematic review, the clinical information quality (CLIQ) framework (table 1) was developed to provide a pragmatic approach to assessing the quality of clinical information from DHTs. This study aims to appraise the CLIQ framework by exploring clinicians’ perspectives on the relevance, definition and assessment of IQ dimensions as defined in the framework. This will help to contextualise the CLIQ framework to the needs of the information users as recommended in IQ literature.8 9 Clinicians are the end users of clinical information from DHTs.
Clinical information quality framework for digital health
Methods and analysis
Study design
This study will use an electronic Delphi (eDelphi) approach, which is a systematic, practical, affordable and transparent method of engaging multiple stakeholders from different locations and integrating their opinions to achieve consensus.10 11 The eDelphi approach promotes equal participation and prevents dominance of the panel by outspoken participants, which often characterises physical committee meetings.12 In addition, the iterative process of the eDelphi method enables participants to reconsider their opinions based on collective responses.11
Steering Committee
This eDelphi study will be coordinated by a steering committee comprising of healthcare professionals and researchers with interest in digital health (KPF, NM, JG, PAW AM, JC). The steering committee developed the CLIQ framework,7 from which the initial items of the eDelphi study will be generated. The committee will be responsible for recruiting the panellists of the eDelphi study. In addition, the committee will make decisions regarding retention, removal or redefinition of IQ dimensions based on the inputs of the panellists according to prespecified decision and stoppage rules.
Generation of initial items
The initial survey for the eDelphi study (online supplemental appendix 1) has been generated from the infographic CLIQ framework7 and the accompanying assessment questionnaire developed based on evidence from literatures. The survey documentation and content cover the following:
Supplemental material
Brief information about the study with a link to the participant information leaflet
Request for informed consent
Collection of demographic data of participants to confirm eligibility for the study and for descriptive purposes. This includes occupation.
Likert scale questions on relevance of IQ dimensions and categories.
Multiple choice questions on definition, assessment and categories of IQ dimensions.
Free-text questions on modification of definition, assessment and categories of IQ dimensions.
Collection of email addresses of participants for feedback purposes and as a contact method for the next round of survey.
Thus, the survey questions relating to the CLIQ framework are divided into two parts. The first part will explore the relevance of the dimensions in the CLIQ framework from the perspective of the panellists. The second part will obtain their suggestions on modification to the definitions, assessment and categories of the IQ dimensions in the CLIQ framework. The relevance of the IQ dimensions will be assessed based on the panellists’ perspective on the relevance of the dimensions to quality and safety of care using a five-point Likert scale. This captures different range of options and allows to distinguish between categories that people make naturally, without a strong cognitive load (strongly relevant, somewhat relevant, neither relevant nor irrelevant, somewhat irrelevant and strongly irrelevant).
Decision rules
Although there is no standard criteria for consensus in an eDelphi study, there is a need to predefine what constitutes a consensus to enhance objectivity and reduce analysis bias.11 Most previous Delphi studies use 60% agreement or higher as threshold for consensus.13 In this study, an IQ dimension will be considered relevant and retained in the final framework when at least 70% of the panellists, in any round of the survey, choose the options of strongly relevant or somewhat relevant when rating it. On the contrary, a dimension will be considered irrelevant and removed when at least 70% of the experts, in any round of the survey, choose the options of strongly irrelevant or somewhat irrelevant when rating it. The decision on whether to retain or remove any dimension for which no consensus is reached by the end of the study will be made by the steering committee based on the data from all the rounds.11
Stoppage rule
The eDelphi rounds will be stopped when consensus has been reached as described above on the relevance of at least 80% of all the IQ dimensions. The stoppage rule will be applied from the first round if no new dimensions are suggested by the respondents or from the second round after the respondents may have scored any suggested new dimension. The eDelphi study will be terminated at the end of the third round irrespective of the level of consensus achieved.13 This alternative stoppage rule is necessary to prevent the need to continue the eDelphi rounds if consensus is not achieved within a reasonable time frame,11 which will be regarded as 6–8 months in this study.
Participant recruitment
A heterogeneous group of clinicians will be selected including doctors, nurses, pharmacists and other healthcare professionals with patient-facing and/or information governance roles. Heterogeneity of panellists will allow a wide range of perspectives and enhance external validity of the framework.10 There are no clear guidelines about the sample size of an eDelphi study.13 However, the literature suggests 8–15 participants when the sample is homogeneous with a caveat to avoid extremely large sample sizes because the amount of data could be unmanageable.10 We therefore estimated that 40 participants will be required to accommodate different categories of clinicians (doctors, nurses, pharmacists and others), but increased the sample to 50 to account for 20% drop-out during the eDelphi rounds.13 Thus, we aim to recruit up to 50 participants to accommodate various clinician groups and compensate for drop-out during the eDelphi rounds as well as ensure geographical diversity.
The following eligibility criteria will be used to nominate clinicians that will be invited for the survey:
Prior or current experience with using DHTs in patients’ care.
Information governance role or personal interest in information governance.
Proficiency in English language to understand and complete the surveys.
Willingness to participate in a multiple-round eDelphi study (up to three rounds).
We are particularly interested in clinicians with information governance roles (chief clinical information officer, chief nursing information officer, Caldicott guardian, etc) as they typically have prior or current experience with using DHTs. Thus, the study will benefit simultaneously from their subject matter expertise and practical user experience. However, we did not limit participation to this group of clinicians with information expertise alone as we are aware that these roles do not exist in many countries especially in low-income and middle-income countries. In addition, recruiting clinicians with varying levels of expertise will encourage wide range of opinions.
Study procedures
The survey will be set up using Qualtrics software (Qualtrics, Provo, Utah, USA). The functionality of the survey will be tested by the members of the steering committee prior to its administration. The study will start with purposive nomination of the panellists by the members of the steering committee. Steering committee members will be asked to nominate panellists both within and beyond their professional networks. Nomination of the panellists by the steering committee members will be based on the pre-determined eligibility criteria discussed above, subject to confirmation by another committee member who will check the profile of the nominees against the eligibility criteria. Each of the panellists will be invited by an introductory email containing a brief overview of the study and the link to the survey. The snowball sampling technique will then be used to recruit additional panellists by asking the nominated panellists to share the eDelphi invitation to other eligible participants. Questions about participants’ occupation and prior digital health experience will be included in the survey to further confirm the eligibility of the panellists. Up to two reminders will be sent at least 2 weeks apart to encourage participation by those who did not respond to the initial email.
Only items on which consensus has not been reached and any newly suggested item(s) in the previous round will be included in the next round. The survey will be terminated based on the stoppage rule earlier listed. The first round of the survey started in June 2021. The study is expected to last between 6 and 8 months.
Data analysis plan
Survey responses on the relevance of dimensions will be summarised using descriptive statistics including frequencies, percentages, ranges and medians. The descriptive statistics will be used to provide concise feedback to the participants and to inform decisions on retention of IQ dimensions and termination of the study as already described. The feedback on the statistical summary of group response will be sent in the email inviting participants for the next round of the survey.
The free-text suggestions on the modification of the definition, assessment and categories of IQ dimensions will be analysed based on the reflexive thematic analysis approach.14 This will provide an opportunity to go beyond the texts to decode the intended meaning of the suggested modifications. It is however important to highlight that the purpose of thematic analysis in this study is to provide an in-depth understanding of the contributions of the panellists with the aim of revising the definition of IQ dimensions and the approach of assessment, as appropriate. We have therefore adapted the thematic analysis process to include the following steps:
Familiarisation with the data by reading the free-text contribution of the panellists repeatedly.
Coding of the data to highlight the issues raised with regard to the definition and assessment of CLIQ dimensions.
Development of themes by identifying patterns of the suggested modifications, reflecting on them in the context of the overall dataset and defining the essence of each theme.
The themes will then be considered by the steering committee and used to revise the definition and assessment of dimensions as appropriate. The feedback on the free text suggestions and the changes that have been made will be incorporated into the subsequent round of the survey.
Data management and storage
A data impact assessment and dataset registration were completed through the Imperial College Faculty of Medicine Data Privacy Impact Assessment Tool. This was done to address potential gaps and comply with relevant legal obligations. Data will be stored securely in an access-restricted Imperial College shared drive in accordance with General Data Protection Rules,15 the Data Protection Act (2018) and the Imperial College Data Protection Policy.16 Data will be stored for a minimum of 10 years after the study completion or longer if needed for further reference.
Ethics and dissemination
Ethics approval has been obtained from the Imperial College Research Governance and Integrity Team (Imperial College Research Ethics Committee (ICREC) Reference number: 20IC6396). Detailed information about the study will be presented in a participant information sheet containing information on the study objectives, expectation of the participants, duties of the researchers and relevant contacts (online supplemental appendix 2). Informed consent will be obtained electronically from each participant at the beginning of the online survey and before the eDelphi study questions. Participants may refuse to participate or withdraw from the study without giving any reasons at any point. However, any data collected and analysed prior to participant withdrawal will be retained.
Supplemental material
Individual responses of the participants will be pseudonymised before being added to the secure drive. Feedback to each participant will only contain descriptive statistical summaries of the group responses. Each research participant will be assigned a research code, known only to the first author. Personal information, which could be used to directly identify participants such as their email addresses, will be kept confidential and known only to the first author. The results of the eDelphi study will be published in a peer-reviewed journal and presented at scientific conferences. Panellists will only be listed in the publication with their prior consent.
Patient and public involvement
Patients will not be involved directly in the design and conduct of the study as the study is aimed at DHTs used by healthcare professionals in a clinical setting. The members of the steering committee who designed and will oversee the study are mostly clinicians with research interest in digital health and the members of the expert panel will be clinicians with practical experience of using DHTs.
Discussion
This study seeks to appraise the CLIQ framework by exploring clinicians’ perspectives on the definition, relevance and assessment of IQ dimensions in the framework. The initial CLIQ framework defined IQ dimensions that are relevant to assessing DHTs, based on systematic review of literature, without obtaining inputs from information users or specifying how IQ could be assessed.7 However, this study will obtain direct inputs from clinicians, which will ensure that the dimensions in the contextualised CLIQ framework are those considered relevant by clinicians—the users of clinical information from DHTs. Inputs from the clinicians will also ensure that the questionnaire for assessing clinical information from DHTs is written in a clear and concise language that is well-understood by clinicians. The contextualised CLIQ framework from this study will comprise of two related instruments—an infographic framework and an assessment questionnaire. The infographic framework will define IQ dimensions that are relevant to assessing clinical information, thus providing a useful guide to understanding IQ requirements for DHTs. The questionnaire will offer a pragmatic approach to assessing clinical information from DHTs. The questionnaire could be used, for example, to obtain feedback about IQ of named DHTs from clinicians using them in clinical practice.
This study has several strengths and limitations. First, the eDelphi methods offers a systematic, practical, affordable and transparent approach to integrating opinions of clinicians on IQ of DHTs. Heterogeneity of the expert panel, with panellists drawn from multiple clinical professions and countries, will ensure variety of inputs and enhance the external validity of the CLIQ framework. In addition, this study will take advantage of the clinical experience and information governance expertise of participating clinicians thus combining practical user experience and subject matter expertise.
However, we acknowledge that validation based on expert panel approach is limited to face and content validity.17 We are therefore planning a pilot assessment to evaluate the construct validity of the contextualised CLIQ framework and assess its applicability in clinical practice. We acknowledge that the initial nomination of the panellists may lead to selection bias as steering committee members may tend to recruit colleagues they know personally, rather than via their wider professional networks. These colleagues may be more likely to participate than people invited through other sources. We have therefore put in place multiple measures to reduce the risk of selection bias. The snowball sampling technique will ensure that only a fraction of participants will likely be recruited directly by the steering committee members. The eDelphi approach will make it impossible for any of the panellists to dominate the decision-making process. Finally, we will compare the responses of the panellists who were recruited directly and those who were recruited by snowball techniques.
The contextualised CLIQ framework will offer a pragmatic approach to assessing clinical information from DHTs. The framework could be used in quality improvement initiatives relating to DHTs especially in health facilities. Such use may help to identify and forestall IQ problems that can compromise quality and safety of care.
Ethics statements
Patient consent for publication
Acknowledgments
We would like to thank the Peer Review Office, Imperial College London for organising the peer review of the draft of the protocol as part of the ethical approval process. We are also grateful to the reviewers undertaking this work and providing constructive feedback at that stage.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @PF_eHealth, @petrawark
Contributors KPF conceived the study and drafted the manuscript. KPF, NM, PAW, JG, AM and JC are part of the steering committee. They contributed to the development of methods, including participant recruitment, data collection and data analysis. They also revised the manuscript for important intellectual content.
Funding This study is part of the doctoral research of the lead author (KPF) sponsored by the Federal Government of Nigeria. The study is independent research supported in part by the National Institute for Health Research (NIHR) Applied Research Collaboration Northwest London (NIHR200180).
Disclaimer The views expressed in this publication are those of the authors and not necessarily those of the National Institute for Health Research or the Department of Health and Social Care.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.