Objectives Screening for symptoms of postintensive care syndrome is based on a long list of questionnaires, filled out by the intensive care unit (ICU) survivor and manually reviewed by the health professional. This is an inefficient and time-consuming process. The aim of this study was to evaluate the feasibility of a web-based triage tool and to compare the outcomes from web-based questionnaires to those from paper-based questionnaires.
Design A mixed-methods study.
Setting Nine Dutch ICU follow-up clinics.
Participants 221 ICU survivors and 14 health professionals.
Interventions A web-based triage tool was implemented by nine ICU follow-up clinics. End users, that is, health professionals were interviewed in order to evaluate the feasibility of the triage tool. ICU survivors were invited to fill out web-based questionnaires 3 months after hospital discharge.
Primary outcomes Outcomes of the questionnaires were merged with clinical data from a national quality registry to assess the differences in outcomes between paper-based and web-based questionnaires.
Results 221 ICU survivors received an invitation to fill out questionnaires, 93 (42.1%) survivors did not respond to the invitation. Respondents to the web-based questionnaires (n=54) were significantly younger and had a significantly longer ICU stay than those who preferred the paper-based questionnaires (n=74). The prevalence of mental, physical and nutritional problems was high, although comparable between the groups. Health professionals’ interviews revealed that the software was complex to use (n=8) and although emailing survivors is very convenient, not all survivors have an email address (n=7).
Conclusions Web-based screening software has major benefits compared with paper-based screening. However, implementation has shown to be rather difficult and there are important barriers to consider. Although different in age, the health status is comparable between the users of the web-based questionnaire and paper-based questionnaire.
- health informatics
- information management
- world wide web technology
- organisational development
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
A strength of this study is that we implemented the web-based triage tool in a clinical care setting instead of a clinical trial setting.
Outcomes and characteristics of patients which preferred the web-based questionnaires were compared with the outcomes and characteristics of patients which preferred the paper-based questionnaires.
By using mixed methods, we were able to verify the statements of health professionals with the clinical data and questionnaire outcomes.
Intensive care unit (ICU) survivors frequently suffer long-term and severe complaints after ICU discharge1 2 and a single term is used to identify the presence of one or more impairments after critical illness: Postintensive care syndrome (PICS).3
Because of the complexity and magnitude of the complaints, multidisciplinary care after ICU discharge is required.4 ICU follow-up care aims to detect PICS in an early stage and the ICU survivors will be referred to the appropriate health professional(s) during consultation. In some ICU guidelines, it is recommended to have an ICU follow-up clinic.5
Generally, screening for symptoms of PICS is based on a long list of paper-based questionnaires, filled out manually by the survivor and reviewed by the health professional before or during consultation. This is an inefficient and time-consuming process. Moreover, there is a high rate of non-responders due to the age and medical conditions of survivors and because survivors cannot always be traced on their home address.6 7
We created a web-based triage tool to collect patient-reported screening data. The tool supports automatic processing of the data before presenting it to the health professional. Web-based screening has major benefits compared with paper-based screening, for example, more complete data, less entry errors and easy storage of data,8 leading to enhanced integrity and accuracy of outcome data.9 In previous literature, the benefits of web-based screening software have been pointed out in clinical trial settings.10 However, research on the implementation of software in clinical care and the use of web-based screening in ICU survivors and ICU personnel is scarce.
The aim of this study was to evaluate the feasibility of our web-based triage tool in the ICU follow-up clinic and to assess the outcomes gained by web-based questionnaires compared with those from conventional paper-based questionnaires.
Materials and methods
Based on the recommendations of Van der Schaaf et al,11 (box 1), a new web-based triage tool was created and tested during a pilot study. The tool supports automatic collection and processing of data for ICU follow-up care. The study was conducted between 1 June 2014 and 30 June 2015. All ICUs participating in the Dutch National Intensive Care Evaluation (NICE) registry that had an ICU follow-up clinic were invited to participate in this pilot study. The NICE registry is a quality registry which contains demographic data, physiological data and clinical data for all ICU patients in the Netherlands.12 13 We aimed to include 10 ICU’s in this pilot study.
Recommendations for eligibility of intensive care unit (ICU) survivors for ICU follow-up clinics11
Invite all survivors who received >48 hours mechanical ventilation.
Invite the partners of survivors.
Plan the first visit to the ICU follow-up clinic 12 weeks after hospital discharge with the possibility for a follow-up at indication.
Screen survivors with respect to their needs and ICU-related sequelae.
Use electronic patient-reported screening instruments to identify survivors in the need for ICU follow-up care.
Have an ICU nurse, whether or not with an intensivist, carrying out the ICU follow-up clinic.
Involve a physiotherapist to perform a comprehensive physical screening.
Integrate follow-up care data into a national quality registry for ICU to monitor and improve quality of life and functional status of survivors.
Web-based triage tool
The triage tool includes a module for health professionals to be used in the follow-up clinic and a web-based questionnaire module for ICU survivors.
During the development of the triage tool, both modules were tested for usability. The module for health professionals was evaluated with four health professionals by means of semistructured interviews.14 The usability of the web-based questionnaire module was evaluated with four ICU survivors using the think aloud method.14 15 Outcomes of the semistructured interviews and the think aloud sessions resulted in minor adjustments of the triage tool prior to implementation of the triage tool in this pilot study.14
The triage tool automatically extracted data of eligible survivors from the hospital information system (HIS). Nine weeks after hospital discharge, the health professionals received a prompt to send the survivor an invitation by email to fill out a set of online questionnaires and to invite the survivor to visit the ICU follow-up clinic 3 months after hospital discharge. If there was no response from the survivor within the next week, the health professional received a prompt to call the survivor. During this phone call, the health professional would ask for the reason of the non-response and explain the importance of screening for PICS and a visit to the ICU follow-up clinic. If survivors stated that they were unable to fill out the online questionnaires, paper-based questionnaires were issued. The paper-based questionnaires were entered in the system manually by the health professional or the secretary.
The pilot study included the questionnaires described in table 1. Besides these validated questionnaires, work and income-related questions, common problems after ICU admission and visits to health professionals after ICU admission were queried (online supplementary appendix 1).
The results of the questionnaires were automatically processed by the triage tool and compared with the cut-off points. During the follow-up consultation, the health professional and the survivor discussed the outcomes of the questionnaires and the survivor was referred to a specialist if necessary. This was similar to the process before the implementation of the triage tool except for the fact that the outcomes of the questionnaires were calculated and present before the start of the consultation.
Health professionals were trained to use the software before the start of the study. The 3-hour training was given by the developers of the tool and a researcher (IvB or FB-R). During the pilot study, the health professionals were contacted regularly and offered assistance when necessary.
Evaluation of the feasibility of the triage tool
After finishing the pilot study, semistructured interviews were conducted with health professionals who used the tool, to gain insight in the feasibility of the triage tool. The semistructured interviews were hold from July 2015 until September 2015 and conducted by one researcher (IvB). All health professionals were interviewed in their own working environment and an informed consent was verbally issued and recorded before the interview started.
All interviews were recorded digitally and transcribed verbatim. The thematic content analysis method was used to analyse the qualitative data.16 All interviews were coded individually by two researches (IvB and FB-R). Both researchers extracted the statements from the transcripts and grouped the statements by themes. The themes and statements were discussed until 100% agreement was achieved on the coding.
The statements of the health professionals were compared with the characteristics of the survivors and the outcomes of the questionnaires in order to relate the qualitative data to the quantitative data.
Finally, the health professionals were requested to fill out the System Usability Scale (SUS).17 The SUS is a tool to evaluate software tools. Scores range from 0 to 100 and a SUS score above 68 is indicating an above average usability.17
Questionnaire outcomes of the ICU survivors
The outcomes of the questionnaires were used to evaluate the type and severity of symptoms of PICS present in survivors. The anonymised data of the questionnaires were linked with clinical data from the NICE registry to gain insight in the demographics and clinical differences between survivors who filled out the web-based questionnaires compared with those who filled out the paper-based questionnaires. Data linking was based on a unique identifying number available in both databases.
Categorical data were presented as numbers and percentages, continuous data as medians and IQR. Differences between the web-based questionnaire group and the paper-based questionnaire group for non-normally distributed data were calculated using the Mann-Whitney U test. Differences between the two groups for normally distributed data were calculated using the t-test. For categorical data, the χ2 test was used to assess the differences between the study groups. All analyses were performed using IBM SPSS Statistics V.24.18
Patient and public involvement
No patients were directly involved in the development of the research question, design of the study or interpretation of the results. However, the usability of the web-based questionnaire module was evaluated with ICU survivors. Outcomes of the evaluation resulted in minor adjustments of the module prior to the implementation of the triage tool in this pilot study.
Of the 23 Dutch ICUs with an ICU follow-up clinic, nine ICUs (39.1%) participated in the pilot study. One ICU withdrew due to reorganisation 8 months after the start of the study. Of the eight participating ICUs, one (12.5%) was located in a university hospital, one (12.5%) in a teaching hospital and six (75.0%) in community hospitals.
Evaluation of the feasibility of the triage tool
During this pilot study, 531 survivors were eligible for follow-up care and were extracted from the HIS. Before sending out the invitations, the health professional would check if the survivor was still alive and 42 (7.9%) survivors were reported as ‘deceased after hospital discharge’. Of the remaining survivors, 221 (45.2%) received an invitation to fill out the questionnaires and to attend follow-up care. Other reasons for not inviting the survivor, besides death, were not collected. There were no significant differences in characteristics between survivors who were invited or not.
Ninety-three (42.1%) survivors did not respond to the invitation. Of the non-responders, 28 (30.1%) were phoned by the health professional to ask for the reason for non-response; three (10.7%) could not be reached on their phone number, 8 (28.6%) said they were well and did not need follow-up care, three (10.7%) said they were unable to fill out questionnaires and to attend follow-up care due to their poor health status, two (7.1%) had no recollection of the ICU admission, six (21.4%) were already involved in a rehabilitation programme, one (3.6%) had no computer and five (17.9%) gave other reasons. It is unknown whether the other 65 non-responders were not contacted or that the phone calls were not registered. There were no significant differences in characteristics between non-responders and responders.
Fourteen health professionals worked with the system and were interviewed; five intensivists, six ICU nurses, one physical therapist and two medical secretaries. The duration of the interviews ranged from 21 to 39 min. Ten health professionals filled out the SUS with an average score of 56.
Table 2 shows the main barriers to using the tool for survivors, according to the health professionals. The email addresses of survivors or family members were not always routinely collected before the start of the study. During the study, this was implemented in the regular workflow in the HIS.
Health professionals were surprised to find out that a large part of survivors mentioned not to have an email address, even the ‘younger’ survivors of 40–50 years old. Over 70% of the health professionals said that the ICU population in general is older, and that survivors are not ready to use web-based questionnaires because of their age, that survivors were too sick to fill out the questionnaires or that survivors did not want to be confronted with the ICU admission.
According to the health professionals, if follow-up care is offered on a voluntary basis, some survivors will reject it (28.6%). Lack of interest, avoidance as part of post-traumatic stress disorder (PTSD), distance to hospital, burden to ask caregivers for support are frequently stated reasons by the health professionals for survivors to reject ICU follow-up care. Most health professionals (85.7%) would like to see follow-up care as part of the routine care, only few health professionals think of the follow-up care as an extra service to the survivor.
Questionnaire outcomes of the ICU survivors
In total, 54 survivors filled out the web-based questionnaires and 74 survivors used the paper-based version. Eighty-seven survivors attended ICU follow-up care. Table 3 gives an overview of characteristics of survivors, grouped by paper-based or web-based data collection. Survivors who preferred web-based questionnaires were significantly younger compared with survivors who filled out the paper-based questionnaires (p<0.05) and had a longer ICU stay (p<0.05). Survivors who filled out the web-based questionnaires had a significant higher prevalence of PTSD, measured with the Trauma Screening Questionnaire. For all other patient-reported outcomes, there were no significant differences between survivors who filled out the web-based questionnaires as opposed to survivors who filled out paper-based questionnaires.
In the paper-based group, less questionnaire outcomes could be calculated due to missing items, that is, in the paper-based group 13.2% of the results were missing, in the web-based questionnaire group this was 2.8%.
Within both questionnaire groups, there was a large prevalence of possible mental problems, physical problems and nutritional problems (table 3 and figure 1). Not all survivors with possible problems had contact with the appropriate health professionals during the time of filling out the questionnaires.
We implemented a web-based triage tool to evaluate its feasibility and to assess the outcomes of web-based questionnaires compared with paper-based questionnaires. In previous literature, the benefits of web-based screening software have been pointed out in clinical trial settings.10 However, our study showed that the implementation in daily practice might be difficult and we identified important barriers to consider. Survivors who responded to the web-based questionnaires were significantly younger and had a significantly longer ICU stay than those who preferred the paper-based questionnaires. Health status at the time of filing out the questionnaire did not differ between the two groups. Strikingly, the prevalence of mental, physical and nutritional problems was equally high in both groups and the majority did not receive care for these complaints before they visited the ICU follow-up clinic.
Though the tool was evaluated and adjusted before implementation, eight (57.1%) health professionals found the software too complex to use. The average SUS score was 56, indicating a less than average usability and necessitating improvement of the software. A point of interest is the time between the training and the start of the pilot study. Not all ICU follow-up clinics started the pilot study at the same time, while the training was given on three dates during two consecutive weeks. Moreover, during the evaluation the pilot study, five health professionals mentioned that they used the software so little, they forgot how to send an email with it. For future research, it is advised to plan the training shortly before the start of the study and to use the new software on a regular basis.
Over 40% of the respondents filled out the web-based questionnaires. Health professionals stated that many survivors did not have an email address and expressed that survivors in general are not ready yet to use the web-based triage tool because of their age. This was not in line with the results of the telephone calls where only one (3.6%) survivor stated that they did not had an email address. Moreover, as our society is focussing and relying more and more on digital systems, survivors not having an email address will be no barrier in the future. Already in 2013, 95% of all Dutch households had access to a computer with an internet connection.19
Digitally issued questionnaires have major benefits compared with paper-based questionnaires, such as more complete data, less entry errors and easy storage of data.8 Our study confirmed this finding as we found that in the paper-based questionnaire group, there was more information missing. A possible explanation can be the use of checks and prompts in the web-based questionnaires when items were not filled out. Another major benefit is that by using web-based screening software, survivors with possible health problems can be identified without visiting the hospital. The outcomes of the questionnaires can be used in clinical decision-making and tailored care. This will improve the effectiveness of the treatments.
The prevalence of possible mental, physical and nutritional problems was high among the respondents. However, not all survivors received the appropriate care after hospital discharge. Even though there is no consensus on the (cost-) effectiveness of intensive care follow-up programmes,20–22 we believe that our triage tool is a step in the right direction. Follow-up care should be offered as stepped care, so it can be tailored to the needs of survivors. The triage tool makes it possible to highlight the problem areas so they can be addressed during consultation. Furthermore, the triage tool can be used to reach large groups of survivors as the data collection and processing is less labour intensive.
People choosing to fill out questionnaires online were significantly younger compared with those preferring paper-based questionnaires.23 According to previous published studies, younger age has been found to be a risk factor for PTSD24 and a prolonged hospital stay is associated with lower mental or physical quality of life.25 In our study, survivors who used the web-based questionnaires were also younger compared with survivors in the paper-based group and had a longer ICU stay. This can be a possible explanation for the finding that survivors in the web-based questionnaire group had a significantly higher risk of developing PTSD compared with survivors in the paper-based group.
A strength of this study was the use of mixed methods, that is, qualitative and quantitative methods. By using mixed methods, we were able to verify the statements of health professionals with the clinical data and questionnaire outcomes of survivors. For example, health professionals stated that a large part of survivors did not have an email address and that survivors were sometimes not able to fill out questionnaires due to their health status. However, these concerns were not validated with the phone calls. A possible explanation can be that survivors that could not been reached had the worst health status.26
Though 531 survivors were eligible for follow-up care, eventually only 128 survivors responded to the questionnaires. This is first due to the fact that only 221 of the 531 eligible survivors received an invitation to fill in the questionnaire and visit the follow-up clinic. A limitation of this study is that we have little information on why certain survivors were, and others were not, invited. During the interviews, the health professionals mentioned the absence of financial support from the department as a major problem. Some health professionals provide follow-up care in their own time, this makes it difficult to offer ICU follow-up care customarily. Second, of the 221 ICU survivors invited to fill out the questionnaires, 93 did not respond resulting in a response rate of 57.9%. A review conducted on the quality of life after ICU admission described that three (6%) of their included studies had a response rate of <50% and 24 studies (45%) had a response rate between 50% and 79%.2 In sight of this review, we consider the response rate of our study average.
During the interviews, all health professionals repeatedly stressed the importance of follow-up care for survivors, to address the burden these survivors suffer after their ICU admission. They all endorse the necessity and the benefits of ICU follow-up care, however, these ideas are not yet supported by scientific research. Filling out web-based questionnaires will have added value due to digitalising society. Questionnaire outcomes are present during consultation and can be discussed with survivors and their families. The results of these web-based questionnaires can be used to gain insight in the efficiency of the ICU follow-up care, if stored in a national database with options to benchmark the long-term outcomes of survivors.
Web-based screening software has major benefits compared with paper-based screening, however, the implementation has shown to be difficult and there are important barriers to consider. In order to successfully implement a new web-based triage tool, health professionals need time and support to use it. The email addresses should be queried at hospital admission so that it will not be necessary to collect the email address after hospital discharge. In both web-based and paper-based population, there was a large prevalence of survivors with possible mental, physical and nutritional problems and we suggest ICU follow-up care in order to address these problems. We think that our software is a starting point of making ICU follow-up care feasible and effective.
We would like to thank all ICU survivors and health professionals of the participating hospitals who were willing to participate in this study. We would like to thank ItéMedical for all services provided with respect to the development of the web-based triage tool and the technical support during the pilot study.
Contributors IvB gave the training to use the triage tool, conducted all semistructured interviews with the health professionals, transcribed the interviews verbatim, coded the interviews and drafted the manuscript. FB-R gave the training to use the triage tool, coded the interviews and helped to draft the manuscript. NFdK participated in the design and coordination of this study. DAD helped in analysing and interpreting the results. MvdS participated in the design and coordination of this study. All authors discussed the themes and statements of the coding, read and approved the final manuscript.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent Not required.
Ethics approval The NICE registry is registered according to the Dutch Personal Data Protection Act. The need for ethical approval for this study was waived by the Medical Ethics Committee of the Academic Medical Center and stored under number W17-354.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement We obtained permission from the research board of the NICE registry and the participants to use data at the time of the study. The NICE board assesses each application to use the data on the feasibility of the analysis and whether or not the confidentiality of patients and ICUs will be protected. To protect confidentiality, raw data from ICUs are never provided to third parties. For the analyses described in this paper, we used an anonymised dataset. The use of anonymised data does not require informed consent in the Netherlands.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.