Objectives The clinical distinction between vegetative state/unresponsive wakefulness syndrome (UWS) and minimally conscious state (MCS) is a key step to elaborate a prognosis and formulate an appropriate medical plan for any patient suffering from disorders of consciousness (DoC). However, this assessment is often challenging and may require specialised expertise. In this study, we hypothesised that pooling subjective reports of the level of consciousness of a given patient across several nursing staff members can be used to clinically detect MCS.
Setting and participants Patients referred to consciousness assessment were prospectively screened. MCS (target condition) was defined according to the best Coma Recovery Scale-Revised score (CRS-R) obtained from expert physicians (reference standard). ‘DoC-feeling’ score was defined as the median of individual subjective reports pooled from multiple staff members during a week of hospitalisation (index test). Individual ratings were collected at the end of each shift using a 100 mm Visual Analogue Scale, blinded from the reference standard. Diagnostic accuracy was evaluated using area under the receiver operating characteristic curve (AUC), sensitivity and specificity metrics.
Results 692 ratings performed by 83 nursing staff members were collected from 47 patients. Twenty patients were diagnosed with UWS and 27 with MCS. DoC-feeling scores obtained by pooling all individual ratings obtained for a given patient were significantly greater in patients with MCS than with UWS (59.2 mm (IQR: 27.3–77.3) vs 7.2 mm (IQR: 2.4–11.4); p<0.001) yielding an AUC of 0.92 (95% CI 0.84 to 0.99).
Conclusions DoC-feeling capitalises on the expertise of nursing staff to evaluate patients’ consciousness. Together with the CRS-R as well as with brain imaging, DoC-feeling might improve diagnostic and prognostic accuracy of patients with DoC.
- disorders of consciousness
- clinical assessment
- minimally conscious state
- group decision making
- coma recovery scale - revised
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- disorders of consciousness
- clinical assessment
- minimally conscious state
- group decision making
- coma recovery scale - revised
Strengths and limitations of this study
We designed a new behavioural tool called disorders of consciousness ‘(DoC)-feeling’ to help face the clinical challenge of the detection of minimally conscious state in brain-injured patients suffering from DoC.
‘DoC-feeling’ pools the subjective reports of patient’s consciousness obtained from multiple caregivers (‘wisdom of the crowds’).
The obtained score shows a very good accuracy when compared with the gold standard (repeated expert clinical assessments using the Coma Recovery Scale-Revised (CRS-R)).
A validation in a separate cohort would help to precise its value in routine consciousness assessment.
This approach should be further compared to the CRS-R and brain-imaging techniques in detecting covert signs of consciousness.
Accurate diagnosis of the level of consciousness in a brain-damaged patient is of great importance to better predict recovery. Disorder of consciousness (DoC) taxonomy has been recently challenged1–3 but schematically includes the unresponsive wakefulness syndrome (UWS, also termed vegetative state) and the minimally conscious state (MCS). The detection of MCS has a huge prognostic impact since the functional outcome is dramatically better for patients with MCS.4–8 However, assessing consciousness in patients with DoC can be challenging and in such cases, clinicians may need dedicated clinical tools and brain-imaging techniques specifically designed to probe consciousness.9 Even when using dedicated clinical tools such as the Coma Recovery Scale-Revised (CRS-R10), a unique assessment remains associated with a high frequency of diagnostic error.11 This can be due to fluctuations of consciousness level over time. To circumvent this limitation, repeated clinical assessments have been proposed, but this can be limited by the availability of trained clinicians.12 13
In this study, we aimed at evaluating the diagnostic accuracy of pooled nursing staff estimations of the level of consciousness in patients with DoC. Through their clinical practice, nursing staff (ie, nurses and nursing assistants) accumulates extended observation time of patient’s behaviour. Interacting with patients through standardised procedures (such as nursing care, medication administration, blood sample, etc…), they spontaneously generate a subjective estimation of the level of consciousness of the patient. Pooling opinions of several individuals have been shown to outperform individual judgements in specific settings (effect known as ‘wisdom of the crowds’).14 15 In this study, we hypothesised that pooling individual nursing staff estimations of the level of consciousness can help in the detection of MCS.
All patients referred for evaluation of consciousness at the Department of Neurology of La Pitié-Salpêtrière Hospital, Paris, between February 2016 and October 2017, were screened prospectively. On hospital admission, patients’ relatives were approached to give consent for participation to the study. All patients with a UWS or MCS condition and consent were eligible.
Patient and public involvement
No patients or patients’ relatives were involved in the study design or the management of this study. Results of the study have been released as a preprint on a public repository16 and the dataset of this study is available on Dryad (https://doi.org/10.5061/dryad.1m03145).
Evaluation of consciousness
Patients were hospitalised in the neurointensive care unit (neuro-ICU) and were observed for at least 1 week during which they encompassed multiple neurological assessments and brain imagery such as high-density electroencephalogram, event-related potentials, magnetic resonance imaging and [18F]-fluorodeoxyglucose positron emission tomography. Clinical assessments consisted of repeated neurological examinations which included the CRS-R,17 performed by expert clinicians (BH, BR, FF, LN) belonging to an external expert team in patients with DoC. CRS-R scoring ranges from 0 to 23 and is based on the presence or absence of responses on a set of hierarchically ordered items testing auditory, visual, motor, oromotor, communication and arousal function. State of consciousness (ie, UWS, MCS) is determined by specific key behaviours probed during the CRS-R assessment. For instance, visual pursuit, reproducible movements to command and/or complex motor behaviour scores for MCS.17 Since consciousness level can fluctuate over time, we used the highest level of consciousness among all the CRS-R performed on a given patient as the reference standard. Following this procedure, each patient was thus labelled as being in a UWS or MCS. MCS was the target condition.
Nursing staff members (nurses and nursing assistants) taking care of a DoC patient were asked to fill in a form at the end of their shift containing a scale called ‘DoC-feeling’. DoC-feeling was designed as a 100 mm Visual Analogue Scale (VAS) aiming at quantifying the caregiver subjective reports of patient’s best consciousness level observed during the shift. We specifically asked caregivers to rate their ‘gut feeling’ about the best level of consciousness observed during the shift or the ‘présence’ (presence), using the French idiom ‘le patient est-il là?’ which is very close to the English one ‘Is there anybody home?’ (figure 1; see online supplementary material for the original VAS and its English translation). This wording reproduced the commonly used language to communicate observations relative to consciousness level of a patient among caregivers. Individual DoC-feeling ratings were collected prospectively. Caregivers were blinded to the previous caregivers’ ratings and to the reference standard (the CRS-R) and expert physicians were blinded to the index test. In order to obtain a final global metric, for each patient, all individual ratings were pooled using the median to obtain the DoC-feeling score that constituted the index test of this study.
Demographics, aetiology and delay since the acute brain injury (ABI) were collected. In addition to CRS-R and DoC-feeling ratings, we also collected complementary metrics (such as the classical distinction between wakefulness and awareness, interaction during nursing and/or painful care) using the same VAS approach as well as the best FOUR-score observed during each shift18 (online supplementary material).
Our primary objective was to evaluate the diagnostic accuracy of the index test called ‘DoC-feeling score’ to detect the target condition (MCS) as defined by the standard reference (best CRS-R).
First, to evaluate the association of individuals’ DoC-feeling ratings with the standard reference, we computed a linear mixed model (LMM) using DoC-feeling individual ratings as the dependent variable, the state of consciousness as the fixed effect explanatory variable and patients as well as raters as random effects. Normality of residuals distribution was assessed by visual inspection. LMM provides the optimal approach in order to take into account the non-independence between DoC-feeling ratings due to the repeated measurements over time at both the patient level (same patient rated by several raters) and the rater level (several ratings by rater).
We next pooled the individual ratings obtained for each patient using the median to obtain the DoC-feeling score (index test). We, thus, obtained a DoC-feeling score as well as a reference standard label (UWS or MCS) for each patient. We performed a direct comparison of the scores between the two populations using a Wilcoxon-Mann-Whitney test. In order to assess the diagnostic accuracy of DoC-feeling scores to detect MCS (target condition), we computed the area under the receiver operating characteristic (ROC) curve (AUC) and report sensitivities and specificities for several cut-offs of DoC-feeling scores. All statistical tests were two sided with a type I error rate of 5%. Categorical variables were expressed as numbers (percentage), quantitative variables as median (IQR). Analyses were performed using the R statistical software V.22.214.171.124 LMM was performed using the lme4 package.20 AUC, sensitivity and specificity with their 95% CIs were computed using 2000 stratified bootstrap replicates (AUC) and binomial test (sensitivity and specificity) respectively using the pROC package.21
The Standards for Reporting Diagnostic Accuracy was followed thoroughly.22
Seventy-two patients were eligible during the inclusion period, 23 were not included because of a lack of informed consent from a legal representative. Two patients were excluded because they had been diagnosed as conscious (‘Exit-MCS’). Forty-seven patients were included in the analysis (see figure 2).
Median age was 49 (32–62) years, 66% (n=31) were female. Main aetiologies of brain injury included anoxia (53%) and traumatic brain injury (17%). Delay between ABI and the evaluation was 134 (40–762) days (see table 1).
One hundred and forty-seven CRS-R assessments were performed, with a median of 32–4 per patient (ranging from 2 to 6). According to the best CRS-R, 27 patients (57%) were diagnosed as being in an MCS and 20 (43%) were classified as being in a UWS. Patients with MCS less frequently suffered from anoxia and had a longer delay between the ABI and the study inclusion (see table 1). No differences were found in the number of CRS-R assessments per patient or brain-imaging explorations between patients with UWS and MCS.
Six hundred and ninety-two DoC-feeling individual ratings were obtained (median of 129–19 ratings per patient). Eighty-three caregivers, 57 nurses and 26 nurses assistants (composed of 47 neuro-ICU regular staff members and 36 float staff members) participated in the study. Each nursing staff member filled a median of 41–12 evaluations. Median delay between the first and the last individual rating was 6 days.5–9 No statistical differences were found between UWS and MCS in the number of DoC-feeling ratings per patient, a number of raters per patient or in terms of number of ratings per rater (table 1).
Analysis of individuals DoC-feeling ratings
Inspection of the 692 DoC-feeling ratings’ distribution revealed higher values for patients with MCS than for UWS but with an important variability of ratings for a given patient (figure 3). The LMM analysis revealed a strong significant association between DoC-feeling individuals’ ratings and the state of consciousness (t=6.47, df=45, p<0.001).
Diagnostic accuracy of DoC-feeling scores
Overall, patients underwent 129–19 DoC-feeling individual ratings, performed by 75–10 different raters. All DoC-feeling ratings obtained for a given patient were summarised using the median to obtain the pooled metric called DoC-feeling score (index test, figure 4A). DoC-feeling scores were smaller for patients with UWS than for MCS (7.2 mm (2.4–11.4) vs 59.2 mm (27.3–77.3), respectively; p<0.001; figure 4B). ROC curve revealed excellent accuracy at detecting MCS (AUC=0.92 (95% CI 0.84 to 0.99); figure 4C) with, for instance, a sensitivity of 89% (95% CI 71% to 98%) and a specificity of 85% (95% CI 62 to 97) when using a DoC-feeling score cut-off at 16.7 mm (figure 4D). Note that this cut-off is only used to give the reader an idea about the diagnostic performances using the more intuitive sensitivity and specificity metrics (see the Discussion section). The six misclassified patients using this cut-off are described in the online supplementary material. Simulations of AUCs using a various number of ratings per patient suggested that a minimal number of 4 ratings is needed to reach an AUC of 0.9 (online supplementary material). Of note, DoC-feeling score also helped discriminate UWS patients from MCS ‘minus’ patients (patients with non-reflexive behaviours but absence signs of language at bedside)23 (see online supplementary material for additional details).
In the present study, we developed and assessed a new behavioural tool called DoC-feeling to help diagnose MCS. This score, which pools multiple subjective reports obtained among several caregivers over several days of evaluation, showed a very good accuracy to diagnose MCS.
DoC-feeling is not intended to replace the clinical examination nor the current CRS-R gold standard. However, taking advantage of valuable information collected by all caregivers involved in the care of a patient with DoC, the implementation of DoC-feeling could improve the overall diagnostic accuracy of patients with DoC. Caregivers are trained to evaluate pain and suffering in patients during all delivered procedures. These procedures constitute standardised interactions that can allow the generation of very reliable heuristic processes to assess one’s percept in terms of pain suffering and also consciousness.
Pooling opinions of several individuals have been previously shown to outperform individual judgements in specific settings. Recently, there has been a growing interest for this kind of approach (called collective intelligence or ‘wisdom of the crowds’) in the medical field, especially in diagnosis procedure (diagnosis of skin cancer, mammography screening, etc…).14 24–26
In that perspective, quantifying expertise that is not restricted to physicians might be of prime interest. Capitalising on assessments of consciousness gathered at any hour of the day and through multiple observers may also potentially increase our ability to detect signs of consciousness in these patients who usually show large fluctuations of cognitive state and arousal.12 DoC-feeling may also help to better describe and quantify these fluctuations. Additionally, it also enables to acknowledge the caregiver group expertise and to increase care team attention through a coherent and cumulative set of observational data.
The good accuracy of DoC-feeling obtained in our setting is likely to be generalisable elsewhere. First, as the distribution of CRS-R scores obtained in this cohort spanned most of the possible CRS-R scores, it is unlikely that the good accuracy of DoC-feeling results from two easily discernible patients’ clusters. Second, as all the patients included in this study, either in an acute or a chronic stage, were specifically referred to our institution for expertise, it is most likely that our cohort was actually representative of patients for whom the diagnosis is the most difficult. However, we would like to emphasise that the used cut-off in the result section might be variable across teams and across time for a given team. This is why DoC-feeling should only be used in addition and not instead of CRS-R.
Our study presents some limitations inherent to the aim of developing a pragmatic and easily implementable tool in daily clinical practice. First, as for all studies on consciousness disorders, we faced a typical situation of an imperfect gold standard. Although CRS-R is still the most widely accepted reference, the optimal number of assessments remains unknown.13 According to a recent study, using three CRS-R assessments can lead to a 17% rate of misdiagnoses.12 It is worth noting that this is exactly the reason why we developed DoC-feeling. CRS-R requires a specialised expertise that is not available everywhere and that can be extremely time-consuming, especially now that multiple assessments are recommended to take into account fluctuations of consciousness over time.13 In sharp contrast, DoC-feeling scale could be implemented in any team, is much faster and allows to gather multiple observations per day. Second, caregivers might have been influenced by other factors that would have been very difficult to control. For instance, they might have been influenced by insights from other caregivers or, in case of multiple ratings for a given rater, by their previous ratings. However, the variability of individual ratings for a given patient (that tended to increase over time, see online supplementary material) suggests that caregivers did report their own perception independently from each other and their eventual previous ratings. Moreover, interactions among small groups of people could, in fact, have had a positive effect since the aggregation of small groups’ insights have been shown to outperform the overall judgement of the whole group.27 This kind of tool might thus be less prone to individual subjective bias which is frequent during decision-making under a high degree of uncertainty such as assessment of patients with DoC.28 Staff members could also have been biased by classical predictors of consciousness recovery such as aetiology or delay from ABI or by the perception of patients’ relatives, although it is commonly acknowledged that relatives frequently lack objectivity (in both directions) in such dramatic situations.29 Finally, although the number of float staff members involved and the result of a preliminary survey assessing prior knowledge of regular nursing staff on DoC (online supplementary material) suggest together that DoC-feeling should be accurate in other settings, the monocentric design of this study requests external validation.
Despite these limitations, we think that the implementation of DoC-feeling score can significantly improve diagnostic accuracy and confidence in the diagnosis when supporting other metrics (ie, CRS-R and functional brain imaging at rest or during cognitive tasks). Moreover, even when incongruent with other metrics, DoC-feeling score could be still useful. Indeed, this could either suggest that key clinical elements have been missed by physicians while performing punctual CRS-R assessments, but it could also reveal, in case of discrepancy with all the other elements (clinical and brain imagery), a possible misperception of a patient’s consciousness level that needs to be acknowledged and considered in any further medical decision processes. This last point could be crucial in bridging the gap between the caregiver’s team and the patient’s relatives in situations of conflict.
In conclusion, we propose a new behavioural tool, called DoC-feeling, based on the ‘wisdom of the crowds’ effect (or, in our case, the ‘wisdom of the caregivers’), which can help to improve the diagnostic of MCS and thus to promote a better prognostication and decision-making in patients suffering from DoC.
We thank all the members of the Pitié-Salpétrière hospital Neuro-ICU led by SD (medical director), JB and LR-G (head nurses) who participated in this study (alphabetic order): Jérémie Abitbol, Fatiha Ait Yata Azzi, Fatoumata Bah, Francis Bolgert, Sandrine Briand, Sandra Coelho, Alexia Camuzat, Marie-Chantal Colmar, Flora Cherruault, Cecile Chordi, Véronique Cottin, Bintou Coulibaly, KC, Mélanie Dalibard, LD, Estelle Dumarey, Bouchra El Aouni, Atef El Ouarghi, Helene Espiand, Cécilia Eltebert, Fabrice Fanhan, Agnès Flament, Marie-Suzelle Fontano, Pascale Fournier, Céline Frammezelle, GG, Alexandra Grinéa, Nouara Harchaoui, Marie Harmancij, Claire Jacqueminet, Charlotte Janvier, Jamila Kebli, SL, Aurélie Lemoal, Kim Louis-Joseph, Brice Lucas, Valérie Maes, Sophie Maillard, Romain Maurel, Madely Petit, Floriane Pépin, Isabelle Picot, Eva Proneur, Manuela Roselmac, Sylviane Saintini, Mélody Seidel, Yolène Sully, Kelly Tcha, Laura Verbaux, Nicolas Weiss, Kelly Yanganju. We thank all the members of the PICNIC-Lab ’DoC-Team', led by LN and dedicated to the improvement of care of patients suffering from disorder of consciousness (alphabetic order): Athena Demertzi, Denis Engemann, FF, BH, Pauline Pérez, Federico Raimondo, BR; Johan Stender, MV and JDS. We thank Raphael Porcher for his help on statistical issues and Jan Claassen for his final review of our manuscript and finally, the two reviewers for their very constructive comments.
Patient consent for publication Not required.
BH, GG and KC contributed equally.
Contributors Study concept and design: BR, FF, GG, JB, JDS, KC, LD, LN, LR-G, SD and SL. Data collection: BH, GG, KC, LD, MV and SL. Analysis and interpretation of data: BH and BR. Drafting of the manuscript: BH, BR and LN. Critical revision of the manuscript for important intellectual content: BH, BR, JDS, LN and SD. Statistical analysis: BH, BR and LN. Study supervision: BH, BR, GG, KC and MV had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. BH, GG and KC contributed equally to this work.
Funding This work was supported by: Amicale des Anciens Internes des Hôpitaux de Paris & Syndicat des Chefs de Cliniques et Assistants des Hôpitaux de Paris (AAIHP—SCCAHP; BR), Assistance Publique—Hôpitaux de Paris (AP-HP; BR and LN), Institut National de la Santé et de la Recherche Médicale (Inserm; BH, JDS and LN), Sorbonne Université (LN), the James S. McDonnell Foundation (LN), Académie des Sciences- Lamonica Prize 2016 (LN) and Philippe Foundation (BR). The research leading to these results has received funding from the program ’Investissements d’avenir' ANR-10- IAIHU-06.
Competing interests None declared.
Ethics approval The protocol conformed to the Declaration of Helsinki, to the French regulations, and was approved by the local ethic committee (Comité de Protection des Personnes; CPP no 2013-A01385-40; Ile de France 1; Paris, France).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The dataset of this study is available on https://doi.org/10.5061/dryad.1m03145.
Collaborators Jérémie Abitbol; Fatiha Ait Yata Azzi; Fatoumata Bah; Francis Bolgert; Sandrine Briand; Sandra Coelho; Alexia Camuzat; Marie-Chantal Colmar; Flora Cherruault; Cecile Chordi; Véronique Cottin; Bintou Coulibaly; Mélanie Dalibard; Athena Demertzi; Estelle Dumarey; Bouchra El Aouni; Atef El Ouarghi; Denis Engemann; Helene Espiand; Cécilia Eltebert; Fabrice Fanhan; Agnès Flament; Marie-Suzelle Fontano; Pascale Fournier; Céline Frammezelle; Alexandra Grinéa; Nouara Harchaoui; Marie Harmancij; Claire Jacqueminet; Charlotte Janvier; Jamila Kebli; Aurélie Lemoal; Kim Louis-Joseph; Brice Lucas; Valérie Maes; Sophie Maillard; Romain Maurel; Madely Petit; Floriane Pépin; Pauline Pérez; Isabelle Picot; Eva Proneur; Federico Raimondo; Manuela Roselmac; Sylviane Saintini; Mélody Seidel; Johan Stender; Yolène Sully; Kelly Tcha; Laura Verbaux; Nicolas Weiss; Kelly Yanganju.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.