Article Text
PDF
Abstract
Objectives COVID-19 has altered standard thresholds for identifying anxiety and depression. A brief questionnaire to determine when individuals are at a tipping point for severe anxiety or depression would greatly help decisions about when to seek assessment or treatment.
Design Data were collected as part of the Frontline-COVID Study, a cross-sectional national online survey with good coverage of health and social care settings. New questionnaire items reflecting when coping was actually breaking down were compared with standard measures of severe anxiety and depression. Data were collected between 27 May and 23 July 2020.
Setting The majority of participants worked in hospitals (53%), in nursing or care homes (15%), or in other community settings (30%).
Participants Of 1194 qualifying respondents, 1038 completed the six tipping point items. Respondents included nurses, midwives, doctors, care workers, healthcare assistants, allied healthcare professionals and other non-medical staff. Over 90% were white and female.
Main outcome measures Threshold for severe anxiety according to the Generalised Anxiety Disorder Scale-7 or moderately severe depression according to the Patient Health Questionnaire-9.
Results Answering yes to one of two simple questions (‘Over the last week have you been often feeling panicky or on the point of losing control of your emotions?’, ‘Over the last week have you felt complete hopelessness about the future?’) demonstrated very high sensitivity (0.95, 95% CI 0.92 to 0.97) and negative predictive value (0.97, 95% CI 0.95 to 0.98). Answering yes to both questions yielded high specificity (0.90, 95% CI 0.87 to 0.92) and positive predictive value (0.72, 95% CI 0.67 to 0.77). Results were replicated in two random subsamples and were consistent across different genders, ethnic backgrounds, and health or social care settings.
Conclusions Answering two simple yes/no questions can provide simple and immediate guidance to assist with decisions about whether to seek further assessment or treatment.
- COVID-19
- health & safety
- mental health
Data availability statement
Anonymised data that support the findings of this study are available from TG upon reasonable request. The data have not been made publicly available due to their personal and sensitive nature.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
A large sample of health and social care staff involved in the current pandemic contributed.
Items were generated by specialist clinicians and experts by experience.
The design involved replication across two random subsamples and across different staff groups.
There was under-representation of men and of black and minority ethnic staff.
The sample was not representative of all National Health Service and social care staff.
Introduction
Documented high rates of psychological distress in healthcare workers confronted with pandemics including COVID-19 are evident in numerous countries and settings.1–3 Continuing uncertainty about clinical or workplace demands, the many risks inherent in exposure to patients with COVID-19, the possibility of moral injury, and the impact of social restrictions on healthcare workers and their families have created a situation in which a certain level of stress is unavoidable.4 5 Thus, the continuing pandemic, as common with other major incidents,6 7 has raised baseline levels of anxiety and depression in healthcare workers. As a result, the threshold at which formal psychological interventions such as cognitive–behavioural therapy are needed, normally determined by a recognised cut-off point on standardised instruments, has become unclear.
In addition to this calibration problem, most existing questionnaires address anxiety and depression separately, involve answering several items using rating scales with multiple points and are thus burdensome to complete. They are also not routinely familiar or available to everyone who might need them across health and social care settings. One solution to both problems is to develop a new kind of instrument calibrated to signal when general coping mechanisms might be on the point of breaking down. At this point, someone may be on the point of tipping into a state highly likely to earn a diagnosis of either major depression or an anxiety disorder based on a gold-standard structured clinical interview. For maximum utility, such an instrument should comprise a very small number of items that are readily understood, require no numerical calculation and provide instant guidance on the likelihood that clinical assessment is required. This study investigated whether a small number of newly formulated questionnaire items (a ‘Tipping Point Index’) could efficiently predict moderate to severe anxiety and depression in a large sample of COVID-19 frontline staff, ensuring that questions demonstrated sensitivity against the background of the ongoing pandemic. In view of the fact that the specificity of similar instruments has sometimes found to be lower in minority ethnic respondents,8 we further assessed whether the index was equally effective in men versus women, in white versus black and minority ethnic respondents, and in different National Health Service (NHS) and social care settings.
Method
Participants and procedure
Frontline health and social care workers from all nations of the UK were invited to participate in the study via a social media campaign.9 The questionnaire was administered using online survey methods, via the ‘Qualtrics’ data collection platform. Data were collected between 27 May 2020 and 23 July 2020. This represents the post-peak phase of the initial COVID-19 wave in the UK; during this period, deaths related to COVID-19 in the UK rose from 37 430 to 41 160, while reported weekly deaths fell from 2000 (29 May 2020) to 231 (24 July 2020) (see https://coronavirus.data.gov.uk/details/deaths).
In total, 2447 individuals opened the link to read the participant information sheet, 1311 consented to participate and 1205 provided data. Participants who indicated that they did not work in healthcare (n=5) were excluded. In cases where participants completed the questionnaire on more than one occasion, the first response was used and the second was excluded from analysis (n=6). This resulted in a sample of 1194 individuals.
Measures
The Frontline-COVID Survey included background demographic questions (see Billings et al,4 for further details). Participants included nurses, midwives, care workers (mostly working in care home or community settings), clinical support staff (including healthcare assistants), doctors, non-medical staff (including cleaners, porters, administrators, maintenance, security roles), allied healthcare professionals (including physiotherapists, occupational therapists and other allied roles as defined by the NHS), and other roles. Participants reported their work setting, which was operationalised as hospitals, nursing or care homes, and any other community setting.
Tipping Point Index
Potential items were devised by members of the multidisciplinary COVID-19 Trauma Response Working Group, consisting of specialist clinicians, coordinators of the psychosocial response to trauma, well-being leads at NHS Trusts and people with lived experience of psychological trauma (www.traumagroup.org). An initial group of nine binary yes/no items designed to indicate when levels of stress were dangerously high and required immediate action was reduced to six following rating and discussion by the group. Survey respondents indicated whether or not over the last week they had noticed any of these reactions (see table 1 for list of items).
Endorsement and predictive power of tipping point items for anxiety and depression across two random samples
Depression symptoms
These were assessed using the Patient Health Questionnaire-9 (PHQ-9),10 a widely used nine-item self-report questionnaire corresponding to the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) criteria for depression. Participants reported how often symptoms occurred during the previous fortnight on a 4-point Likert scale ranging from 0 (‘not at all’) to 3 (‘very much’). The threshold for moderately severe depression is ≥15. Recent meta-analyses have found that this threshold on average underestimates prevalence of major depression as determined by the Structured Clinical Interview semistructured clinical interview by 2%11 and is associated with a specificity of 0.96.12
Anxiety symptoms
These were assessed using the Generalised Anxiety Disorder Scale-713 (GAD-7), which is also widely used and effective in assessing for post-traumatic stress disorder, panic disorder and social anxiety disorder.14 Participants report how much they have been bothered by each symptom over the past 2 weeks on a 4-point scale from 1 (‘not at all’) to 3 (‘more than half the days’). Severe anxiety (score ≥15 maximises specificity and approximates a prevalence in line with estimates of GAD prevalence in primary care (Spitzer et al, 2006). A recent meta-analysis has found this cut-off to be associated with a specificity of 0.90.15
Data analysis
Missing data were handled by complete case analysis. A receiver operating characteristic (ROC) curve was computed on the six tipping point items to determine the optimum number of positive endorsements needed to determine whether participants met the threshold of ≥15 on the GAD-7 or on the PHQ-9. The sample was then randomly divided into two equal halves. In each half, endorsements of the six items were cross-tabulated with whether participants met this threshold. The number of items specified by the ROC analysis with the highest ORs for both anxiety and depression across both subsamples were determined and combined into an index, with the effect of different combinations of yes/no responses being explored. Sensitivity, specificity, and positive and negative predictive values of these combinations were computed in both subsamples. Subsamples were then combined for prespecified analyses of how gender, ethnic background and work setting impacted performance. Statistics were calculated using SPSS (V.24). Reporting follows Standards for Reporting Diagnostic Accuracy 2015 guidelines.16
Patient and public involvement
Research design and delivery were shaped by consultation with our Expert Reference Group which includes clinicians with frontline NHS and social care experience and representatives with lived experience of mental health difficulties. Health and social care staff were involved in disseminating information about the study to aid recruitment.
Results
The mean age of the participants was 41.54 years. Most were female (92.38%) and white (90.79%). The settings in which participants worked included hospitals (53.43%), nursing and care homes (14.82%), and other community settings (29.65%). A total of 75.63% reported that they had worked directly to treat, support or care for patients with COVID-19. Of the total sample, 1038 completed the six potential tipping point items. Frequency of endorsement is shown in table 1. Four of the items were endorsed by approximately 40% of the sample and two by approximately 10%. The proportion in the overall sample with GAD-7 scores ≥15 was 19.56% and with PHQ-9 scores ≥15 was 25.37%; 29.38% scored ≥15 on at least one of the measures.
ROC analysis on the ability of the six potential tipping point items to predict individuals scoring ≥15 on either measure found that the area under the curve was 0.88 (95% CI 0.86 to 0.90). The optimum combination of sensitivity (0.95) and specificity (0.65) was obtained when two items were endorsed. On average across the two random samples, two items had consistently high ORs when predicting both high GAD-7 and PHQ-9 scores (see table 1). These were ‘Been often feeling panicky or on the point of losing control of your emotions’ and ‘Felt complete hopelessness about the future’.
These were combined in a two-item index. Prediction of severe anxiety or depression was based on a yes to either or both of the items rather than requiring a yes to both items. The OR for this index predicting either a GAD-7 score or a PHQ-9 score ≥15 in the entire sample was 28.54 (95% CI 16.64 to 48.96), and in the random subsamples it was 19.47 (95% CI 9.92 to 38.19) and 47.13 (95% CI 18.86 to 117.76). Table 2 shows the performance of this index. Both in the random subsamples and the entire sample sensitivity and negative predictive power were extremely high. The positive predictive value was approximately 0.5.
Performance of two-item Tipping Point Index (minimum one item endorsed) for prediction of significant anxiety or depression
Also shown in table 2 are similar analyses with the two-item index, splitting the entire sample by gender, by ethnicity and by employment setting, in order to test whether its ability to detect either anxiety or depression or both was consistent across important subsamples. Sensitivity (range 1.0–0.93) and negative predictive power (range 1.0–0.96) remained very high. Specificity (range 0.49–0.79) and positive predictive power (range 0.44–0.57) were at similar levels to the entire sample, but were highest for men and lowest in care homes.
The final analysis investigated the effect of requiring a yes to both the tipping point items. The OR for this index predicting either a GAD-7 score or a PHQ-9 score ≥15 in the entire sample was 15.55 (95% CI 11.13 to 21.72). Sensitivity (0.64) and negative predictive power (0.86) were reduced, but specificity (0.90) and positive predictive power (0.72) were increased. As shown in table 3, across the different subsamples, sensitivity (range 0.60–0.72) and negative predictive power (range 0.84–0.91) were at similar levels to the entire sample. Specificity (range 0.83–0.97) and positive predictive power (range 0.62–0.90) remained high, being highest in the black and minority ethnic sample and lowest in the care homes sample.
Performance of two-item Tipping Point Index (both items endorsed) for prediction of significant anxiety or depression
Discussion
Consistent with the idea of an instrument that can efficiently capture a tipping point, two simple dichotomous questions provided valuable information about high levels of anxiety or depression that are reliably associated with a need for a formal psychological intervention. Respondents who did not endorse either question were unlikely to meet this threshold (negative predictive power in the whole sample was 0.86). Almost half the respondents who endorsed at least one question met the threshold, and the positive predictive power increased from 0.49 to 0.72 for those who endorsed both questions. In terms of sensitivity and specificity, this brief measure performed almost identically to the use of all six items.
Strengths of the research included a large sample from all parts of the UK that permitted within-study replication. Roles such as clinical support worker, care worker and healthcare assistant, and settings such as social care are relatively neglected in the literature, but are essential to include when making decisions about the health of the workforce. The study was limited by the lack of an existing, validated pool of potential tipping point items and by under-representation of men and of black and minority ethnic staff. The results are also only applicable to frontline staff dealing with COVID-19. A final limitation was the use of a self-selected sample. The decision whether or not to complete the survey once opened may have been affected by unknown biases, but there is no obvious reason why these should have affected estimates of diagnostic accuracy.
Four potential Tipping Point Index items, worded to capture the point at which stress is dangerously high and the ability to cope is near breakdown, were endorsed by around 40% of respondents. This is consistent with early research showing baseline level of stress in frontline COVID-19 workers being higher than normal, and with documented high rates of anxiety and depression.1 2 9
Non-endorsement of the two items, although suggesting that levels of anxiety or depression are unlikely to be at this severe level, should not however be a justification for withholding assessment if there are other reasons why it might be desirable. Staff members may sometimes minimise their own reactions, or there may be other symptoms such as an urge to self-harm that may take precedence.
An effective workforce is essential during an ongoing pandemic, when staff sickness may put additional demands on already overstretched health systems. The information provided by the Tipping Point Index should prove useful to healthcare workers themselves, who are often highly motivated to keep working during the pandemic and may find it hard to assess when their own levels of stress have become dangerously high. Equally, managers who are concerned with maintaining a well-functioning service need to know the point at which assessment is important to protect the mental health of an employee from severe deterioration. Individual staff and managers may differ over the point at which they feel assessment, and possible intervention, are required. At a time of greatly increased need across many NHS and social care settings, the Tipping Point Index provides a simple and accessible tool to assist decisions about seeking formal psychological interventions to protect mental health.
Data availability statement
Anonymised data that support the findings of this study are available from TG upon reasonable request. The data have not been made publicly available due to their personal and sensitive nature.
Ethics statements
Ethics approval
All procedures were approved by the University College London Research Ethics Committee (ref. 18341/001).
References
Footnotes
Twitter @docbloomfield
Contributors TG, JB and MB initiated and carried out the survey, TG and JH-S constructed and ensured the integrity of the database, CB proposed the Tipping Point Index, analysed the data and wrote the first draft. All authors contributed to and approved subsequent drafts of the manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.