Objectives The rapid influx of patients with COVID-19 to intensive care at a rate that exceeds pre-existing staff capacity has required the rapid development of innovative redeployment and training strategies, which considered patient care and infection control. The aim of this study was to provide a detailed understanding of redeployment and training during the first year of the COVID-19 pandemic by capturing and considering the merit of the strategies enlisted and the experiences and needs of redeployed healthcare workers (HCWs).
Design The review involved a systematic search of key terms related to intensive care AND training AND redeployment AND healthcare workers within nine databases (Medline, CINAHL, PsychINFO, MedRxiv, Web of Science, The Health Management Consortium database, Social Science Research Network, OpenGrey and TRIP), which took place on 16 July 2021. Analysis consisted of a synthesis of quantitative study outputs and framework-based thematic analysis of qualitative study outputs and grey literature. These results were then combined applying an interpretative synthesis. We followed Preferred Reporting Items for Systematic Reviews and Meta-Analyses, and the review protocol was available online.
Results Forty papers were analysed. These took place primarily in the UK (n=15, 37.5%) and USA (n=17, 42.5%). Themes presented in the results are redeployment: implementation strategies and learning; redeployed HCWs’ experience and strategies to address their needs; redeployed HCWs’ learning needs; training formats offered and training evaluations; and future redeployment and training delivery. Based on this, key principles for successful redeployment and training were proposed.
Conclusions The COVID-19 pandemic presents unique challenges to develop flexible redeployment strategies and deliver training promptly while following infection control recommendations. This review synthesises original approaches to tackle these challenges, which are relevant to inform the development of targeted and adaptative training and redeployment plans considering the needs of HCWs.
- health services administration & management
- infection control
- intensive & critical care
- medical education & training
- organisation of health services
Data availability statement
Data are available on reasonable request. All data relevant to the study are included in the article or uploaded as supplementary information.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
- health services administration & management
- infection control
- intensive & critical care
- medical education & training
- organisation of health services
Strengths and limitations of this study
This is the first review to capture the redeployment and training strategies enlisted during the first year of the COVID-19 pandemic and the needs of redeployed healthcare workers (HCWs).
The study took the form of a systematic review that synthesised 40 papers identified across nine databases.
Papers included research studies (n=21), commentaries (n=12) and reports or guidelines (n=7), which were predominantly from the UK (n=15) and USA (n=17) (limiting the exploration of redeployment in low-income and middle-income countries).
Themes were identified through conducting a synthesis on quantitative study outputs and thematic synthesis on qualitative study outputs, followed by combining the analyses through interpretive synthesis.
Searches were limited to nine databases and restricted from December 2019 to 16 July 2021. The review will have missed literature published outside of these dates and not included in these databases.
To accommodate for the rapid fluctuations in the number of patients with COVID-19, healthcare organisations have been forced to optimise resource and staff allocation procedures. The unprecedented increase in the demand for intensive care services due to the COVID-19 pandemic involved the rapid redeployment of healthcare workers (HCWs) to these units. This posed multiple challenges, including devising new ways of working and rapid development and delivery of training. This review aimed to document key aspects of redeployment implementation, HCWs’ experiences and training.
Redeployment of HCWs from other specialties to intensive care can be used to achieve the sustainable delivery of patient care.1 2 Redeployment can be used to facilitate the daily work of intensive care units (ICU) when implementing task-based models, where key tasks of patient care (ie, hygiene) are carried out promptly by competent redeployed HCWs.3 Staff redeployment can also help to address staffing gaps caused by staff sickness and previous vacancies.
Building HCWs’ competence and confidence is an essential principle of safe redeployment.4 Induction training must be provided to reacquaint redeployed HCWs with ICU ward multidisciplinary team practice, introduce them to clinical practices to care for COVID-19 patients and adequate use of personal protective equipment (PPE).5 Time constraints and infection control measures pose unprecedented logistical challenges for training delivery, with traditional training methods such as those imparted in classrooms or at conferences not being possible. The impact of redeployment on HCW well-being has also been identified.6 7 Redeployed HCWs have expressed concern in relation to their safety, the impact of their work on family members (including infecting them with the virus) and their own training and career progression.6 8
A range of novel strategies to implement redeployment and share knowledge in the context of the pandemic have been proposed and trialled.9 Gaining a detailed understanding of what worked and the unmet needs of HCWs will facilitate the development of redeployment plans for this pandemic as well as future ones.
The aim of this review was to provide a detailed understanding of the characteristics of redeployment to ICU and training provision during the first year and a half of the COVID-19 pandemic. It sought to identify what worked in redeployment and training, and concerns regarding future redeployment planning.
The research questions guiding the review were:
What were the main strategies developed to redeploy HCWs to ICU?
What were the principles of redeployment?
What were redeployed HCWs’ experiences and perceived training needs?
Were these needs addressed? If so, how? If not, why not?
What were the areas of good practice identified for both redeployment and training?
This review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement,10 and a protocol was developed a priori. The protocol was published on the authors’ institutional website (see online supplemental material 1) as it was not eligible for publication on PROSPERO.
Studies and commentaries published in peer-reviewed journals or official reports were included in this review if they focused on training or redeployment to ICUs and related wards during COVID-19. The publication dates were restricted from December 2019 to 16 July 2021. There were no restrictions on language.
Articles were excluded if the focus was on redeployment to other areas of care, other viral infection emergencies or changes in healthcare activities such as shifting to remote working.
Search strategy and study selection
Nine electronic databases were searched in December 2020 and again in July 2021 (including peer-reviewed and grey literature): Medline, CINAHL, PsychINFO, MedRxiv, Web of Science, The Health Management Consortium database, Social Science Research Network, OpenGrey and TRIP. The search strategy consisted of key terms referring to intensive care AND training AND redeployment AND healthcare workers. The search strategy was simplified when necessary for grey literature databases. A complete search strategy is provided in online supplemental material 2.
Search results were imported into Rayyan11 and deduplicated. Title and abstract screening was conducted independently by two researchers (CV and NVSJ for the first search, SEC and GC for the second search), and discrepancies were resolved via discussion until consensus was reached. Full texts of articles deemed relevant for inclusion were then screened against full review eligibility criteria. The references of included full-text articles were reviewed to identify additional articles. In the event that non-English papers were identified, members of the broader team who spoke the language were asked to review these.
Data extraction and risk of bias assessment
NVSJ and SEC extracted the information of the articles using a data extraction form developed on Qualtrics based on a predefined list of data (see box 1). The sections of the list relevant to redeployment and training were created after a preliminary scan of the selected articles.
List of data extracted from articles
Paper information: full citation; type of entry; and location.
Study design: data collection and analysis methods.
Redeployment: factors that determine redeployment; redeployment objectives; implementation strategies; redeployed HCWs’ experience; redeployment learnings; and redeployed HCWs’ needs.
Training: training programme offered; programme evaluation; pre/post results; and what worked.
Study details such as setting, population and methodological characteristics were collected from all articles. The core findings collected from the studies included details of redeployment experiences and implementation strategies (research question 1) and training programmes offered (research question 2). Special attention was paid to extract information about lessons learnt and concerns for the future.
We expected a heterogeneous group of studies using different questions and outcomes; therefore, the Authority, Accuracy, Coverage, Objectivity, Date, Significance checklist12 was applied during data extraction to assess the veracity of the source, clarity of the methods, acknowledgement of bias and the relevance of the contribution to the field.
Data synthesis method
We conducted a narrative synthesis of the study characteristics and quantitative study outputs13 and a framework-based thematic analysis of qualitative study outputs and grey literature.14 15 Quantitative and qualitative results were combined using an interpretative synthesis to develop an understanding of how they related and answer the research questions.16 17 The interpretive synthesis was based on identifying emerging patterns, or lines of argument, across studies by repeatedly comparing them, and integrating these lines of argument interpretatively to provide a bigger picture.17
Patient and public involvement
No ICU patients were involved in the development of this study; however, all authors had first-hand experience of being infected or having close relatives and friends who were infected with COVID-19. The development of the research question and interpretation of the findings was informed by MC, JPJ and AM, who worked as clinicians and training providers during this pandemic. An infographic was developed to facilitate the dissemination of findings outside of academic settings.
The screening and selection process is presented below in figure 1 according to the PRISMA guidelines.
At full-text screening stage, 41 articles were excluded as these did not include information relevant to the research questions; were not specific to redeployed HCWs; or were abstract pieces from conferences.
From the 40 papers included in this systematic review, 21 (52.5%) were research studies; 12 (30%) were opinion pieces or commentaries; and 7 (17.5%) were reports or guidelines. Papers were primarily from the UK (n=15, 37.5%) and USA (n=17, 42.5%). Other locations included China, India, Belgium, Italy, Germany and France and two studies involving multiple countries. Fourteen (35%) studies focused on redeployment implementation and experiences of redeployed HCWs exclusively and 26 (65%) on training delivery and evaluation, and dissemination of knowledge. The professional groups that were considered included nurses, physician assistants, anaesthetists, otolaryngologists, ophthalmologists, paramedics, radiologists, urologists, neurologists, orthopaedic surgeons, pharmacy residents, junior doctors, physical therapists, occupational therapists and physiotherapists.
From the 21 research studies included in the review, sample sizes ranged from 1018 to 2527.19 One study reported 59% (n=19) of participants were male, and 41% (n=13) female,20 while another study reported that the majority of the participants were female (70.8%, n=17).21 Other participant demographic characteristics such as ethnicity were not provided.
Of the studies, 35 met 75% or more of the quality criteria assessed as these were written by recognised experts, included reference lists, targeted a clear aim and stated details such as date, location and limitations. However, five of the studies only met 50%–70% of the criteria as they did not have a clear aim and methodology or did not identify bias.
A full summary of the studies included in this review can be found in table 1.
Findings from this review are presented under the headings redeployment: implementation strategies and learnings; redeployed HCWs’ experience and strategies to address their needs; redeployed HCWs’ learning needs; training formats offered and training evaluations and distilled in a final section where we propose key principles for successful redeployment.
Table 2 provides a summary of the themes and their description.
Theme 1: redeployment: implementation strategies
Due to the shifting nature of the pandemic, areas of need were moving targets that constantly changed.22 Redeployment varied depending on the individual institution configuration, geographical and population context and stage of the pandemic.2 Strategies to redistribute HCWs to meet surge demands varied across countries and health systems. In terms of sources from which to redeploy HCWs, for example, most places had access to volunteers for internal redeployment. Some of the places with the greatest burden of COVID-19 cases recruited from outside of the health system (eg, travelling nurses).23
Clear decision-making processes were facilitated, for example, by clear definitions of ‘urgent’ and efficient channelling of information to create a clear and consistent message.24 25 For the latter, suggestions included limiting the use of email chains, using online platforms such as Dropbox or apps like Induction of Clinbee, and bespoke WhatsApp groups.2 25–29 In the UK, Regional Emergency Preparedness Response and Resilience (EPRR) teams were reported as key to assisting hospitals in the management of acute surges. Clinicians and managers were advised to define warning triggers to anticipate a surge and establish communication lines with EPRR teams.2 Healthcare settings also made efforts to enable two-way communication to allow for feedback from redeployed HCWs through webinars, briefing meetings and feedback forms.25 26 28 30 Transparency and involvement of frontline clinicians as stakeholders were important components and helped to maintain trust among HCWs.31
Redeployment planning aimed to minimise training needs and maximise the use of redeployed HCWs’ previous knowledge by placing HCWs in roles where their existing skills could be more easily transferrable.3 25 30 32 Panda et al found that health system leaders (ie, chief medical officers, chairs or division chiefs) favoured a decentralised leadership approach, where organisation-wide goals were disseminated to frontline leaders for implementation. This facilitated targeted training, as local leaders were better able to identify the training needs of each redeployee. A successful approach to redeployment was allocating redeployed HCWs to designated treatment teams,that is task-based groups made up of multidisciplinary teams with clear leadership and constant communication that aimed to complete a specific necessary step of intensive care when requested by experienced ICU HCWs.2 25 27 28 31 33–37 Another successful strategy was a tiered staffing model where critical care trained physicians or nurses oversaw non-ICU clinicians.31 This represented an important shift in ways of working and understanding collaborations between health specialists.3 There were benefits in some specialists taking over ICU roles, examples of this were otolaryngologists examining epistaxis, peritonsillar abscess and facial trauma22; experienced renal physicians, together with trainee radiologists developing line insertion teams or orthopaedists and physiotherapists assisting with proning.3 Benefits included the reduction of personnel required for procedures, reduction of aerosolisation of the virus, shorter time dedicated to procedures and minimal or no training required for them to provide assistance.
A key barrier for successful redeployment planning was not being able to measure the need of human resources, that is, identifying which specific roles were in demand and which members of the workforce were available and healthy.23
Theme 2: redeployed HCWs’ experiences and strategies to address their needs
Increasing staff buy-in was key for redeployment to work. Redeployed HCWs experienced anxiety and stress, particularly when lacking adequate support or PPE, during night shifts when fewer HCWs were available and due to last minute rota changes.19 21 24 28–30 32 37–39 HCW well-being needs that required attention included accommodation, food subsidisation, access to clean scrubs, parking, a 24-hour PPE hotline, testing and family responsibilities.24 25 30 38 40 Lim et al41,41 reported that redeployed ophthalmologists’ anxiety reduced once their redeployment role began. This was attributed to receiving support from HCWs in the redeployed area, the sufficient availability of PPE and adequate training. Some HCWs often felt that the redeployment period had encouraged them to learn and change their ways of thinking in challenging situations.21 26
Coughlan et al42,42 described solutions for stressors that junior doctors experienced due to working in unfamiliar ICU settings. The interpersonal communication required for intense multidisciplinary teamwork was facilitated by visual aids, anonymised whiteboards and the use of walkie-talkies. Worries about potential negligence proceedings resulting from working beyond their usual competencies were mitigated by emergency legislation to protect doctors.
Guidelines developed by National Health Service (NHS) England and NHS Improvement (2020a) proposed addressing redeployed HCWs’ needs by placing more experienced HCWs on nightshifts; encouraging questions; providing psychological support; accepting lower turnaround of patients; and addressing issues about limited common areas and constant revision of rota patterns. One paper argued that HCWs should be given the opportunity to opt out of redeployment and self-isolate without divulging any personal information.43
Theme 3: redeployed HCWs’ learning needs
Redeployed HCWs’ learning needs varied depending on their previous experience and redeployment role. Training needed to be targeted at the right level of difficulty depending on HCWs’ most recent work experience and focused solely on content that was relevant for redeployed roles. An example of prioritising learning objectives was teaching non-specialists to recognise worsening conditions and the need for ventilation, while specialists mastered details of the operation of ventilators.44
Across the literature, training needs considered essential to provide critical care services included learning the basics of ICU monitoring such as conducting and interpreting systematic clinical assessments; mechanical ventilation; response to acute respiratory distress syndrome (ie, intubation and cardiac arrest); and conducting lung ultrasounds, management of circulatory shock and operating advanced life support technology (eg, refs 28 34 44).
Additionally, HCWs providing care specifically to patients with COVID-19 required an introduction to diagnosis and anticipated patient needs. These included prone and positioning, maintaining vascular catheters and dialysis circuits, sedation and administering vasoactive medication and continuous positive airway pressure services.4 28 33 34 37 45–47 Redeployed HCWs were also often asked to liaise with families and required training on communicating bad news.20 A particular emphasis was made on learning needs related to PPE and infection control, especially during aerosol-generating procedures.3 30 40 44 47 Regarding the latter, studies reflected on the difficulty of ensuring HCW safety due to permanently changing and sometimes conflicting advice on the use of PPE.3 31 As a result, teams assigned aerosol-generating procedures such as intubation were preferably staffed with experienced airway experts such as anaesthetists or otorhinolaryngologists to minimise the personnel required and the aerosolisation of the virus during procedures.22
Theme 4: training formats offered and training evaluations
Most courses reported in the literature included blended approaches (theory+practice) and were collaboratively designed by a variety of clinical educators, intensive care experts and frontline HCWs.1 26 32 34–36 39 45 46 48 49 HCWs found it particularly useful when course content and practical sessions were repeated over time in consecutive sessions, rather than receiving an overwhelming amount of information in one session.18 This required allocating time to access training on a continuous basis and making sure it was kept up to date.42
Some of the training development and delivery was carried out by HCWs who were shielding or had recently retired.46 Elder clinicians or those with other risk factors for severe COVID-19 infection contributed to the pandemic response by leading simulation-based education sessions. Training formats included simulation sessions; online videos; live webinars; webcasts of conference presentations from previous years; competency checklists, podcasts and blogs to discuss recent publications; and infographics presenting main results.9 21 27 28 30 32 35 37 39 45 Online learning formats allowed HCWs to access material at their own pace and check understanding; however, there was a need for practical follow-up sessions to consolidate learning.1 18 46
In addition, courses were generally complemented by buddy systems (pairing up redeployed HCWs with more experienced ICU HCWs) or other similar set-ups to provide support for redeployed HCWs during clinical practice.3 18 32 35 37 Most training programmes were evaluated through surveys or interviews, and one article reported the use of cycles of iteration composed of daily interactive feedback sessions with tutors and candidates to enable rapid improvement.46 Another article shared the role of bedside learning coordinators, who updated HCWs on any new changes to protocols and checked their understanding of these updates.26
Innovations in the peer review process and format of presenting results helped to disseminate scientific evidence in a timely way.9 However, reaching a consistent message for best practice was a significant challenge during the pandemic and affected training development. Guidelines recommended teaching principles rather than strict procedures to allow for adaptability to the local context and as circumstances changed.2
Theme 5: future redeployment and training delivery
As the pandemic evolved, the numbers of patients with COVID-19 fluctuated rapidly and a need to return to ‘business as usual’ when possible was noted in the literature.4 24 For this reason, redeployment implementation strategies started to focus on facilitating de-escalating and escalating redeployment when necessary. Examples of practices that allowed for greater flexibility were the stepwise approach to redeployment, or only redeploying discrete teams based on skills (as described in the theme redeployment: implementation strategies and learning).3 32 43
Regarding training, a need for balance between e-learning and face-to-face learning was mentioned. While e-learning was considered to have significant advantages such as a lower cost and reduced environmental impact, many felt there was still a need for face-to-face learning. There were aspects of in-person meetings that enhanced learning and well-being such as social interaction, hands-on teaching and the opportunity to travel and visit venues.9 Lastly, Camilleri et al46 pointed out that as COVID-19 cases evolve, training focus will shift to rehabilitation.
Proposed key principles for successful redeployment and training
Four key principles for successful redeployment emerged from the findings in this review: (1) developing HCW work groups based on skills rather than specialty; (2) maximising the use of redeployed HCWs’ transferable skills to minimise training; (3) having a supportive work environment, including continuous support from more experienced HCWs; and (4) developing flexible arrangements that allowed for scaling redeployment up or down.
Inductions and sustained training were key. These should be targeted at the right level depending on HCWs’ previous experience. Central training content included the basics of ICU monitoring, response to acute respiratory distress syndrome, proning and positioning, and PPE donning and doffing. HCWs assessed blended training (online+in-person practical sessions) taught in consecutive sessions in a positive way.
The implementation of these principles depended on each individual institution’s context, facilities, equipment and the stage of the pandemic.
This review synthesised data from 40 studies to identify the core aspects of redeployment implementation, redeployed HCWs’ experiences and training during the first year and a half of the COVID-19 pandemic.
To our knowledge, this is the first review of literature about redeployment and training during this pandemic. Key strengths of this work include following the systematic approach of PRISMA guidelines and combining authors with systematic review as well as clinical expertise. Searches were conducted in a wide range of databases, including grey literature and preprint servers, which have been important sources of information during the pandemic. However, searches were restricted from December 2019 to 16 July 2021. The review will have missed literature published outside of these dates and not included in these databases. It is possible that some studies from non-English speaking countries were not identified due to all the search terms being in English. It is also important to mention that most of the included papers were from high-income countries, limiting our exploration of the experiences of low-income and middle-income countries.
The challenges faced by redeployed HCWs during the COVID-19 pandemic overlapped with previous studies assessing experiences of junior HCWs in emergency departments or HCWs redeployed to disaster and war zones.50–52 Theyyunni et al53,53 analysed reflections from medical students after their emergency medicine rotation. The most common themes involved novice anxiety around critically ill patients and intubation procedures, miscommunication with other HCWs and challenges stemming from the tension between textbook medicine and complex social situations. Challenges specific to the current pandemic were PPE use and everchanging guidelines, and the paucity of testing at the initial stages.23 54 PPE use resulted in difficulties with communication, responding to emergencies in a timely manner and increased physical burden.55 Also, frontline HCWs mentioned that not having testing available increased their anxiety about possibly infecting their families and exacerbated their guilt when being off work due to suspected COVID-19 infections that could not be confirmed.6 Other challenges included aspects outside of clinical practice that had an impact on redeployed HCWs’ well-being, such as access to breakout rooms, safety in lonely streets during stricter lockdowns and family caring responsibilities.6 56 Overall, effective healthcare leadership was found to consist of identifying and understanding the sources of fear and anxiety among individuals and designing strategies to mitigate their effects.23
The essential role of training and need for innovative approaches has also been highlighted in the literature from previous virus outbreaks.57 Multimodal, multidisciplinary and realistic simulation delivered in consecutive sessions were recommended options for successful training.49 58–60 Doulias et al61,61 summarised changes in surgical training during the COVID-19 pandemic. Key aspects of successful training overlapped with those identified for redeployment to ICU. The necessary shift to e-learning was possible due to innovative and collaborative approaches that mitigated the loss of access to other modes of learning during this time. In particular, interactive surgical simulation platforms offered a model of mentoring and continued guidance. Consultants were more available to impart simulation training and engage in discussions with trainees due to the reduced elective surgery services.
Recommendations for future redeployment plans included a shift towards flexible, innovative and adaptive workforce management approaches. In the UK, the initial focus of the NHS response to COVID-19 was establishing critical care capacity; this is now shifting towards developing pathways to support people to continue their rehabilitation and assessment in community settings.4 62 This will require that HCWs receive training on regular remote monitoring, communication with patients and families and remote end-of-life care.
Tackling ongoing challenges for healthcare provision in the current pandemic will require intense collaboration from multidisciplinary teams to build organisational resilience and optimise resources through successful execution of redeployment and training. Literature about healthcare provision in disaster contexts and war zones is the closest example of rapid redeployment to emergency care and can be a source of useful recommendations. However, the COVID-19 pandemic presents unique challenges and has resulted in original and innovative approaches, which have been analysed in this review.
The key principles of redeployment distilled from our findings can inform future practice for redeployment and training in ICU settings within the evolving context of COVID-19 and future pandemics. Furthermore, similar strategies could be implemented to mitigate the negative effects of the workforce shortage crisis that affects many countries around the world.
Data availability statement
Data are available on reasonable request. All data relevant to the study are included in the article or uploaded as supplementary information.
Patient consent for publication
This study does not involve human participants.
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Twitter @NorhaVera, @CeciliaVindrola
Contributors NVSJ has acted as the guarantor of the study, led the conception and design, acquisition of data, analysis, interpretation, drafting and revision of the manuscript. CV-P contributed to the conception and design, acquisition of data, analysis, interpretation and drafting the manuscript. GC and SEC contributed to the acquisition of data, analysis, interpretation and drafting the manuscript. MC, JPJ and AM contributed to the conception and design, interpretation and drafting the manuscript.
Funding The study was funded by Health Education England (HEE), through 33N. Study funders contributed to the design, interpretation of the data and the writing of the report. Researchers conducted data collection and initial analysis independently from funders.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.