Article Text

Download PDFPDF

Performance of injury severity measures in trauma research: a literature review and validation analysis of studies from low-income and middle-income countries
  1. Amber Mehmood1,
  2. Yuen W Hung1,
  3. Huan He1,2,
  4. Shahmir Ali3,4,
  5. Abdul M Bachani1
  1. 1 Johns Hopkins International Injury Research Unit, Health Systems Program, Department of International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA
  2. 2 School of Public Administration, Southwestern University of Finance and Economics, Chengdu, Sichuan, China
  3. 3 Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, Maryland, USA
  4. 4 Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA
  1. Correspondence to Dr Amber Mehmood; amehmoo2{at}jhu.edu

Abstract

Introduction Characterisation of injury severity is an important pillar of scientific research to measure and compare the outcomes. Although majority of injury severity measures were developed in high-income countries, many have been studied in low-income and middle-income countries (LMICs). We conducted this study to identify and characterise all injury severity measures, describe how widely and frequently they are used in trauma research from LMICs, and summarise the evidence on their performance based on empirical and theoretical validation​ analysis.

Methods First, a list of injury measures was identified through PubMed search. Subsequently, a systematic search of PubMed, Global Health and EMBASE was undertaken on LMIC trauma literature published from January 2006 to June 2016, in order to assess the application and performance of injury severity measures to predict in-hospital mortality. Studies that applied one or more global injury severity measure(s) on all types of injuries were included, with the exception of war injuries and isolated organ injuries.

Results Over a span of 40 years, more than 55 injury severity measures were developed. Out of 3862 non-duplicate citations, 597 studies from 54 LMICs were listed as eligible studies. Full-text review revealed 37 studies describing performance of injury severity measures for outcome prediction. Twenty-five articles from 13 LMICs assessed the validity of at least one injury severity measure for in-hospital mortality. Injury severity score was the most commonly validated measure in LMICs, with a wide range of performance (area under the receiver operating characteristic curve (AUROC) between 0.9 and 0.65). Trauma and Injury Severity Score validation studies reported AUROC between 0.80 and 0.98.

Conclusion Empirical studies from LMICs frequently use injury severity measures, however, no single injury severity measure has shown a consistent result in all settings or populations and thus warrants validation studies for the diversity of LMIC population.

  • injury severity measures
  • trauma score
  • injury severity scores
  • low- and middle-income countries
  • validation studies

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • The study comprises three parts: summary of all injury severity measures, description of their use in low-income and middle-income countries (LMICs) and their performance to predict in-hospital mortality in LMIC settings.

  • Injury severity measures, whether developed exclusively for characterising trauma and injuries, or non-injury severity measures incorporated in trauma research, are both included in this study.

  • A systematic electronic search of PubMed, Global Health and EMBASE on literature published from January 2006 to June 2016.

  • Validation studies conducted in LMICs are used to estimate the performance of injury severity measures.

  • Performance of injury severity measures to predict other outcomes such as blood transfusion requirement, intensive care unit admission or hospital length of stay are not the focus of this study.

Introduction   

Injury remains a major public health problem globally, causing significant death and disability across all the age and sex spectrum.1 A disproportionate share, 90%, of all trauma deaths occur in low-income and middle-income countries (LMICs), where resources to deal with this crisis are inadequate. An efficient and effective trauma system has been found to be a key component. It is estimated that approximately two million lives could be saved annually if LMICs could implement trauma systems comparable with trauma care systems available in high-income countries (HICs).2 However, this would require a careful assessment of the gaps and planning to ensure the most efficient use of available resources. Injury severity scoring systems can provide a foundation for benchmarking and performance improvement in the arena of trauma care.3Characterisation of injury severity is a critical pillar in the provision, and improvement of trauma care for key activities such as field triage, prognostication, prediction of risk-adjusted outcomes, quality improvement, evaluation of cost and effectiveness of trauma service delivery, planning of services and organisation of resources.4 Many injury measures have been formulated over time with a wide range of methodologies.5 While no single injury measure is considered the best or the most comprehensive, assessment of injuries in a patient has been aided by assigning numerical values to several indicators including physiological or biochemical parameters, anatomical descriptors, age and so on, and combining these values to an overall measure of injury severity.6 7 Although injury severity measures are most often used for the purpose as they were developed, such as triage or mortality prediction, it is not uncommon to validate and use them for other functions.8 9

There has been a proliferation of injury severity measures over the past few decades.7 10 While a variety of injury severity measures have been developed exclusively for trauma and injuries, other non-injury severity measures have also been incorporated in trauma research on many occasions.11–14 These severity measures use a range of clinical, biochemical, demographic and physical attributes to create indicators for prognostic predictions and performance evaluation.4 15 However, both the utilisation and validation of injury scores in clinical care or outcome research has been sparse in LMICs.16 There are multiple reasons for this but in many cases, especially for those injury severity measures developed in high-income settings, the information needs are challenging for a low-resource environment.11 15 17–19 Many well-recognised injury measures were sometimes applied without being validated in the populations under study. Subsequently, studies have documented poor performance of injury severity measures such as Trauma and Injury Severity Score (TRISS), when applied to other populations using the coefficients derived from the Major Trauma Outcome Study.20–26 However, there is a dearth in the literature on utilisation of common injury severity measures, and whether they show acceptable performance in terms of validity and reliability to support their use in LMICs. This gap limits our ability to translate high-quality injury research methods developed in HICs into effective decision support and quality improvement systems for LMICs. The aim of this study was therefore to fill this gap in the literature through a thorough review of the literature; specifically we sought to: (1) identify all the measures and scoring systems that were ever developed to measure injury severity, and summarise their characteristics; (2) describe how widely and frequently the key measures are used in LMICs and (3) summarise the evidence on their measurement performance based on empirical validation​ analysis and theoretical analysis of their applicability.

Methods

For our first aim, we conducted a literature search for terms ‘injury AND severity measures’ OR ‘injury AND scores’ OR ‘Injury AND scales’, as well ‘Trauma AND severity measures’ to include those that are not exclusive to injuries but have been used in trauma and injury research. A list of injury measures was identified through PubMed search. Subsequently, using bibliographies of the results of the primary search, a secondary search was performed to find the original literature of the injury measure development. Full text of all publications was reviewed to understand and describe the initial purpose and scope of development of the injury measure, its main components, year of first publication and country of development.

For the specific aims 2 and 3, we conducted a detailed literature review to assess the application and performance of injury severity measures to predict in-hospital mortality, conducted in LMICs. We included studies of global trauma populations and specific injury pathologies and used World Bank’s classification for LMICs in the year 2016.

Eligibility criteria

For the purpose of determining the applications of different injury severity measures in LMICs, we included studies that applied one or more global injury severity measure(s) on any type of injury population, except for studies that focused only on poisoning, drowning and ocular trauma. We excluded studies that applied exclusively organ specific injury severity measure(s), population from low income country treated in a high-income setting, as well as studies describing only combat injuries or those from military trauma registries due to the environment and contexts largely different from general LMICs settings.

Information sources and search strategy

We conducted a systematic electronic search of PubMed, Global Health and EMBASE on literature published from January 2006 to June 2016. We used combinations of search terms including medical subject heading and keywords on two groups: ‘trauma or injury measures’, and a list of ‘LMICs’ (online supplementary file 1). We applied human subjects restrictions but language restrictions were not applied. All references were exported to Endnote V.7 and duplicated studies were excluded using Endnote before exporting them into an Excel spreadsheet.

Supplementary file 1

Two authors (AM and SA) independently screened the titles and abstracts of all studies resulted from the above search strategy to identify the eligible studies for the applications of injury severity measures in LMICs. Full-text version of all the eligible articles were sought, and if full text was not available in English language, the abstracts were excluded from further analysis. All eligible full-text articles were reviewed for relevance and data collection.

Data abstraction

Data were extracted from the selected studies using a predesigned electronic data collection form. The studies were further categorised into validation studies or empirical/non-validation studies, or excluded if they did not match the inclusion criteria on full-text review (figure 1).

Figure 1

Flow diagram of search strategy and study selection according to Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines.

To assess the performance of injury severity measures and prediction of in-hospital mortality, we selected studies that estimated the Area Under the Receiver Operating Characteristic curve (AUROC) or correlation between specific injury severity measure and in-hospital mortality, based on the studies identified with applications of injury severity measures in LMICs. Studies that did not specify the outcome of assessment or did not include any estimates of AUROC, correlation or sensitivity and specificity were excluded. Three authors (AM, HH and YWH) screened these identified studies for the performance on predicting in-hospital mortality. Any disagreements were resolved by discussions among the three authors.

For the purpose of determining applications of different injury severity measures in LMICs, three authors (AM, HH and YWH) extracted information on the injury severity measures used in each study, whether performance was assessed on in-hospital mortality prediction, and the country in which the study was conducted. The studies and corresponding injury measures were assessed in detail for study population, type of injury and injury mechanism, injury severity measures, study methods, in-hospital mortality prediction and their corresponding performance in predicting in-hospital mortality. The performance of the injury severity measures is reported as AUROC and calibration as Hosmer-Lemeshow (H-L) goodness of fit test.

Patient and public involvement

This study did not involve patients or human subjects directly or indirectly, and the results of the analysis were solely based on the previously published literature.

Results

The results are described in order of specific objectives of the study. Our study demonstrates considerable growth in the science of injury severity measurement globally as well as in LMICs. Table 1 summarises the search results of different injury measures, categorised according to the primary purpose of their development and their core components. It shows clearly that the science of injury severity measures had essentially taken off in early 1970s, and it is still ongoing with similar enthusiasm. Almost 60 severity measures or scoring systems have been developed either exclusively for injury and trauma research, or have been used in measuring the severity of injuries. Many injury severity measures were developed to support epidemiological research and performance evaluation; examples include Abbreviated Injury Scale (AIS), Injury Severity Score (ISS) and New Injury Severity Scores (NISS), A Severity Categorization of Trauma and International Classification for Diseases-9 ISS (ICISS). Others, such as Revised Trauma Score (RTS); Circulation, Respiration, Abdomen, Motor and Speech (CRAMS); Acidosis, Blood loss, Cold, Damage (ABCD) and Kampala Trauma Scores (KTS) were developed to help in decision making, for example, prehospital triage and in-hospital patient disposition. A number of injury measures were developed for the purpose of outcome prediction; Trauma Mortality Prediction Model (TMPM), Rapid Emergency Medicine Score and Glasgow Coma Scale (GCS), Age, Pressure are some examples.

Table 1

List of injury severity measures, their purpose and components

Table 1 highlights that a number of empirically developed anatomical, physiological and composite measures such as AIS, or GCS, later became the basis of more complex measures such as RTS, ISS and Revised Injury Severity Classification score, and some of them (RTS, ISS, NISS) in turn became components of a more complex scoring system such TRISS, Sequential Trauma Score and so on. The use of injury measures in studies published by different LMICs is depicted in figure 2. A total of 597 studies from 54 LMICs were listed as eligible studies between 2006 and 2016 which were a combination of empirical, epidemiological, review and validation studies. China, Turkey, Iran, South Africa, Colombia and Brazil are some of the upper-middle-income countries that contributed to the majority of injury literature published in the last 10 years (figure 3), whereas India, Pakistan, Nigeria and Tanzania are some of the lower-middle-income and low-income countries that extensively used injury measures in a number of injury and trauma-related publications. Thirty-one publications described multicountry studies which may also include an HIC. Approximately 31% (n=186) of all studies were related to head or traumatic brain injuries (TBI).

Figure 2

Low-income and middle-income countries’ publications using trauma/injury severity measures: 2006–2016. 

Figure 3

Top 10 countries with trauma/injury publications.

Table 2 outlines different injury measures used in publications from 54 LMICs in injury-related research. GCS, ISS, TRISS and RTS are the most commonly used injury measures; however, some attempts have been made to develop new injury measures. Examples include Exponential Injury Severity Score (EISS), Ganga Hospital Score for lower limb fractures, Tangent Injury severity score (TISS) and some novel biomarkers such as lactate and serum acetylcholinesterase. Other scores that were not traditionally used in injury or trauma research such as McLaughlin, Modified Rankin, South African Triage Score, Modified Early Warning System and Rwanda mortality prediction model have also been used for prediction of mortality in trauma populations. Glasgow Outcome Scale is widely used in documenting the outcomes of TBI, and Functional Independence Measure was used in some studies focusing on functional outcomes of injured patients. Some attempts have been made to modify existing injury measures; for example, in Simplified RTS, Glasgow Coma Scale was replaced by five graded levels of consciousness, or NISS was used instead of traditional ISS in TRISS method.

Table 2

Injury measures used in last 10 years’ published literature from LMICs

Full-text review of eligible articles was conducted to understand the validity of these new or existing injury measures and revealed that 37 studies examined the performance of injury severity measures for the prediction of hospital length of stay, in-hospital mortality and functional outcome of injured patients. Online supplementary file 2 details 25 of 37 validations studies, as the remaining 12 use different outcomes (eg, respiratory failure, intensive care unit (ICU) admission, etc) or use a different algorithm. These 25 articles from 13 LMICs assessed the validity of at least one injury severity measure in hospital settings. ISS was the most commonly validated measure in LMICs in the past 10 years, assessed in 11 studies. TRISS was the second most commonly validated injury severity measure in LMICs, followed by GCS, APACHE II and NISS. GCS was more commonly assessed among head/TBI, while also validated among patients with general injuries. The majority of validation studies included all injury mechanisms, some studies included critically ill populations such as ICU patients, while others included patients admitted to the emergency room. The proportion of mortality also varied widely among different settings, ranging from 0.6% to 40%.

Supplementary file 2

Among injury severity measures that were validated in multiple contexts, many presented a wide range of AUROC estimates. Out of the 11 validation studies on ISS, 5 estimated AUROC above 0.90, and 2 of the studies had AUROC below 0.70 with 95% CI overlapping 0.65. Similarly, as majority of the validation studies on TRISS reported AUROC between 0.80 and 0.98, three studies reported 95% CI of AUROC overlapping 0.70. More than a third of the validation studies did not present 95% CI estimates of AUROC, and more than half of the validation studies did not provide estimates on calibration (15 studies).

A majority of the validation studies included only adults and sometimes adolescents. A third of the validation studies included both adults and children, and one study included only paediatric injury population. Many of the validation studies also did not report proportion of missing data. Of those articles that mentioned about missing data, all excluded records with missed information from analyses.

Besides using in-hospital death as outcome, other studies included morbidity outcomes such as length of hospitalisation, damage control resuscitation, severe trauma, life-threatening injury, respiratory failure and sepsis. These morbidity outcomes are less standardised and therefore limit the ability for comparison.

Discussion

Our review points to an ongoing search for a comprehensive yet simple scoring system applicable to LMICs research and trauma care needs. While Glasgow Coma Scale, AIS and its derivatives, and TRISS methodology have established themselves as gold standards in injury research, there seems to be a need for injury severity measures that are reliable even in the light of the realities facing patient care systems in LMICs. Looking closely at the components of injury measures, it is evident that many complex measures require a host of information starting from prehospital phase until the discharge from the hospital. Henceforth, resources required to record the anatomical and biochemical evidence of injury severity are more readily available in high-income settings but may be difficult to obtain in resource-constrained environments.

Injuries and their physiological response are complex mechanisms, and the outcome of injuries is frequently affected by a number of factors ranging from age and pre-existing conditions of the patient to biochemical response of the body. It is difficult to account for all factors in a single model or severity measure; therefore, use of non-injury-specific-measures such as APACHE II, SOFAS and SAPS has gained traction in trauma research. Simple yet composite measures such as MGAP and KTS have become more popular which have been widely used and validated across the globe.9 25–27 Our review demonstrated that, although a number of injury severity measures were developed during the 1990s and early 2000s, there have been limited applications in LMICs. Furthermore, very few validation studies were conducted in low-income settings (online supplementary file 2). Over 70% of publications on injury research in LMICs have been published from only 11 countries (figure 3) which is obviously incomparable with their burden of injuries; moreover, the body of research comprises mostly of descriptive or epidemiological studies. Comparison of the most commonly applied injury measures aligns with the most commonly validated injury severity measures, including GCS, ISS, TRISS, APACHE and KTS scores. It is important to note that the majority validation studies have been conducted in upper-middle-income countries such as China, Turkey, Brazil and Thailand; involved single centres; or included specific study population such as head or abdominal injuries. New methods and models such as EISS, TISS and new TRISS have not been validated in other LMICs, outside of their origin.

A subset of studies found relatively low performance of injury severity measures which demonstrates large deviation from studies conducted in predominantly high-income settings (eg, TRISS, ISS). These differences may be due to a wide range of factors, such as delays in recording time sensitive injury data (such as blood pressure or GCS), training of personnel administering AIS codes, limited resources and equipment available for diagnosis, missed injuries and so on. Some recent studies confirm that commonly used injury severity measures that depend on in-depth information may not perform well in mortality prediction, especially with limited or incomplete data.25 26 Such differences underline the importance of assessing the performance and calibration of measures in specific contexts prior to their use in trauma registries or for outcome prediction. A review of publications on validation studies demonstrated that limited statistical analysis was performed in validation studies and the issue of missing data was not addressed. This may introduce bias in the estimates of performance of the injury severity measures. As mentioned before, many of the validation studies were limited with small sample size and single institutions, restricting to the specific setting and a lack of comparison among similar institutions within the country. Very often, the validation studies did not include statistical inference of the estimation, further restricting the ability to compare performance among injury severity measures inspected. Calibration is another feature of the measure that should be more commonly assessed.

Overall, our study has been able to highlight several important issues. First, the ‘10–90’ funding and research gap are also quite evident for injury and trauma, and we have observed that the amount of injury research from LMICs is still far less than the burden of injuries faced by these countries.28 The quality and depth of research is also not sufficient, being mostly limited to small empirical studies. The findings of validation studies focusing on mortality prediction highlight large variability in performance of commonly applied injury measures including GCS, ISS, RTS, TRISS and KTS. However, lack of large multicentre databases restricts the generalisability of results in large populations, even within a country.

The results nevertheless corroborate the assumption that no single injury measure has shown a consistent result in all settings and thus underscores the importance of context specific validation studies. This has also been reported previously from systematic reviews for injury severity measures such as ISS, NISS, ICISS and TMPM, mainly featuring studies from high-income settings.29 30 Furthermore, application of injury measures in field triage or emergency room disposition is also heavily influenced by the system of trauma care delivery, and hence, their performance in terms of prediction of survival, hospital length of stay or complications has to be tested and validated in specific settings where they are being used.

Our study has a few limitations. First, we conducted this literature review between 2006 and 2016, covering a 10-year period, and studies that were published outside of this timeframe are not included. Second, we have limited our literature search to three databases; nonetheless, inclusion of the Global Health database enabled us to review several Latin/South American publications that would have been otherwise missed. Third, we limited our detailed analysis of validation studies to those that focused on mortality prediction; this was due to a very limited number of studies focusing on a specific non-fatal outcome. We also did not focus on studies that used alternative coefficients for some of the established measures, as they were not consistently tested across settings.

Conclusion

The science of injury severity measurement has been growing to predict injury outcomes, help in decision-making and support epidemiological research. Empirical studies from upper-income and lower-middle-income countries frequently use injury severity measures. However, there is still a lack of large multicentre validation studies. The evidence base from low-income countries is even less established, where most of the burden of injury and trauma lies. No single injury severity measure has shown a consistent result in all settings and thus underscores the importance of context specific validation studies.

Acknowledgments

We acknowledge the support of Ms Peggy Gross, Ms Monika Kochar and Mr Armaan Rowther in acquiring scientific material and providing editorial assistance.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
  30. 30.
  31. 31.
  32. 32.
  33. 33.
  34. 34.
  35. 35.
  36. 36.
  37. 37.
  38. 38.
  39. 39.
  40. 40.
  41. 41.
  42. 42.
  43. 43.
  44. 44.
  45. 45.
  46. 46.
  47. 47.
  48. 48.
  49. 49.
  50. 50.
  51. 51.
  52. 52.
  53. 53.
  54. 54.
  55. 55.
  56. 56.
  57. 57.
  58. 58.
  59. 59.
  60. 60.
  61. 61.
  62. 62.
  63. 63.
  64. 64.
  65. 65.
  66. 66.
  67. 67.
  68. 68.
  69. 69.
  70. 70.
  71. 71.
  72. 72.
  73. 73.
  74. 74.
  75. 75.
  76. 76.
  77. 77.
  78. 78.
  79. 79.
  80. 80.
  81. 81.
  82. 82.
  83. 83.

Footnotes

  • Patient consent for publication Not required.

  • Contributors AM, HH and YWH conceptualised the study. SA, AM, YWH and HH participated in data extraction and analysis. AM and YWH produced the first draft of the manuscript, while AMB provided overall guidance and final review of all manuscript drafts.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Ethics approval This paper is based on detailed literature review; no personal or medical information are included in this study.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement There is no other unpublished data to share.