Article Text

Download PDFPDF

Validation of an algorithm using inpatient electronic health records to determine the presence and severity of cirrhosis in patients with hepatocellular carcinoma in England: an observational study
  1. Robert J Driver1,2,3,
  2. Vinay Balachandrakumar4,
  3. Anya Burton5,
  4. Jessica Shearer2,
  5. Amy Downing1,3,
  6. Tim Cross4,
  7. Eva Morris1,3,
  8. Ian A Rowe1,2,3
  1. 1Leeds Institute of Medical Research, University of Leeds, Leeds, UK
  2. 2Department of Hepatology, Leeds Teaching Hospitals NHS Trust, Leeds, UK
  3. 3Leeds Institute for Data Analytics, University of Leeds, Leeds, UK
  4. 4Royal Liverpool and Broadgreen University Hospitals NHS Trust, Liverpool, UK
  5. 5National Cancer Registration and Analysis Service, Bristol, UK
  1. Correspondence to Dr Robert J Driver; r.j.driver{at}


Objectives Outcomes in hepatocellular carcinoma (HCC) are determined by both cancer characteristics and liver disease severity. This study aims to validate the use of inpatient electronic health records to determine liver disease severity from treatment and procedure codes.

Design Retrospective observational study.

Setting Two National Health Service (NHS) cancer centres in England.

Participants 339 patients with a new diagnosis of HCC between 2007 and 2016.

Main outcome Using inpatient electronic health records, we have developed an optimised algorithm to identify cirrhosis and determine liver disease severity in a population with HCC. The diagnostic accuracy of the algorithm was optimised using clinical records from one NHS Trust and it was externally validated using anonymised data from another centre.

Results The optimised algorithm has a positive predictive value (PPV) of 99% for identifying cirrhosis in the derivation cohort, with a sensitivity of 86% (95% CI 82% to 90%) and a specificity of 98% (95% CI 96% to 100%). The sensitivity for detecting advanced stage cirrhosis is 80% (95% CI 75% to 87%) and specificity is 98% (95% CI 96% to 100%), with a PPV of 89%.

Conclusions Our optimised algorithm, based on inpatient electronic health records, reliably identifies and stages cirrhosis in patients with HCC. This highlights the potential of routine health data in population studies to stratify patients with HCC according to liver disease severity.

  • hepatology
  • hepatobiliary tumours
  • epidemiology

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:

View Full Text

Statistics from

Strengths and limitations of this study

  • First study to use inpatient electronic health records to identify and stage cirrhosis severity in a population with hepatocellular carcinoma.

  • The presence of cirrhosis predicted by inpatient electronic health records is accurate and advanced stage disease identified by the algorithm is associated with increased disease severity scores in validation.

  • A potential limitation is a variation in coding practices between centres and over time.

  • This algorithm may be used in population studies to understand outcomes in hepatocellular carcinoma, which require an assessment of liver disease severity.


Primary liver cancer accounts for 2% of all cancers diagnosed in the UK, with approximately 5700 new cases each year1 and these are most commonly hepatocellular carcinoma (HCC). It is estimated that 70%–90% of HCC occurs in the background of cirrhosis2 3 and global outcomes are poor despite a number of treatment options.4 Curative treatments may be limited by poor liver function due to underlying cirrhosis, or late presentation of advanced cancer in patients not known to have cirrhosis. Therefore, to understand outcomes of patients with HCC, it is essential to consider the presence and severity of cirrhosis in all analyses.

Population-based cancer registry data are used to describe trends in cancer incidence and mortality in a number of cancer sites, as well as regional variation in clinical outcomes.5 In HCC research, registry data have been used to describe geographical variation in incidence, survival and treatment allocation in France.6 In England, the National Cancer Registration and Analysis Service (NCRAS) dataset contains patient-level information about individuals with HCC, but information on the presence of cirrhosis is not currently included. Also, blood test results are not collected, so cirrhosis severity using tools such as the Child-Pugh score and the Model for End-Stage Liver Disease (MELD) score cannot be calculated.

Previous international studies have outlined methods to use electronic health records (EHRs) to identify cirrhosis.7–11 In the UK, Ratib et al used a combination of inpatient and outpatient records to identify cirrhosis and its complications, including oesophageal varices and ascites.12 These complications relate to advanced stage or ‘decompensated’ cirrhosis and they often result in admitted patient care. In England, all inpatient records are captured by the Hospital Episode Statistics (HES) database, which is linked to the cancer registry data within NCRAS.

We present a clinical validation study using EHRs from two regional cancer centres in England to assess the performance of an algorithm to determine the presence and severity of cirrhosis using the local inpatient HES records, which are subsequently transmitted to the national HES dataset. This study aims to demonstrate that the use of routinely collected diagnosis and treatment codes from inpatient records alone is sufficient to identify cirrhosis and grade its severity in patients with HCC. This will facilitate future studies of outcomes for patients with HCC by considering the severity of any underlying cirrhosis.


All patients diagnosed with HCC between 1 January 2007 and 31 December 2016 and residents in the secondary care catchment area of Leeds Teaching Hospitals NHS Trust (LTHT) were identified. The diagnosis of HCC was confirmed for all patients in a weekly hepatobiliary cancer multidisciplinary team (MDT) meeting and the reporting of all cases to the national cancer registry is mandatory. HCC was usually diagnosed by radiology, using the European Association for the Study of the Liver non-invasive criteria,13 and if indicated a targeted biopsy was performed. Live minutes are taken at these meetings and details collected into the clinical records along with a confirmed date of diagnosis. The cohort was identified from the data submitted to the central registry. We only had access to the inpatient codes from hospital episodes which occurred at LTHT. Therefore, only those patients registered with a Clinical Commissioning Group local to LTHT were included, where we would expect them to have received their inpatient cirrhosis care. The local HES records were searched to identify inpatient episodes containing codes related to cirrhosis within the International Classification of Diseases, 10th revision (ICD-10) and Office of Population, Censuses and Surveys’ Classification, fourth revision (OPCS4), together with the corresponding time interval from the HCC diagnosis date. These codes are used routinely for reimbursement and are submitted to the national HES dataset. An algorithm was developed to characterise patients from these codes, and comparison made with the clinical records. External validation of the algorithm was undertaken using the same search within the local HES records for patients diagnosed with HCC between 1 January 2013 and 31 December 2014 and local to Royal Liverpool and Broadgreen University Hospitals NHS Trust (RLBUHT).

Patient and public involvement

Patients and public were not involved in this study.

Identification of cirrhosis

To determine the presence of cirrhosis at HCC diagnosis, episodes containing cirrhosis-related codes which occurred up to 5 years before the HCC diagnosis date were initially included. However, to improve the sensitivity of the algorithm by maximising the number of available inpatient codes, additional episodes occurring after HCC diagnosis were subsequently included. This approach assumes that if an inpatient cirrhosis code occurs after the HCC diagnosis, the patient is likely to have had cirrhosis at the time of HCC diagnosis. The time frame post-HCC diagnosis of included episodes was increased incrementally and the performance of the algorithm tested to validate this assumption.

Different definitions of cirrhosis within ICD-10 have been used in population studies.14 Some investigators7 8 used cirrhosis diagnosis codes only, whereas others9 11 also included varices codes. Ratib et al15 additionally included OPCS4 procedure codes for treatment of varices and version 1 of our algorithm is based on this approach. Patients are classified as cirrhotic if they had inpatient episodes containing the diagnosis and treatment codes for cirrhosis or varices outlined in table 1. In version 2, a broader definition of cirrhosis proposed by Leon and McCambridge16 was used, including codes for ‘alcoholic liver disease’ (ALD, K70.9) and ‘alcoholic hepatic failure’ (AHF, K70.4). To assess the accuracy of including ascites as a cirrhosis-defining condition in HCC, codes for ascites and paracentesis were included in version 3 of the algorithm. Previously, some investigators9 15 excluded ascites in their definitions because this may be due to malignancy in the absence of cirrhosis in a general population. In version 4, only ascites codes occurring before the HCC diagnosis date were included.

Table 1

Treatment and procedure codes included in the algorithm to determine cirrhosis status and cirrhosis severity

The clinical records were reviewed between April and August 2018 and data abstracted by three clinical investigators (RJD, VB and JS), each experienced hepatology fellows working in this field for at least 2 years. A standard abstraction form was used and discrepancies were resolved by consensus view. Cirrhosis at the time of HCC diagnosis was identified based on explicit mention of cirrhosis in the clinical record or MDT minutes, evidence of portal hypertension on radiological imaging or endoscopy reports, explicit mention of cirrhosis on liver biopsy or a consistent result on transient elastography. This was used as the gold standard for testing different versions of the algorithm to classify cirrhosis status. For comparison, published algorithms7–9 15 were also tested in the LTHT cohort of patients with HCC.

Classification of cirrhosis severity

Cirrhosis severity was classified using the Baveno IV consensus.17 Compensated cirrhosis is defined by Baveno stage 1 (no ascites or varices) and stage 2 (non-bleeding varices). Decompensated cirrhosis is defined by Baveno stage 3 (ascites, with or without varices) and stage 4 (bleeding varices, with or without ascites). In this model of the natural history of cirrhosis,3 patients progress to a higher Baveno stage over time, but do not return to a lower stage. For each hospital episode, the Baveno stage and compensation status were calculated using the diagnosis and treatment codes for ascites and varices in table 1. Three definitions of bleeding varices were tested; version A (based on Goldberg et al10) contains ICD-10 codes for variceal bleeding, version B (based on Ratib et al12) also includes OPCS4 codes for treatment of varices, and version C limits the inclusion of these treatment codes to those occurring in a hospital episode with a concurrent ICD-10 code for gastrointestinal haemorrhage (K92.0, K92.1 and K92.2). This is to distinguish between bleeding varices and the prophylactic treatment of non-bleeding varices.

Cirrhosis severity at the time of HCC diagnosis was determined by the highest Baveno stage recorded in hospital episodes occurring in the 5 years before HCC diagnosis. In order to increase the accuracy of this assessment, additional episodes occurring after the HCC diagnosis date were also included. The time frame post-HCC diagnosis of included episodes was increased incrementally up to 4 months. The clinical records were reviewed to determine the true Baveno stage at the time of HCC diagnosis, along with routine blood tests for calculation of Child Pugh and MELD scores. Baveno stage 2 was identified by non-bleeding varices explicitly mentioned in the clinical records or endoscopy reports, but excluded a report of portal hypertensive gastropathy. Baveno stage 3 was identified by explicit mention of ascites in the clinical record, requiring diuretic therapy or paracentesis, but a small volume of ascites only visible on cross-sectional imaging was excluded. Baveno stage 4 was identified by explicit mention of variceal haemorrhage in the clinical record or endoscopy reports. Clinical evidence of decompensation was identified by the presence of bleeding varices or ascites, as per the Baveno IV classification.

Statistical analysis

Data management and statistical analysis were performed using Stata V.15.1 (StataCorp). The diagnostic accuracy of the algorithm to identify cirrhosis status and decompensation status involved comparison of sensitivity and specificity derived from 2×2 contingency tables.18 For Baveno stage, agreement between the algorithm and the clinical records were assessed using the kappa statistic. This is used to assess observer agreement for categorical variables and allows for agreement occurring by chance.19 20


Study population

During the study period, 289 patients (median age 69, 79% male) with a new diagnosis of HCC were included (table 2) and 249 (86.2%) of these had an inpatient record. Review of the clinical record identified 191 (66%) of these as cirrhotic at HCC diagnosis, 50 (26%) of whom had evidence of previous decompensation. The median age of the cirrhotic group was 67 compared with 73 in the non-cirrhotic group (p<0.001). An additional 15 patients had histological evidence of advanced fibrosis but cirrhosis was not mentioned explicitly in the clinical records. Among the patients who did not have an inpatient record, 12 had cirrhosis according to outpatient case note review. In the external validation cohort at RLBUHT, 50 patients meeting the inclusion criteria were assessed (median age 71, 82% male), 31 (62%) of whom were cirrhotic and 11 (35%) with previous decompensation.

Table 2

Baseline characteristics of the LTHT cohort

Identification of cirrhosis

Limiting the inclusion of episodes to those occurring before the HCC diagnosis results in a sensitivity of less than 50% for cirrhosis detection (table 3). When additional episodes are included up until 3 years after the HCC diagnosis, the sensitivity increases to greater than 80% for all versions of the algorithm, without significant loss of specificity.

Table 3

Performance of different versions of the cirrhosis status algorithm

The sensitivity of algorithm 1 is increased by including ALD and AHF (version 2), and further increased by including ascites (version 3). However, the inclusion of ascites also reduces the specificity. This is overcome by limiting the inclusion of ascites to episodes that occurred before the HCC diagnosis (version 4). Using this optimised algorithm and including records up to 3 years post-HCC diagnosis, the sensitivity is 86% (95% CI 82% to 90%) and the specificity is 98% (95% CI 96% to 100%), with a positive predictive value (PPV) of 99% and negative predictive value (NPV) of 79% (95% CI 74% to 83%) (online supplementary table 1). For external validation, when V.4 of the algorithm was applied to the RLBUHT cohort with 3 years of follow-up, the sensitivity was 79% and specificity was 100%. Additionally, version 4 of the algorithm outperformed published algorithms for cirrhosis detection when they were applied to the LTHT cohort of patients with HCC (table 4).

Table 4

Performance of different published algorithms for cirrhosis detection in the LTHT cohort of patients with HCC

Classification of cirrhosis severity

Table 5 shows the performance of the three versions of the algorithm for determining cirrhosis severity according to Baveno stage. Compared with version A, there is slightly less agreement between the calculated Baveno stage and the clinical record in version B, where Baveno stage 4 is defined by procedure codes for varices. Similarly, the sensitivity for detecting decompensation (defined by Baveno stages 3 and 4) is increased in version B, but with reduced specificity (online supplementary table 2). Agreement between the algorithm and the clinical record is optimised in version C, when bleeding varices are defined by a concurrent gastrointestinal haemorrhage code. Agreement was further improved when episodes occurring within 60 days of the registered HCC diagnosis were included. The performance characteristics of the component codes are summarised in online supplementary tables 3 and 4; the sensitivity for detecting bleeding varices is increased in algorithm B, but the PPV and overall agreement with the Baveno stage is reduced due to the misclassification of non-bleeding varices. The sensitivity for detecting ascites is increased when both diagnosis and paracentesis procedure codes are included.

Table 5

Performance of different versions of the Baveno stage algorithm

Using version C with a 60-day interval in the LTHT cohort, agreement between the clinical record and calculated Baveno stage was 84%, with a kappa coefficient of 0.74 (95% CI 71% to 77%). The sensitivity for detecting prior decompensation is 80% (95% CI 75% to 85%) and specificity is 98% (95% CI 96% to 100%), with a PPV of 89% (95% CI 85% to 93%) and NPV of 96% (95% CI 94% to 98%). When this version was applied to the RLBUHT cohort for external validation, the agreement of Baveno stage with the clinical record was 81% (kappa 0.70). The sensitivity for detecting decompensation was 73% and specificity was 90%.

Finally, among the 167 LTHT patients identified as cirrhotic by the algorithm, 45 (27%) were coded with prior decompensation. At the time of HCC diagnosis, Child-Pugh class and MELD scores were each higher in those individuals identified with decompensation (figure 1).

Figure 1

Box-and-whisker plots showing the distribution of MELD scores (A) and pie graphs showing the distribution of Child-Pugh class (B) within compensated and decompensated cirrhosis groups determined by the algorithm. MELD, Model for End-Stage Liver Disease.


Main findings

This study demonstrates the reliability of an algorithm using inpatient HES records to identify and stage cirrhosis in patients with HCC. This is the first such algorithm validated in a UK population that uses only inpatient codes. Using inpatient codes from the whole follow-up period improves the sensitivity of the algorithm in cirrhosis identification, without loss of specificity. This validates the assumption that if a patient had an inpatient cirrhosis code during follow-up, they had cirrhosis at the time of HCC diagnosis. Using a broad definition of cirrhosis (versions 2–4) improves sensitivity and accounts for variations in coding practice in which ALD and AHF are coded synonymously with cirrhosis. Excluding ascites after HCC diagnosis (version 4) improves the specificity; ascites in liver disease without HCC is most likely to be due to cirrhosis, whereas it may be malignant ascites in the context of HCC. Algorithm 4 is an improvement over published algorithms for cirrhosis detection when they are applied to our cohort of patients with HCC. Algorithm C (for assessing cirrhosis severity) also outperformed published versions in this population. Inclusion of a concurrent gastrointestinal haemorrhage code alongside variceal procedures distinguishes between treatment of bleeding varices and treatment of non-bleeding varices for primary prevention.

Strengths and limitations

The strengths of this study are the systematic development of an algorithm which uses routinely available inpatient episode codes, and its applicability to large population studies in HCC. These patients often require hospital admission to manage complications in advanced cirrhosis and to receive HCC therapies, or day case procedures such as paracentesis and endoscopy which are also coded in the HES dataset. The high-performance characteristics (particularly the PPVs) derived from inpatient codes here are in part a consequence of the high pretest probability of cirrhosis in patients with HCC. This observation is supported by the improved PPVs seen in existing algorithms in our cohort. In summary, this suggests that inpatient episodes are sufficient for high-quality analyses of the impact of cirrhosis and its severity on the outcomes of patients with HCC.

This study benefits from robust case note evaluation, using both a development and external validation cohort. In the UK, previous validation of inpatient coding was achieved using free-text analysis of primary care and death certification data,12 and the original case note validation of the cirrhosis algorithm included only 36 patients.21 The algorithm benefits from exploiting the ‘anchor point’ of the HCC diagnosis date, so that inpatient codes can be associated with a time interval. This has led to optimised cirrhosis detection and severity classification. The algorithm for cirrhosis detection was optimised using 3 years of follow-up after HCC diagnosis, but the high sensitivity and specificity using 1-year of follow-up may be sufficient in some settings.

The limitations include its location in specialist cancer centres, which may not reflect coding practices throughout the UK and these may change over time. However, portal hypertensive complications are common and often result in inpatient care and, since these are high-cost procedures, we anticipate them to be reliably coded. The analysis was limited to patients local to the two centres, in order to capture cirrhosis-related episodes. Additional episodes may have been missed if patients were admitted elsewhere, but these would be captured by the algorithm when extended to a national dataset. The majority of patients had an inpatient record, suggesting high rates of hospital admission in patients with cirrhosis and those undergoing HCC treatment. The limitations of using inpatient codes alone are common to other studies which have utilised the linked inpatient HES dataset to produce impactful analyses.22 23

The proportion of patients with cirrhosis identified from their clinical records was 66% and this is lower than previous reports.2 3 By limiting to inpatient codes, the algorithm missed 12/191 (6.3%) patients with cirrhosis and those with histological evidence of advanced fibrosis were classified as non-cirrhotic. Many patients were diagnosed with HCC in the absence of known liver disease; 68.4% of those without cirrhosis had no known underlying liver disease aetiology (table 2). If patients had advanced cancer at presentation, their clinical record may not have explicitly stated the presence of cirrhosis. Additionally, they may have not been investigated further to establish a diagnosis of cirrhosis if not clinically appropriate. It is also notable that there was a high proportion of patients aged over 80 years who were not identified to have cirrhosis. Finally, the definition of decompensation using the Baveno IV classification is limited because it does not capture hepatic encephalopathy (HE), which may occur without variceal bleeding or ascites. HE can be coded in ICD-10 code as ‘hepatic coma’, but we found that this was used uncommonly in our cohort and so we did not broaden our definition of decompensation beyond that used by Ratib et al.12


This algorithm can be applied to population cancer registries in the UK, enabling the identification and staging of cirrhosis in patients with HCC. This is essential for assessing clinical outcomes in population-based studies of individuals with HCC both in the UK and elsewhere. It is anticipated that this will lead to a better understanding of outcomes in HCC, including progression of underlying liver disease severity as well as overall survival. The algorithm may also be used in other population-based applications, which require the identification of cirrhosis and an assessment of severity.

In this study, we demonstrated the use of inpatient HES records to determine the cirrhosis severity at the time of HCC diagnosis. The algorithm may be adapted to classify the Baveno stage at different time intervals following HCC diagnosis or date of treatment, so that subsequent cirrhosis decompensation events can be identified over time. This approach is likely to have value in other health systems and we anticipate that the algorithm described will be evaluated by other investigators in outcomes-oriented research in cirrhosis and HCC.


This study demonstrates the reliability of an algorithm based on inpatient EHRs to stratify patients with HCC according to the presence and severity of cirrhosis. It may be used in routine health data in order to assess outcomes in HCC in population studies.


This work uses data provided by patients and collected by the NHS as part of their care and support. The planning of this project was discussed with members of the HCC-UK Partnership, which is coordinated by AB.


  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
View Abstract


  • Contributors IAR and RJD had the original idea for the study and AB, AD, TC and EM contributed to its design and planning. RJD, JS and VB performed the case note reviews. RJD was responsible for data management, statistical analyses and wrote the first draft of the paper. IAR reviewed the paper critically and VB, AB, JS, AD, TC and EM all approved the final version.

  • Funding RJD and JS were funded by clinical research fellowships at Leeds Teaching Hospitals NHS Trust. AB is an analyst within Public Health England and receives funding from the British Association for the Study of the Liver (BASL) as part of the HCC-UK Partnership.

  • Disclaimer No industrial sponsor had input into study planning, discussion of results or manuscript preparation.

  • Competing interests VB has received an unrestricted educational grant from Bayer, Bristol-Myers Squibb and Sirtex.

  • Ethics approval This retrospective study comprises an assessment of the accuracy of clinical coding for service evaluation and as such does not require formal ethical approval. All patient data were anonymised and permission was granted from the Caldicott Guardian for sharing of routinely collected anonymised data.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement Statistical code is available from thecorresponding author RJD.

  • Patient consent for publication Not required.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.