EHR phenotyping

Phenotyping is the process of identifying cohorts and variables from raw EHR.16 Because we use OMOP CDM-normalised EHR, phenotyping is the process of finding ‘OMOP concepts’ that correspond to specific cohorts (eg, all patients with ART-related visits) or variables (eg, myocardial infarction).17 An OMOP concept is a unique identifier that is mapped to diverse medical codes that have the same semantic meaning but may be from different medical nomenclatures and EHR systems.

We will use standard phenotyping procedures. The logical procedures are as follows: (1) To identify existing OMOP concept sets (available in OHDSI’s Atlas system and N3C) that can be used with minor revision. (2) If no appropriate concept sets available, we will follow the generic phenotyping procedures16 18 and curate OMOP CDM concepts from the Athena vocabulary repository, which allows retrieval of OMOP CDM concepts.19 (3) To validate the revised or newly developed concept sets by using EHR chart review that is performed independently by two domain experts.16

Data retrieval and integration

To link external epidemiological and community-level data with individual-level EHR data, we will use individual patients’ residential ZIP/county as the reference. We will use data imputation algorithms to impute missing values because population-towards-individual data integration automatically creates missing values. For existing missing values in EHR, we will infer missing values based on semantic relationships of OMOP CDM concepts. For geographical locations, we will use cities, counties and states to infer locations at appropriate levels. For the rest of the missing values, we will use multiple imputation methods. Specifically, we will use selection models or pattern-mixture models for systematic missingness. We will also selectively use mean/median imputation, principal component analysis, singular value decomposition, k-nearest neighbour, least squares, expectation maximisation and random forest.20 21 Data retrieval and integration will be implemented using SQL, R, Python and PySpark, whichever appropriate.

Graphical EHR model

We will develop a customised graphical model to represent the temporal relations among patients’ demographic, clinical and sociobehavioural data. A Graphical model encodes clinical events as nodes and their semantic relations (including temporal relations) as edges. Figure 1 shows the proposed design of the model. General modelling procedures include: first, we will retrieve time stamps of clinical events. Some clinical events have explicit time stamps as captured by structured EHR. Many others do not (eg, patient-reported symptoms as documented in clinical notes, trimester and/or gestational age). For those that do not have explicit time stamps, we will infer the time of the event based on neighbouring EHR data. For example, symptom onsets could be found in clinical notes (eg, admission, discharge); trimester and gestational age could be estimated by gestation-related diagnoses (eg, Z3A: weeks of gestation), procedures (eg, ultrasound procedures) and clinical notes (eg, last menstrual period).22 23 Second, we will represent temporal information of clinical events by modelling the occurrences of an event and the instantaneous impact of the event. The occurrences of an event are resulted from the first step. The instantaneous impact of an event is formulated using exponential kernel functions and association rules based on clinical observation. Intuitively, two clinical events with a long interval in between would have less impact on one another, but this can be overwritten by events that hold special clinical meanings. Third, we will create recurrent states for every clinical event by embedding the event, its semantic relations (ie, edge in a Graph) including the instantaneous impact of an event, and time. These recurrent states will form a recurrent layer to be used for training machine learning models, which will be discussed later.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Electronic Health Records model design.

Cohort

Based on the estimated >13 000 patients with the coinfection in our dataset, we will blend in controls (COVID-19 patients without HIV) for each output variable using a match ratio of 1:2, stratified by sex, race/ethnicity and age. For a possible occasion of small sample, for example, death cases (n<1000 with coinfection), the alternative strategy is to create synthetic cases to impute and balance the sample.

Machine learning input

A complete and longitudinal health history together with linked external epidemiological and community-level data will be included as the input of machine learning models by which the models can learn from the input to predict individuals’ in-time clinical outcomes and disease prognosis. We will include but are not limited to the following phenotypic traits: demographics, SDOH, diagnoses, underlying conditions, vitals, laboratory tests, procedures, medication prescriptions/dispensing, medication exposure (eg, vaccine) and annotated clinical notes. Because there is no gold standard measure for ART adherence, we will use ‘multiple measures’ to estimate levels of ART adherence.25 Multiple measures include medication events (inpatient dispensing), HIV-1 RNA copies (laboratory results) and medication adherence data from clinical notes. For those with complete medication adherence data, we use the proportion of days covered to categorise ART adherence levels (eg, <50%, 50%–80%, 80%–85%, 85%–90%, ≥90%).26 We will categorise antiviral medications into integrase inhibitor-based, non-nucleoside reverse transcriptase inhibitor-based, protease inhibitor-based and other regimens. The approach to measuring ART adherence using EHR has limitations, but the limitations can be mitigated by the well-presented and large-scale national data. We will collect both CD4⁺ counts as an indicator of existing damage and plasma HIV-1 RNA copies as an indicator of projected disease progression.

Machine learning output

Clinical outcome measures as machine learning output include inpatient admissions, length of stay (LOS), ICU admission, ICU LOS, comorbidities and primary discharge diagnosis.27 In addition to the measures within the acute phase of COVID-19, we will also use symptoms, diagnoses, comorbidities and PASC-associated readmissions as outcome and prognosis measures for individuals in the postacute phase.

Model design

We will use RNN as the machine learning architecture to learn from patients’ longitudinal EHR and make the prediction of current and future clinical outcomes and disease prognosis. We adopt RNN models because this neural network architecture is specialised for capturing temporal dependency among event sequences. The RNN models will be trained to learn from model input and to make the prediction of model output. With respect to the embedding, we will include standard long short-term memory (LSTM) as well as phased LSTM and other variants wherever appropriate.24 We will use the bag-of-pattern matrix as the baseline embedding method to be compared against LSTM, in which this baseline method does not fully consider temporal dependency. To test against RNN, we will use Support Vector Machine (SVM) as the benchmark algorithm, in which SVM is a well-performed kernel-based algorithm28 but does not take full advantage of temporal dependency (hypothesis 2) and personalised health records (hypothesis 1). Because the nature of our machine learning output is binary variables, the proposed machine learning tasks are essentially binary classification tasks. We will use Python for machine learning modelling.

Model evaluation (internal validity)

To test the effectiveness of the prediction model, we will use 10-fold cross validation. With respect to evaluation metrics, we will use the F score, precision, recall, and the area under the receiver operating characteristic (AUC) to assess the models’ predictive performance. We expect the F score, assuming balanced data, to reach a minimum of 0.8. If trained models fail to meet this expectation, alternative strategies include manually adding features handpicked by researchers after error analysis of models.

CDS workflow

The proposed CDS is a hybrid of knowledge-based and non-knowledge-based system.29 It has (1) a machine learning-based prediction module (non-knowledge based) for identifying high-risk patients and (2) a provider-curated medical logic module (knowledge-based) for generating clinical actions for identified high-risk patients. The CDS testing takes place in a retrospective way (ie, using retrospective EHR).

Cohort definition and data collection

Using retrospective EHR data (2-year baseline~2023) from Prisma Health’s Epic system, we will first group the existing PLWH who have COVID-19 (sampling n>500) based on their state at the point of CDS screening. The patient states include (1) ambulatory patients with COVID-19, (2) hospitalised patients for COVID-19 with moderate disease, (3) hospitalised patients for COVID-19 with severe disease and (4) post-acute phase of COVID-19 (ie, from beyond 4 weeks after symptom onset).30 See table 1 for definitions of states 1–3 based on WHO’s clinical progression scale for COVID-19.

Prediction module

For patients in each state, we will use the trained machine learning model to learn from previous medical records and predict worsening clinical outcomes as time progresses (ie, acute, and postacute phases every 3 months. The prediction will include primary COVID-19 clinical outcomes (Box 1) developed by the WHO Working Group on the Clinical Characterisation and Management of COVID-19.31

Medical logic module

Patients identified by the CDS to have an increased risk of worse clinical outcomes will be reviewed and discussed by two providers who are specialised in HIV and COVID-19. First, the providers will generate gold-standard judgement on whether a patient is correctly identified by the prediction module, which later will be used for assessing the effectiveness of CDS. Second, the providers will generate appropriate clinical actions on chart review. These clinical actions will be made up to date with the ‘NIH Guidance for COVID-19 and People with HIV’, including treatment options based on cohorts and risk factors, medication reconciliation considering ART regimens, consultation with specialists for multiorgan system complications and PASC, referrals and outreach.32 Providers’ decision-making processes will be programmed using Arden Syntax (V.3) or Clinical Quality Language33 in the knowledge base, which is determined by specific Epic EHR data model.

Effectiveness of CDS (external validity)

There are two evaluation metrics: (1) appropriate identification for individuals at high risk for adverse clinical outcomes (Box 1) by comparing model-identified cases against the gold standard generated from chart review. We will use F measure (>0.8), AUC, precision and recall for assessment; (2) appropriate clinical actions using a quasiexperimental design. We will compare outcomes of patients who naturally used the medical logic module-suggested care against those who did not (n=100 each). The outcomes include but are not limited to readmissions (eg, same day, 7, 14, 30 days), healthcare utilisation (eg, LOS, emergency room (ER)/observation visits, ICU admission). We will use mixed-effect generalised regression models to estimate model effectiveness wherever appropriate.

Usability testing

We will assess CDS usability by adopting the ‘think aloud’ protocol.34 The two providers from Prisma Health will participate in the test. Each one will be presented with randomly selected EHR (n=5 at-risk cases+n=5 control cases) along with the CDS output. In each case, participants will be instructed to verbalise their reasoning procedures (eg, phenotypic traits from EHR that can be used in the reasoning, logic flow) towards identifying at-risk patients and corresponding clinical decisions. Sessions are audio recorded and will then be coded (eg, by content, understandability, navigation, workflow, visibility and usability) independently by two researchers for downstream analyses.