Intelligent visualization and exploration of time-oriented data of multiple patients

doi:10.1016/j.artmed.2010.02.001

Artificial Intelligence in Medicine

Volume 49, Issue 1, May 2010, Pages 11-31

https://doi.org/10.1016/j.artmed.2010.02.001 Get rights and content

Abstract

Objective

Clinicians and medical researchers alike require useful, intuitive, and intelligent tools to process large amounts of time-oriented multiple-patient data from multiple sources. For analyzing the results of clinical trials or for quality assessment purposes, an aggregated view of a group of patients is often required. To meet this need, we designed and developed the VISualizatIon of Time-Oriented RecordS (VISITORS) system, which combines intelligent temporal analysis and information visualization techniques. The VISITORS system includes tools for intelligent retrieval, visualization, exploration, and analysis of raw time-oriented data and derived (abstracted) concepts for multiple patient records. To derive meaningful interpretations from raw time-oriented data (known as temporal abstractions), we used the knowledge-based temporal-abstraction method.

Methods

The main module of the VISITORS system is an interactive, ontology-based exploration module, which enables the user to visualize raw data and abstract (derived) concepts for multiple patient records, at several levels of temporal granularity; to explore these concepts; and to display associations among raw and abstract concepts. A knowledge-based delegate function is used to convert multiple data points into one delegate value representing each temporal granule. To select the population of patients to explore, the VISITORS system includes an ontology-based temporal-aggregation specification language and a graphical expression-specification module. The expressions, applied by an external temporal mediator, retrieve a list of patients, a list of relevant time intervals, and a list of time-oriented patients’ data sets, by using an expressive set of time and value constraints.

Results

Functionality and usability evaluation of the interactive exploration module was performed on a database of more than 1000 oncology patients by a group of 10 users—five clinicians and five medical informaticians. Both types of users were able in a short time (mean of 2.5 ± 0.2 min per question) to answer a set of clinical questions, including questions that require the use of specialized operators for finding associations among derived temporal abstractions, with high accuracy (mean of 98.7 ± 2.4 on a predefined scale from 0 to 100). There were no significant differences between the response times and between accuracy levels of the exploration of the data using different time lines, i.e., absolute (i.e., calendrical) versus relative (referring to some clinical key event). A system usability scale (SUS) questionnaire filled out by the users demonstrated the VISITORS system to be usable (mean score for the overall group: 69.3), but the clinicians’ usability assessment was significantly lower than that of the medical informaticians.

Conclusions

We conclude that intelligent visualization and exploration of longitudinal data of multiple patients with the VISITORS system is feasible, functional, and usable.

Section snippets

Introduction: intelligent visualization of time-oriented data of multiple patients

A key task facing clinicians and medical researchers is the analysis of time-stamped, longitudinal medical records, particularly, records of multiple patients. This capability is necessary to support, for example, quality assessment tasks, analysis of clinical trials, and the discovery of new clinical knowledge. Although the task of accessing patient data has been solved mostly through the increasing use of electronic medical record (EMR) systems, there still remains the task of intelligent

Combining domain knowledge, temporal abstraction and information visualization in medical informatics

The use of a domain knowledge base can both support an in-depth analysis of longitudinal patient records and simplify and facilitate the data exploration process, since the user can explore only high-level concepts based on complex temporal patterns (or, in general, on any abstract concepts) previously defined in a domain-specific knowledge base and detected in the patients’ data. In addition, it has been demonstrated that visual representation can often communicate information much more

Desiderata for effective exploration of time-oriented data for multiple patients

Our study of the problem of effective and usable visualization and exploration of raw clinical data and, especially, of derived meaningful abstractions from these data, revealed the following set of desiderata for the intelligent interface and exploration operators supporting the task of exploration of time-oriented data for multiple patients.

1.
Evaluation of the functionality and usability of the KNAVE-II system for exploration of longitudinal data of individual patients [2] demonstrated the

Methods

The specific methods that we used are presented below in six subsections, each in the context of its relevant desideratum as presented in Section 3.

Example of a clinical scenario

In this section, we present an example of an exploration clinical scenario for application of the VISITORS system and an analysis using a TAC. The example, which relates to a retrospective database of bone-marrow transplantation (BMT) patients (see evaluation Section 6), comprises an investigation of the bone-marrow recovery characteristics of patients, who are either young [<20 years] or old [>70 years], following an autologous BMT procedure.

(1)
In a previous study [4] we introduced the graphical

Research questions

We designed and developed the VISITORS system according to the desiderata listed in Section 3, and envision it as potentially useful for two types of users: clinicians and medical informaticians. We also envision that the system can be used to answer different clinically motivated questions. We conducted an evaluation of the system with the aim of answering the following four research questions:

1.
Overall functionality and usability: are the interactive exploration operators of the VISITORS system

Results

This section summarizes the evaluation results in terms of the four research questions defined in Section 6.1.

Contributions and advantages

The major contribution of the VISITORS system is the provision of a comprehensive environment for intelligent, i.e., knowledge-based, investigation of time-oriented data for multiple patients, including the specification and retrieval, visualization, exploration, and analysis of the time and value associations among both raw and abstract clinical concepts. Based on the results of the current study, the VISITORS system might be described as an “intelligent equalizer” for data interpretation and

Acknowledgments

This research was supported by Deutsche Telekom Labs at Ben-Gurion University of the Negev and the Israel Ministry of Defense, BGU award No. #89357628-01. We thank all the clinicians and medical informaticians who contributed their time to the evaluation. We thank Ms. Efrat German for her work on the Tempura system, and Mr. Ido Hacham and Mr. Shahar Albia for their work on the Multi-TOQ system.

References (62)

Y. Shahar
A framework for knowledge-based temporal abstraction
Artif Intell Med
(1997)
Y. Shahar et al.
Distributed, intelligent, interactive visualization and exploration of time-oriented clinical data and their abstractions
Artif Intell Med
(2006)
S. Martins et al.
Evaluation of an architecture for intelligent query and exploration of time-oriented clinical data
Artif Intell Med
(2008)
S. Miksch et al.
Utilizing temporal data abstraction for data validation and therapy planning for artificially ventilated newborn infants
Artif Intell Med
(1996)
C. Combi et al.
Abstraction on clinical data sequences: an object-oriented data model and a query language based on the event calculus
Artif Intell Med
(1999)
L. Chittaro
Information visualization and its application to medicine
Artif Intell Med
(2001)
M. Spenke
Visualization and interactive analysis of blood parameters with InfoZoom
Artif Intell Med
(2001)
G. Falkman
Information visualisation in clinical Odontology: multidimensional analysis and interactive data exploration
Artif Intell Med
(2001)
L. Chittaro et al.
Data mining on temporal data: a visual approach and its clinical application to hemodialysis
J Vis Lang Comput
(2003)
S. Silva et al.
Formalizing visual interaction with historical databases
Inform Systs;
(2002)

J. Dionisio et al.

MQuery: a visual query language for multimedia, timeline and simulation data

J Visual Lang Comput

(1996)

L. Chittaro et al.

Visualizing queries on databases of temporal histories: new metaphors and their evaluation

Data Knowl Eng

(2003)

C. Bettini et al.

Solving multi-granularity temporal constraint networks

Artif Intell

(2002)

A. Narayanan et al.

Iconic SQL: practical issues in the querying of database through structured iconic expressions

J Visual Lang Comput

(2002)

D. Boaz et al.

A distributed temporal-abstraction mediation architecture for medical databases

Artif Intell Med

(2005)

Y. Shahar et al.

Knowledge-based temporal abstraction in clinical domains

Artif Intell Med

(1996)

Klimov D, Shahar Y, Taieb-Maimon M. Intelligent selection and retrieval of multiple time-oriented records. Journal of...

M. Hearst

User interfaces and visualization

M. Goldstein et al.

Graphical displays to improve guideline-based therapy of hypertension

A.-S. Silvent et al.

Multi-level temporal abstraction for medical scenario construction

Int J Adapt Control

(2005)

J. Cooley et al.

An algorithm for the machine calculation of complex Fourier series

Math Comput

(1965)

S. Miksch et al.

Abstraction and representation of repeated patterns in high-frequency data

R. Kowalaski et al.

A logic-based calculus of events

New Gener Comput

(1986)

C. Plaisant et al.

LifeLines: visualizing personal histories

Plaisant C., Mushlin R., Snyder A., Li J., Heller D., Shneiderman B. LifeLines: using visualization to enhance...

T. Wang et al.

Aligning temporal data by sentinel events: discovering patterns in electronic health records

W. Aigner et al.

Visual methods for analyzing time-oriented data

IEEE Trans Visual Comput Graph

(2008)

S. Navathe et al.

Temporal extensions to the relational model and SQL

N. Sarda

HSQL: A historical query language

R. Snodgrass

The TSQL2 temporal query language

(1995)

C. Combi et al.

Querying temporal clinical databases with different time granularities: the GCH-OSQL language

Cited by (91)

Computing and analyzing decision boundaries from shortest path maps
2023, Computers and Graphics (Pergamon)
This paper proposes a methodology for computing, visualizing, and analyzing critical decision boundaries for the selection of shortest paths in a given environment. Decision boundaries are defined as the points in a map from which two or more different shortest paths exist towards a destination. This paper introduces the problem of visualizing their evolution, taking into account moving obstacles, moving goals, and as well multiple goals. The proposed visualizations enable analyzing which paths should be taken and at which departure times, such that a destination can be reached by the shortest possible path when taking into account a moving target or time-varying areas to be avoided. The proposed techniques are also applied to the analysis and improvement of exit placement in a given environment, in order to improve the evacuation flow in emergency situations.
An overview on state-of-the-art electrocardiogram signal processing methods: Traditional to AI-based approaches
2023, Expert Systems with Applications
Over the last decade, cardiovascular diseases (CVD’s) are the leading cause of death globally. Early prediction of CVD’s can help in reducing the complications of high-risk patients. The electrocardiogram (ECG) is an efficient aiding tool, provides accurate information about various cardiac conditions of the human heart. Evaluation and interpretation of ECG signal has become the major goal in current research to identify and mitigate risky cardiovascular conditions. The ECG signal is efficiently analysed and classified using signal processing techniques, ranging from traditional to machine learning approaches and its subbranches, such as deep learning are used for the early detection and diagnosis of cardiac conditions and arrhythmias. The development of novel types of body sensors increases the need for automated, low-cost, real-time, and efficient ECG monitoring systems that can be used at home or in ambulatory settings. IoT in healthcare industry enables the doctors and specialists to diagnose the patient status remotely in a smart and efficient manner. Motivated with all the research done in this area for early diagnosis of cardiac abnormalities and save the life of high-risk patients, we surveyed numerous research works reported in the literature and provide a glance on several aspects of ECG signal monitoring systems. We first introduce the step-by-step framework of ECG signal analysis starting data acquisition to classification, describes each stage analysis of both traditional and advanced machine learning models that have been reported in the literature. We provide a deep discussion on ECG signal acquisition, pre-recorded clinical ECG data taken from the databases, signal processing and denoising, detection of feature based on feature engineering, and signal classification along with the comparative assessment amongst reviewed studies. This work also presents the detailed analysis of smart health care system that employs portable and wearable devices, as well as the Internet of Things (IoT) and wireless technologies, as an innovative medium for data transmission that allows for remote access to a patient’s health status at lower costs and with greater efficiency. Moreover, this work provides an extensive knowledge on various hardware platforms that have been adopted for the development of biomedical processors including microcontrollers, FPGA and ASIC. Additionally, the challenges and limitations are discussed in this research field and directions for future work are suggested. This is the first review paper of its kind to present a comprehensive discussion on all aspects of cardiovascular disease monitoring systems in one place.
Guidelines and workflow models
2023, Clinical Decision Support and beyond: Progress and Opportunities in Knowledge-Enhanced Health and Healthcare
A heterogeneous community is developing computerized methods and systems for supporting health care processes. One approach focuses on assuring the quality of care by delivering decision support to care providers based on clinical knowledge, and the other focuses on supporting management of routine medical actions, such as ordering laboratory tests and medications or scheduling patient visits. The first community is more academic, and studies classical clinical decision support, while the second originated in industry, and focuses on application and adaptation of business process management and mining techniques to the healthcare domain. Both of these approaches have made important achievements, yet their integration is far from being realized. In this chapter, we follow a life-cycle approach, reviewing the contributions of the two communities mentioned, by considering the stage of the life cycle that each achievement addresses: from knowledge acquisition to knowledge (process) modeling and specification, validation and verification, enactment, process mining and improvement, and sharing of executable guideline knowledge. We stress the potential and achievements made in integration of guideline and workflow knowledge, and discuss the success stories and limitations, and the opportunities for future research.
Visualization of frequent temporal patterns in single or two populations
2022, Journal of Biomedical Informatics
Temporal knowledge discovery in clinical problems, is crucial to investigate problems in the data science era. Meaningful progress has been made computationally in the discovery of frequent temporal patterns, which may store potentially meaningful knowledge. However, for temporal knowledge discovery and acquisition, effective visualization is essential and still stores much room for contributions. While visualization of frequent temporal patterns was relatively under researched, it stores meaningful opportunities in facilitating usable ways to assist domain experts, or researchers, in exploring and acquiring temporal knowledge. In this paper, a novel approach for the visualization of an enumeration tree of frequent temporal patterns is introduced for, whether mined from a single population, or for the comparison of patterns that were discovered in two separate populations. While this approach is relevant to any sequence-based patterns, we demonstrate its use on the most complex scenario of time intervals related patterns (TIRPs). The interface enables users to browse an enumeration tree of frequent patterns, or search for specific patterns, as well as discover the most discriminating TIRPs among two populations. For that a novel visualization of the temporal patterns is introduced using a bubble chart, in which each bubble represents a temporal pattern, and the chart axes represent the various metrics of the patterns, such as their frequency, reoccurrence, and more, which provides a fast overview of the patterns as a whole, as well as access specific ones. We present a comprehensive and rigorous user study on two real-life datasets, demonstrating the usability advantages of the novel approaches.
Increasing and standardizing quality of care using computerized guidelines for clinical decision support
2022, Personalized Health Systems for Cardiovascular Disease
In recent decades there has been strong orientation toward greater use of evidence-based medicine approaches and information technology in the provision of personalized healthcare services. The chapter introduces the use of clinical practice guidelines to increase the accessibility of scientific evidence to healthcare professionals and patients. It describes application frameworks and the life cycle of the creation of computerized guidelines to facilitate clinical decision support. Examples are focused on development of computerized guidelines for preeclampsia and heart failure that support automatic inference and patient-specific advice delivered at the point of care. In the preeclampsia scenario, assessments were made regarding the feasibility of application of formalized clinical guidelines, improvement of physician compliance with guidelines, and whether clinicians considered the framework for guidelines application helpful. The heart failure scenario demonstrated the modified knowledge acquisition methodology that resulted in formalized guideline with 167 recommendations for heart failure application in home care.
Implementation and evaluation of a multivariate abstraction-based, interval-based dynamic time-warping method as a similarity measure for longitudinal medical records
2021, Journal of Biomedical Informatics
A common prerequisite for tasks such as classification, prediction, clustering and retrieval of longitudinal medical records is a clinically meaningful similarity measure that considers both [multiple] variable (concept) values and their time. Currently, most similarity measures focus on raw, time-stamped data as these are stored in a medical record. However, clinicians think in terms of clinically meaningful temporal abstractions, such as “decreasing renal functions”, enabling them to ignore minor time and value variations and focus on similarities among the clinical trajectories of different patients. Our objective was to define an abstraction- and interval-based methodology for matching longitudinal, multivariate medical records, and rigorously assess its value, versus the option of using just the raw, time-stamped data.
We have developed a new methodology for determination of the relative distance between a pair of longitudinal records, by extending the known dynamic time warping (DTW) method into an interval-based dynamic time warping (iDTW) methodology. The iDTW methodology includes (A): A three-steps interval-based representation (iRep) method: [1] abstracting the raw, time-stamped data of the longitudinal records into clinically meaningful interval-based abstractions, using a domain-specific knowledge base, [2] scoping the period of comparison of the records, [3] creating from the intervals a symbolic time series, by partitioning them into a predetermined temporal granularity; (B) An interval-based matching (iMatch) method to match each relevant pair of multivariate longitudinal records, each represented as multiple series of short symbolic intervals in the determined temporal granularity, using a modified DTW version.
Three classification or prediction tasks were defined: (1) classifying 161 records of oncology patients as having had autologous versus allogenic bone-marrow transplantation; (2) classifying the longitudinal records of 125 hepatitis patients as having B or C hepatitis; and (3) predicting micro- or macro-albuminuria in the second year, for 151 diabetes patients who were followed for five years. The raw, time-stamped, multivariate data within each medical record, for one, two, or three concepts out of four or five concepts judged as relevant in each medical domain, were abstracted into clinically meaningful intervals using the Knowledge-Based Temporal-Abstraction method, using previously acquired knowledge. We focused on two temporal-abstraction types: (1) State abstractions, which discretize a concept’s raw value into a predetermined range (e.g., LOW or HIGH Hemoglobin); and (2) Gradient abstractions, which indicate the trend of the concept’s value (e.g., INCREASING, DECREASING Hemoglobin value). We created all of the combinations of either uni-dimensional (State or Gradient) or multi-dimensional (State and Gradient) abstractions, of all of the concepts used. Classification of a record was determined by using a majority of the k-Nearest-Neighbors (KNN) of the given record, k ranging over the odd numbers (to break ties) from 1 to $\sqrt{N}$ , N being the size of the training set. We have experimented with all possible configurations of the parameters that our method uses. Overall, a total of 75,936 experiments were performed: 33,600 in the Oncology domain, 28,800 in the Hepatitis domain, and 13,536 in the Diabetes domain. Each experiment involved the performance of a 10-fold Cross Validation to compute the mean performance of a particular iDTW method-configuration set of settings, for a specific subset of one, two, or three concepts out of all of the domain-specific concepts relevant to the classification or prediction task on which the experiment focuses. We measured for each such experimental combination the Area Under the Curve (AUC) and the optimal Specificity/Sensitivity ratio using Youden's Index. We then aggregated the experiments by the types of unidimensional or multidimensional abstractions used in them (including the use of only raw concepts as a special case); for example, two state abstractions of different concepts, and one gradient abstraction of a third concept. We compared the mean AUC when using each such feature representation, or combination of abstractions, across all possible method-setting configurations, to the mean AUC when using as a feature representation, for the same task, only raw concepts, also across all possible method-setting configurations. Finally, we applied a paired t-test, to determine whether the mean difference between the accuracy of each temporal-abstraction representation, across all concept and configuration combinations, and the respective raw-concept combinations, across all concept subset and configuration combinations, is significant (P < 0.05).
The mean performance of the classification and prediction tasks when using, as a feature representation, the various temporal-abstraction combinations, was significantly higher than that performance when using only raw data. Furthermore, in each domain and task, there existed at least one representation using interval-based abstractions whose use led, on average (over all concept subset combinations and method configurations) to a significantly better performance than the use of only subsets of the raw time-stamped data. In seven of nine combinations of domain type (out of three) and number of concepts used (one, two, or three), the variance of the AUCs (for all representations and configurations) was considerably higher across all raw-concept subsets, compared to all abstract combinations. Increasing the number of features used by the matching task enhanced performance. Using multi-dimensional abstractions of the same concept further enhanced the performance. When using only raw data, increasing the number of neighbors monotonically increased the mean performance (over all concept combinations and method configurations) until reaching an optimal saddle-point around $\sqrt{N}$ ; when using abstractions, however, optimal mean performance was often reached after matching only five nearest neighbors.
Using multivariate and multidimensional interval-based, abstraction-based similarity measures is feasible, and consistently and significantly improved the mean classification and prediction performance in time-oriented domains, using DTW-inspired methods, compared to the use of only raw, time-stamped data. It also made the KNN classification more effective. Nevertheless, although the mean performance for the abstract representations was higher than the mean performance when using only raw-data concepts, the actual optimal classification performance in each domain and task depends on the choice of the specific raw or abstract concepts used as features.

View all citing articles on Scopus

View full text

Intelligent visualization and exploration of time-oriented data of multiple patients

Abstract

Objective

Methods

Results

Conclusions

Section snippets

Introduction: intelligent visualization of time-oriented data of multiple patients

Combining domain knowledge, temporal abstraction and information visualization in medical informatics

Desiderata for effective exploration of time-oriented data for multiple patients

Methods

Example of a clinical scenario

Research questions

Results

Contributions and advantages

Acknowledgments

Artif Intell Med

Artif Intell Med

Artif Intell Med

Artif Intell Med

Artif Intell Med

Artif Intell Med

Artif Intell Med

Artif Intell Med

J Vis Lang Comput

Inform Systs;

J Visual Lang Comput

Data Knowl Eng

Artif Intell

J Visual Lang Comput

Artif Intell Med

Artif Intell Med

User interfaces and visualization

Graphical displays to improve guideline-based therapy of hypertension

Multi-level temporal abstraction for medical scenario construction

Int J Adapt Control

An algorithm for the machine calculation of complex Fourier series

Math Comput

Abstraction and representation of repeated patterns in high-frequency data

A logic-based calculus of events

New Gener Comput

LifeLines: visualizing personal histories

Aligning temporal data by sentinel events: discovering patterns in electronic health records

Visual methods for analyzing time-oriented data

IEEE Trans Visual Comput Graph

Temporal extensions to the relational model and SQL

HSQL: A historical query language

The TSQL2 temporal query language

Querying temporal clinical databases with different time granularities: the GCH-OSQL language