Effects of errors in a multicenter medical study: Preventing misinterpreted data

doi:10.1016/0022-3956(94)90003-5

Journal of Psychiatric Research

Volume 28, Issue 5, September–October 1994, Pages 447-459

https://doi.org/10.1016/0022-3956(94)90003-5 Get rights and content

Abstract

Large research projects offer significant advantages for research, but they pose special data quality problems. Data gathered in such projects may contain a greater absolute number of mistakes because of the people collecting data, the complexity of data processing, and the collation required. We wanted to learn from the types and frequencies of errors encroaching on data in a multicenter field trial, and to assess the effects of these errors had they passed through. We used extensive error trapping while processing 688 forms from seven sites in the field trial. Snapshots of the dataset were taken at several points in the process, before and after checking and correcting. We discovered 2.4% of the received data to be mistaken. These errors would have affected the data's reliability, decisions based on the study, and possibly the choice of analysis. Almost all of the mistakes were made at the time of measurement and may be related to raters' perceived importance of the variables. We found that communication and education effectively reduced the number of mistakes and their impact on the study over the course of the field trial. While an estimate of the overall error rate is important, the number of mistakes, in general, is only imperfectly related to the errors' effects on the study's results. Our results also suggest that statistical models that treat mistakes as simple independent events can be misleading.

References (32)

American Psychiatric Association
Diagnostic and statistical manual of mental disorders
(1980)
American Psychiatric Association
DSMIIIR: Diagnostic and statistical manual of mental disorders
(1987)
S. Arndt et al.
Establishing a Biostatistical Core Unit in a Clinical Research Center
The American Statistician
(1991)
S. Arndt et al.
Assessment of data quality: Errors of measurement and errors of process
Biometrical Journal
(1993)
A.J. Coale et al.
The case of the Indians and the teenage widows
Journal of the American Statistical Association
(1962)
J. Cohen
Statistical power analysis for the behavioral sciences
(1988)
W.E. Deming
Out of the crisis
(1986)
A.G. DuChene et al.
Forms control and error detection procedures used at the coordinating center of the multiple risk factor intervention trial (MRFIT)
Controlled Clinical Trials
(1986)
M. Flaum et al.
Diagnostic criteria for schizophrenia and related psychotic disorders
Schizophrenia Bulletin
(1992)
J.L. Fleiss
Statistical methods for rates and proportions
(1981)

J.L. Fleiss

The design and analysis of clinical experiments

(1986)

K.E. Freedland et al.

Data management and accountability behavioral and biomedical research

The American Psychologist

(1992)

H. Goldstein

Data processing for longitudinal studies

Applied Statistics

(1970)

H. Goldstein

The design and analysis of longitudinal studies

(1979)

L.L. Greenhill et al.

Multiple microcomputers for data entry and analysis

Psychopharmacology Bulletin

(1987)

W. Kruskal

Statistics in society: Problems unresolved and unformulated

Journal of the American Statistical Association

(1981)

Cited by (18)

Protocolization of multicenter clinical studies in the digital era. Is useful data centralization by a data-manager?
2023, Cirugia Espanola
En los estudios multicéntricos la protocolización de los datos es una fase crítica que puede generar sesgos, sobre todo en estudios clínicos con presupuesto limitado. El objetivo es analizar la concordancia y la confiabilidad de los datos obtenidos en un estudio multicéntrico clínico entre la protocolización del centro de origen y la protocolización centralizada mediante un data-manager.
Estudio clínico multicéntrico de prevalencia nacional sobre un carcinoma familiar infrecuente, realizándose una doble protocolización de los datos: a) en el centro de origen, y b) centralizada con un data-manager. La concordancia se analiza para el global de los datos y para los dos subgrupos del proyecto: a) grupo a estudio (carcinoma familiar; protocolizan 30 investigadores) y b) grupo control (carcinoma esporádico; protocolizan 4). Las diferencias interobservador se evalúan mediante el índice de Kappa de Cohen.
Se incluyen 689 pacientes: 252 del grupo a estudio y 437 del grupo control. Respecto al análisis de concordancia del estadio tumoral, se han objetivado un 2,5% de discordancias, siendo alta la concordancia entre protocolizadores (Kappa = 0,931). Respecto a la valoración del riesgo de recidiva, las discordancias fueron del 7% de los casos, siendo alta la concordancia (Kappa = 0,819). Respecto a la clasificación ecográfica TIRADS, las discordancias son del 6,9% y la concordancia es alta (Kappa = 0,922). Se han detectado un 4,6% de errores de transcripción.
En los estudios multicéntricos clínicos la protocolización centralizada de los datos por un data-manager parece presentar resultados similares a la protocolización directa en la base de datos en el centro de origen.
In multicenter studies, the protocolization of data is a critical phase that can generate biases. The objective is to analyze the concordance and reliability of the data obtained in a clinical multicenter study between the protocolization in the center of origin and the centralized protocolization of the data by a data-manager.
National multicenter clinical study about an infrequent carcinoma. A double protocolization of the data is carried out: (i) center of origin; and (ii) centralized by a data manager. The concordance between the data is analyzed for the global data and for the two groups of the project: (i) study group (familiar carcinoma, 30 researchers protocolize); (ii) control group (sporadic carcinoma, 4 people protocolize). Interobserver variability is evaluated using Cohen's kappa coefficient.
The study includes a total of 689 patients with carcinoma: 252 in the study group and 437 in the control group. Regarding the concordance analysis of the tumor stage, 2.5% of disagreements were observed and the concordance between people who protocolize was near perfect (Kappa = 0.931). Regarding the evaluation of the recurrence risk, disagreements occurred in 7% of the cases and the concordance was near perfect (Kappa = 0.819). Regarding the sonography evaluation (TIRADS), the disagreements were 6.9% and the concordance was near perfect (Kappa = 0.922). Also, 4.6% of transcription errors were detected.
In multicenter clinical studies, the centralized data protocolization by a data-manager seems to present similar results to the direct protocolization in the database in the center of origin.
A comparison of error detection rates between the reading aloud method and the double data entry method
2003, Controlled Clinical Trials
Citation Excerpt :
Data management is the sum of many processes of which the primary objective is to ensure timely delivery of high-quality data [1]. There are various sources of errors in every step of data management, and these errors may affect the results, reliability, and conclusions of a study [2–5]. Even though sufficient effort should be applied during all phases of data management to ensure the quality of the data, the more we pursue quality, the more its cost (time, resources, or money) increases.
Data entry and its verification are important steps in the process of data management in clinical studies. In Japan, a kind of visual comparison called the reading aloud (RA) method is often used as an alternative to or in addition to the double data entry (DDE) method. In a typical RA method, one operator reads previously keyed data aloud while looking at a printed sheet or computer screen, and another operator compares the voice with the corresponding data recorded on case report forms (CRFs) to confirm whether the data are the same. We compared the efficiency of the RA method with that of the DDE method in the data management system of the Japanese Registry of Renal Transplantation. Efficiency was evaluated in terms of error detection rate and expended time. Five hundred sixty CRFs were randomly allocated to two operators for single data entry. Two types of DDE and RA methods were performed. Single data entry errors were detected in 358 of 104,720 fields (per-field error rate = 0.34%). Error detection rates were 88.3% for the DDE method performed by a different operator, 69.0% for the DDE method performed by the same operator, 59.5% for the RA method performed by a different operator, and 39.9% for the RA method performed by the same operator. The differences in these rates were significant (p<0.001) between the two verification methods as well as between the types of operator (same or different). The total expended times were 74.8 hours for the DDE method and 57.9 hours for the RA method. These results suggest that in detecting errors of single data entry, the RA method is inferior to the DDE method, while its time cost is lower.
Landmark-based registration and measurement of magnetic resonance images: A reliability study
1996, Psychiatry Research - Neuroimaging
We report a magnetic resonance imaging (MRI) study in a large number of subjects that examines various analytic techniques for anatomical landmark information. The initial issue involved interrater reliability in locating landmarks along the midsagittal plane. Another issue dealt with using landmarks to register (linear) individual scans for subsequent analyses or more sophisticated registration techniques. We next explored the use of interlandmark distances as indices of brain size. Twenty-seven landmarks were chosen from the midsagittal slice of 101 normal subjects. Interrater reliability estimates on a subset of the scans were excellent overall. Rotating to anterior-posterior commissure points was the best of the tested two-point registrations, although an average angular rotation was better. Reliability and validity for landmark-derived measures of size were excellent. Landmark-based analyses offer opportunities to explore shape and size questions, although they are not appropriate for addressing all questions. Under specific conditions, landmark data can be generated quickly and accurately.
Extended Reality Environment for Urban Area Environmental Data Analysis
2023, Proceedings of SPIE - The International Society for Optical Engineering
Predicting numerical data entry errors by classifying EEG signals with linear discriminant analysis
2015, Behaviour and Information Technology
Verification of data completeness and accuracy
2015, Pediatric and Congenital Cardiac Care: Volume 1: Outcomes Analysis

View all citing articles on Scopus

^☆: This work was partially funded by NIMH MH46011, Grant # 5 P50 MH 43271, MH 40856, MH 31593 and MH 47200.

View full text

Effects of errors in a multicenter medical study: Preventing misinterpreted data☆

Abstract

Diagnostic and statistical manual of mental disorders

DSMIIIR: Diagnostic and statistical manual of mental disorders

Establishing a Biostatistical Core Unit in a Clinical Research Center

The American Statistician

Assessment of data quality: Errors of measurement and errors of process

Biometrical Journal

The case of the Indians and the teenage widows

Journal of the American Statistical Association

Statistical power analysis for the behavioral sciences

Out of the crisis

Forms control and error detection procedures used at the coordinating center of the multiple risk factor intervention trial (MRFIT)

Controlled Clinical Trials

Diagnostic criteria for schizophrenia and related psychotic disorders

Schizophrenia Bulletin

Statistical methods for rates and proportions

The design and analysis of clinical experiments

Data management and accountability behavioral and biomedical research

The American Psychologist

Data processing for longitudinal studies

Applied Statistics

The design and analysis of longitudinal studies

Multiple microcomputers for data entry and analysis

Psychopharmacology Bulletin

Statistics in society: Problems unresolved and unformulated

Journal of the American Statistical Association