Article Text
Abstract
Background To compare the clinical and demographic variables of patients who present to the ED at different times of the day in order to determine the nature and extent of potential selection bias inherent in convenience sampling
Methods We undertook a retrospective, observational study of data routinely collected in five EDs in 2019. Adult patients (aged ≥18 years) who presented with abdominal or chest pain, headache or dyspnoea were enrolled. For each patient group, the discharge diagnoses (primary outcome) of patients who presented during the day (08:00–15:59), evening (16:00-23:59), and night (00:00-07:59) were compared. Demographics, triage category and pain score, and initial vital signs were also compared.
Results 2500 patients were enrolled in each of the four patient groups. For patients with abdominal pain, the diagnoses differed significantly across the time periods (p<0.001) with greater proportions of unspecified/unknown cause diagnoses in the evening (47.4%) compared with the morning (41.7%). For patients with chest pain, heart rate differed (p<0.001) with a mean rate higher in the evening (80 beats/minute) than at night (76). For patients with headache, mean patient age differed (p=0.004) with a greater age in the daytime (46 years) than the evening (41). For patients with dyspnoea, discharge diagnoses differed (p<0.001). Asthma diagnoses were more common at night (12.6%) than during the daytime (7.5%). For patients with dyspnoea, there were also differences in gender distribution (p=0.003), age (p<0.001) and respiratory rates (p=0.003) across the time periods. For each patient group, the departure status differed across the time periods (p<0.001).
Conclusion Patients with abdominal or chest pain, headache or dyspnoea differ in a range of clinical and demographic variables depending upon their time of presentation. These differences may potentially introduce selection bias impacting upon the internal validity of a study if convenience sampling of patients is undertaken.
- methods
- emergency department
Data availability statement
Data are available on reasonable request. no further detail.
Statistics from Altmetric.com
Key messages
What is already known on this subject
Convenience sampling has been identified as a source of selection bias in a variety of scientific fields.
The extent and direction of convenience sampling in emergency medicine research has been poorly investigated and remains largely unknown.
What this study adds
In this retrospective study of ED patients, we found that patients with abdominal or chest pain, headache or dyspnoea differ in a range of clinical and demographic variables depending on their time of presentation.
Substantial selection bias may be introduced if convenience sampling of patients is undertaken, for example, during office hours.
Convenience sampling is best avoided or demonstrated to have negligible impact on a study’s outcome measures.
Introduction
Convenience sampling refers to a disparate group of non-random sampling methods which are convenient for the researcher.1 In the ED, convenience sampling is widespread2 and often involves the recruitment of patients during business hours (Monday–Friday, 08:00–18:00). It largely results from a relative paucity of research staff available to enrol patients outside of these hours.
Convenience sampling offers advantages. It is often undertaken when EDs can be busy which facilitates patient enrolment. It also maximises the use of available research resources, especially recruitment staff. Notwithstanding these advantages, convenience sampling has one important potential disadvantage. Patients who present during business hours may differ from patients who present at other times and may be unrepresentative of the ED population overall.2 For example, intoxicated patients are less common during the day. Hence, convenience sampling may be subject to selection bias with the potential for influencing a study’s findings. This, in turn, may adversely affect the study’s internal validity and usefulness in the provision of high-quality research evidence.
Convenience sampling has been identified as a source of selection bias in fields as diverse as neurology, primary care and even agriculture.3–9 While the sampling techniques described in these studies differ from the business-hours convenience sampling often used in ED research, each may result in an unrepresentative sample and selection bias.
There is a paucity of literature on convenience sampling in the ED. Consistent with research in other fields, we hypothesised that it may result in substantial selection bias. We aimed to compare the clinical and demographic variables of patients who present to the ED at different times of the day. The findings will determine the nature and extent of selection bias inherent in convenience sampling. This will inform future research project design aiming to mitigate or avoid this preventable bias.
Methods
Study design and setting
We undertook a multicentre, retrospective, observational study of data routinely collected in 2019 in five EDs in Melbourne, Australia (Austin, Box Hill, Maroondah, Angliss and St Vincent’s Hospitals). Four are mixed (adult/paediatric) metropolitan EDs and one is an inner-city, adult ED. Their annual censuses ranged from approximately 42 000 to 95 000 patients. Each ED has a short-stay unit that operates 24 hours a day, 7 days a week. Informed patient consent was not required. The study was unfunded.
Patient and public involvement
No patient or public contributed to the development of this study.
Participants
Patients were included if they were aged 18 years or more, presented to a participating ED between 1 January and 31 December 2019 (inclusive) and had a triage presenting complaint of abdominal pain, chest pain, headache or dyspnoea. These presenting complaints were chosen because they are common, often present diagnostic difficulties and may have serious aetiologies. There were no exclusion criteria.
For many patients, a specific medical diagnosis may not be evident during the ED stay. Hence, we included patients with presenting complaints of interest, not specific diagnoses. This is consistent with the methodology of common prospective emergency medicine studies. For example, a study may investigate whether all patients with acute dyspnoea should have an urgent lung scan.
At each of the five sites, triage nurses routinely assign each patient a presenting complaint code which may be one of the four investigated in this study. These codes allowed the identification of patients with the presenting complaints under investigation by screening the electronic patient log. At each site, four independent datasets were established, one for each presenting complaint. From each of these datasets, 500 patients were randomly selected using the Excel RANDOM function. The selected patients from each of the five sites were merged to yield datasets of 2500 patients for each presenting complaint.
Data collection
The following data items were extracted for each enrolled patient: ED arrival data (arrival date/time, arrival mode), patient characteristics (age, gender, ethnicity, marital status, religion, usual accommodation, preferred language, need for an interpreter) and clinical variables (triage category, triage presenting complaint code, first vital sign measurements, pain score, place of disposition and ICD10 discharge diagnosis code). All data were extracted electronically except for vital sign data at one site. The patients were then categorised according to their arrival time period: day (08:00–15:59), evening (16:00–23:59) and night (00:00–07:59).
Outcome measures
For each presenting complaint patient group, the primary outcome was discharge diagnosis for each of the three arrival time periods. The discharge diagnoses were determined from the assigned ICD10 codes. For each patient group, the five most common discharge diagnoses and an ‘other’ diagnosis group were analysed. Three of the presenting complaint groups (abdominal pain, chest pain and headache) had ‘unspecified’ and ‘unknown’ among their five most common discharge diagnoses. As these diagnostic groups were similar, they were merged prior to data analysis. Secondary outcomes were patient characteristics, clinical variables and departures status for each time period.
Statistical analyses
Within each patient group, we compared the data variables across the three time periods. We believed that a clinically significant absolute difference in proportions of a particular diagnostic category within two time periods would be 5% (eg, 5% and 10% of patients who present in the morning and evening, respectively, would have a particular discharge diagnosis). This 5% difference was also employed by Valley et al.2 Given these proportions, we required at least 718 patients in each of two time periods to have a power of 0.95 (alpha 0.05, two-sided). With three time periods, this amounted to 2154 patients. As we expected that some data items would be missing (underpowering some comparisons), the sample size was rounded up to 2500 for each presenting complaint group.
The results are presented descriptively as numbers (percentages (%)) and medians (IQR). The distributions of categorical and continuous variables were compared using the χ2 and Kruskal-Wallis tests, respectively. All data analysis was performed using SPSS for Windows (V.26.0; SPSS Inc., Chicago, IL, USA). Not all variables were examined statistically—less important ones are reported descriptively in the online supplemental file.
Supplemental material
Each presenting complaint group analysis was an independent investigation that used a separate dataset of patients. In each of these four investigations, 12 variables were statistically analysed. Given this, the level of significance was revised, using the Bonferroni adjustment, to 0.0042.
Results
For patients with abdominal pain, there was a difference in discharge diagnoses across the time periods (p<0.001, table 1). There were greater proportions of unspecified/unknown cause diagnoses in the evening and ‘other’ diagnoses during the day. There were also differences in median heart rate and temperature across the time periods with both lowest at night (p<0.001). The departure status differed across the time periods (p<0.001). A greater proportion of patients was discharged to the short-stay unit (SSU) during the day and greater proportions left before their treatment was complete in the evening and at night.
For patients with chest pain, the discharge diagnoses did not differ over the time periods (p=0.31, table 2). There were significant differences in median heart rate and respiratory rate across the time periods (p<0.001 and p=0.003, respectively). Both were lowest at night. The departure status also differed across the time periods (p<0.001). A smaller proportion of patients was discharged to home at night and greater proportions left before their treatment was complete in the evening and at night.
For patients with headache, there was no difference in discharge diagnoses across the time periods (p=0.70, table 3). Overall, the number of female patients was more than twice that of male patients (1687 vs 821, respectively). Females outnumbered males across all time periods, although this was not as pronounced at night. Patient age differed across the time periods (p=0.004) with the median age greater during the day. The departure status also differed across the time periods (p<0.001). Greater proportions of patients were discharged to home at night and admitted to the SSU during the day. Greater proportions left before their treatment was complete in the evening and at night.
For patients with dyspnoea, there was a difference in discharge diagnoses across the time periods (p<0.001, table 4). There were greater proportions of asthma diagnoses at night and ‘other’ diagnoses during the day. The lowest proportion of ‘unspecified cause’ diagnosis was during the day. The proportion of patients with orthopnoea was slightly greater at night. There was a difference in gender distribution across the time periods (p=0.003) with a greater proportion of females presenting during the evening. Patient age differed across the time periods (p<0.001) with the median age greater during the day. Although the median respiratory rates of each time period were the same, there was a significant difference across the time periods. The departure status differed across the time periods (p<0.001). The greatest proportion of patients admitted to an inpatient ward was during the day and the smallest proportion of patients admitted to SSU was during the night. Greater proportions left before their treatment was complete in the evening and at night.
Discussion
This study demonstrates some significant differences in discharge diagnoses, patient characteristics, vital signs and departure status across the three time periods. The differences varied across the presenting complaint groups and may be associated with the nature of the underlying conditions. However, other variables may have contributed including access to diagnostic imaging and the availability of community support. Notwithstanding these other variables, the findings suggest that if patient enrolment were to occur only during one time period (eg, office hours), those enrolled may be unrepresentative of all patients with one of the four presenting complaints and that the study findings may be biased.
Our findings are consistent with those of Valley et al.2 In a single-centre, retrospective ED study, they reported that ‘business hours’ patient samples differed significantly from truly representative samples in terms of age, language, triage acuity, disposition and mode of arrival. Our findings are also consistent with the growing body of evidence from other settings where convenience sampling is associated with selection bias.3 6 7 10–13
Although we observed significant differences in outcome measures, the absolute differences are of questionable clinical significance. This is particularly evident for the vital sign comparisons. In these cases, the small but significant differences are likely attributable to variations in data distributions (eg, skewness). The differences in data distribution may be due to particular diagnostic conditions (within the presenting complaint groups) more often presenting at a particular time of the day. For example, greater proportions of patients with orthopnoea presented at night. This condition is often associated with severe tachypnoea which may account for the differences observed in the respiratory rates over the time periods. These findings are also consistent with those of Valley et al2 who reported significant but numerically small differences between their sampled groups.
Overall, the results indicate directional bias (a clear difference in outcome variables in one particular time period) for a range of the variables examined. For example, with the exception of patients presenting with abdominal pain, patients presenting during the day were significantly older. Valley et al2 also found significant directional bias for almost all of the variables they examined. As this study was not designed to explain the observed differences in outcome measures, we have deliberately avoided speculation as to the reasons.
Clinical implications of convenience sampling
The clinical implications of convenience sampling bias may depend on the research question and the study outcome measures. If the bias is small or not related to the primary outcome, its effect on the conclusions may be negligible. Alternatively, if it impacts significantly on the primary outcome, the internal validity of the study and accuracy of the conclusions may be compromised.2 This may lessen the chances of emergency medicine (EM) research being published in general high-impact journals.
Our findings suggest that, where possible, consecutive sampling, over all hours of the day, should be undertaken. However, this is largely precluded due to research resource limitations. Furthermore, community hospitals, with fewer resources than larger metropolitan centres, may be less represented in EM research. This, in turn, is likely to introduce selection bias and diminish external validity.
Consideration needs to be given to the nature, extent and direction of the bias. If these are not known, an examination of important outcome data stored in the electronic medical record databases or other sources could be undertaken.2 Alternatively, a pilot study with prospective recruitment over all times of the day could be considered. If there are no clear differences across the time periods, selection bias may not be an issue.
Mitigating the effects of convenience sampling selection bias
Recognition of the extent of the bias may allow general comments to be made that will place a study’s findings in perspective. For example, “Our convenience sampling technique likely recruited a sample of patients that is older than the overall ED population”. However, if the bias is perceived or known to be substantial, a redesign of the study methodology may be required.
Convenience sampling bias may be mitigated by using statistical weighting. Methods such as inverse-probability censor weighting and propensity score matching provide a cost-effective means of correcting bias in convenience and other samples.14–16 The effective use of these techniques relies on the identification of the specific variables for which convenience samples are unrepresentative. It has been recommended that researchers should consider applying statistical weighting to their convenience samples.17 18 The Heckman selection model, an econometric method, has also been used to correct biased population estimates derived from non-random samples.19
Our study has several strengths. The sample sizes were large and patients were randomly selected from all patients who attended during the study periods. Almost all data were available electronically which helped avoid manual data extraction error. The external validity of the study was strengthened by the multicentre study design.
However, the study has important limitations. We used a random sampling technique to select our patients. Although a well-recognised technique, we cannot be sure that our sampled patients were truly representative of all patients with the presenting complaints of interest. Some patient data items were missing which effectively decreased the sample size and precision for some comparisons. However, the study had more power than is routinely used. As fewer patients presented after business hours (especially at night), the time period groups had differing patient numbers. However, a post hoc power calculation indicated that the study had a power of 0.93 to demonstrate a difference between groups (proportions in the two groups: 5% and 10%; patients in the two groups: 1000 and 500; alpha 0.05, two-sided). The relevance of some significant differences across the time periods may not be clinically meaningful. Researchers need to determine if differences observed across time periods are of clinical relevance and likely to impact on their research conclusions. The sample size was based on categorical data—a difference of 5% in proportions for the primary outcome measure. No a priori sample size calculation was undertaken for the secondary outcomes, some of which were continuous data. However, the large sample size likely adequately powered these comparisons.
Only four presenting complaint groups were chosen. Hence, the findings are not relevant to other complaints or studies of specific diagnoses. The study did not explore the potential impact of the observed bias. Others have done so by regressing differences in demographic and clinical variables in a sample against an outcome variable.14 19 However, the results of these studies were mixed because the impact of unrepresentativeness is predicated on the outcome of interest.
The aim of this study was to explore patient differences across the time periods. It is recommended that further research be undertaken to further explore the differences observed, especially for the primary outcome measure of discharge diagnosis. It is possible that some patient diagnoses were incorrect and this may have biased our findings. Furthermore, if the ultimate diagnoses of the patients in the ‘unspecified/unknown cause’ diagnosis subgroups could be determined, the apparent directional bias may be either confirmed or negated.
Arguably, an elegant way of exploring bias associated with convenience sampling would be to undertake three identical studies, each with the same research question. One study would recruit patients during the day, another in the evening and the other at night. The outcomes could be compared with determine if the recruitment period impacted on the answer to the research question. However, being time and resource intensive, such investigations may not be justifiable.
In summary, patients with abdominal or chest pain, headache or dyspnoea differ in a range of clinical and demographic variables depending on whether they present during the day, evening or at night. If convenience sampling of patients is undertaken only during one time period, these differences may introduce selection bias and impact on the internal validity of a study. Convenience sampling should be avoided if possible or demonstrated not to have a significant impact on a study’s primary outcome measure.
Data availability statement
Data are available on reasonable request. no further detail.
Ethics statements
Patient consent for publication
Ethics approval
The study was approved by the Austin Hospital Human Research Ethics Committee.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Handling editor Richard Body
Contributors DMT and TL conceived the study. DMT, TL, HA and BP developed the study protocol and obtained ethics committee approval. TL, CB, XD, EA, TN-W and KW collected the data. DMT, HA and BP supervised the data collection. DMT collated, cleaned and undertook the data analysis. DMT, TL, HA and BP interpreted the results. All authors contributed to writing the manuscript and approved the final, submitted manuscript. DT is the author acting as guarantor.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.