You are here

Article Text

Article menu

Download PDFPDF

Prehospital care
Understanding the value of mixed methods research: the Children's Safety Initiative-Emergency Medical Services
  1. Matthew Hansen1,
  2. Kerth O'Brien2,
  3. Garth Meckler3,
  4. Anna Marie Chang1,
  5. Jeanne-Marie Guise1,4,5
  1. 1Department of Emergency Medicine, Oregon Health & Science University, Portland, Oregon, USA
  2. 2Department of Psychology, Portland State University, Portland, Oregon, USA
  3. 3Department of Pediatrics, University of British Columbia, Vancouver, British Columbia, Canada
  4. 4Department of Obstetrics and Gynecology, Oregon Health & Science University, Portland, Oregon, USA
  5. 5Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon, USA
  1. Correspondence to Dr Matthew Hansen, Department of Emergency Medicine, Oregon Health & Science University, 3181 SW Sam Jackson Pk Rd, Mailcode CR 114, Portland, OR 97239, USA; hansemat{at}ohsu.edu

Abstract

Mixed methods research has significant potential to broaden the scope of emergency care and specifically emergency medical services investigation. Mixed methods studies involve the coordinated use of qualitative and quantitative research approaches to gain a fuller understanding of practice. By combining what is learnt from multiple methods, these approaches can help to characterise complex healthcare systems, identify the mechanisms of complex problems such as medical errors and understand aspects of human interaction such as communication, behaviour and team performance. Mixed methods approaches may be particularly useful for out-of-hospital care researchers because care is provided in complex systems where equipment, interpersonal interactions, societal norms, environment and other factors influence patient outcomes. The overall objectives of this paper are to (1) introduce the fundamental concepts and approaches of mixed methods research and (2) describe the interrelation and complementary features of the quantitative and qualitative components of mixed methods studies using specific examples from the Children's Safety Initiative-Emergency Medical Services (CSI-EMS), a large National Institutes of Health-funded research project conducted in the USA.

  • emergency ambulance systems
  • methods
  • prehospital care
  • research, methods
  • qualitative research

https://doi.org/10.1136/emermed-2015-205277

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Key messages

    What is already known on this subject?

    • Mixed methods research offers promising techniques to answer new and challenging questions in emergency medical services (EMS) research, though mixed methods studies are not commonly reported in the EMS literature.

    What might this study add?

    • This study provides an overview of the benefits of mixed methods research along with specific examples of mixed methods studies in the context of a large National Institutes of Health-funded study of errors in out-of-hospital paediatric care.

    Introduction

    Mixed methods research involves the coordinated use of qualitative and quantitative research approaches to gain a fuller understanding of practice. As Creswell et al1 define mixed methods, this approach ‘involves the collection or analysis of both quantitative and/or qualitative data in a single study in which the data are collected concurrently or sequentially, are given a priority, and involve the integration of data at one or more stages in the process of research.’ Quantitative approaches generate statistical and numerical data and include case–control, cross-sectional, cohort and clinical trials, whereas qualitative approaches generally yield non-numeric or verbal data and include focus groups, interviews and case studies. Mixed methods research brings these two lines of investigation together to provide complementary data. Mixed methods research has been commonly used in biomedical research in nursing and social science fields, but is well suited to help out-of-hospital care researchers understand complex systems, generate new hypotheses and deepen their understanding of quantitative results.

    There are several existing resources where investigators can learn general concepts and best practice in mixed methods research as well as articles from the emergency care/acute care medicine literature, textbooks and a guide to mixed methods research published by the National Institutes of Health Office of Behavioral and Social Sciences Research.2–5 An article from 2003 identifies some of the benefits of mixed methods research for emergency care researchers.6 However, despite resources on methods, a systematic review by McManamny et al7 of mixed methods research in out-of-hospital medicine found only 23 research publications in total, the majority of which were surveys with a small number of studies, including focus groups, observation and medical record reviews. This systematic review illustrates that there is a gap between available methodological guidance and published primary research. In this paper, we seek to bridge the gap by building on existing literature that describes fundamental concepts of qualitative and mixed methods studies2 ,4 by providing specific examples from a large National Institutes of Health (NIH)-funded project. The specific study example will be the Children's Safety Initiative-Emergency Medical Services (CSI-EMS), which is a large NIH (NICHD R01HD062478)-funded mixed methods study on identification and prevention of errors in out-of-hospital paediatric care in the USA with several components that are relevant for emergency medical services (EMS) researchers (figure 1).8 The study was conducted in order to address the paucity of literature on safety in out-of-hospital paediatric care. The specific methods included obtaining provider viewpoints on patient safety using focus groups and a Delphi study followed by a chart review and high-fidelity simulation designed to describe the epidemiology of safety events (figure 1).

    Figure 1
    Figure 1

    Components of the Children's Safety Initiative-Emergency Medical Services Research Project. The type and contribution of the particular study to the mixed methods design are indicated using the rubric described in table 1.

    Introduction to mixed methods research

    Mixed methods research combines quantitative and qualitative approaches. Some typical examples of both quantitative and qualitative methods are included in table 1.

    View this table:
    Table 1

    Examples of qualitative and quantitative research methods

    Quantitative research generally relies on a pool of countable data to reach conclusions about hypotheses using deductive reasoning; where qualitative investigations ‘focus on the contexts and meaning of human lives and experiences for the purpose of inductive or theory-development driven research’.5 Mixed methods research offers the ability to combine both deductive and inductive processes to harness the strengths of both to deepen understanding on a particular topic. For example, CSI-EMS investigators explored previously unknown threats to out-of-hospital paediatric patient safety through focus groups, reflected on these findings in light of previous data from hospital-based safety studies and developed specific hypotheses such as lack of appropriately sized equipment (inductive reasoning). Subsequently, CSI-EMS investigators evaluated these hypotheses quantitatively through chart review (deductive reasoning).

    Mixed methods studies may be particularly relevant given the increasing complexity of healthcare, and can help identify the mechanisms of complex problems such as medical errors, and understand aspects of human interaction, including communication, behaviour and team performance.9–11 For example, an investigator who wants to understand the rate of medication dosing errors in EMS could perform a single methods study. Adding a qualitative component could help identify factors that contribute to medication dosing errors such as labelling of medication bottles.

    Qualitative methods are sometimes described as either exploratory or explanatory.3 In exploratory studies, investigators seek information to aid in hypotheses generation on a topic where little previous data exist. In explanatory studies, investigators may wish to interpret the results of a quantitative study and gain a deeper understanding of study results. Investigators who seek to understand how a complex system, human interactions or psychosocial factors affect the study results may benefit from adding a qualitative component to their study. In the CSI-EMS, we used high-fidelity simulation with quantitative analysis of team performance and qualitative analysis of debriefing to gain insight into the potential aetiology of medical errors identified through a chart review.

    Hypothesis generation

    Mixed methods studies may be particularly useful for EMS researchers studying topics that lack sufficient information to create an informed hypothesis. In CSI-EMS, there were little previous data on potential threats to patient safety in paediatric EMS and mixed methods were used to begin to understand this complex area—specifically focus groups were used as a starting point.

    Focus groups

    Focus groups overview

    A focus group is a guided discussion facilitated by a trained moderator that is conducted to explore the viewpoints, experiences and perspectives of participants. In mixed methods research, focus groups may help investigators determine how members of a study population think and talk about a given problem. These insights often generate data that inform subsequent investigation.12 Focus groups are often composed of relatively homogeneous groups of participants with specific characteristics, such as gender, ethnicity or job status similarities so that participants will be comfortable interacting with each other.13 The moderator is a critical component of a successful focus group, delivering questions, soliciting a range of experiences and perspectives, guiding discussion and ensuring that no single participant dominates. In focus group analysis, the transcripts from the sessions are reviewed and statements are systematically coded into themes and subthemes. The number of focus groups needed is based on the concept of achieving thematic saturation, where group participants no longer introduce new or noteworthy themes.

    Focus group study example

    The CSI-EMS was developed in part because there was little previous knowledge regarding specific contributors to paediatric safety events. The investigative team planned to use a national Delphi survey to address this (see below for details of Delphi methodology), but did not have specific hypotheses based on the literature to inform the initial survey development. To address the paucity of data, we used focus groups to generate hypotheses and to develop preliminary questions for the larger Delphi survey. We segmented the focus groups in order to separate fire department providers, private transport agency providers and rural providers from each other, on the basis that providers working in these various settings have potentially different needs and perspectives on paediatric patient safety. Selecting participants for focus groups, and other qualitative studies, is different than in quantitative studies. In qualitative studies, investigators are not seeking to describe insights that are representative of an entire population, but are seeking to obtain insight into a particular topic and groups should be selected purposefully rather than randomly. Focus groups were advantageous for this study because they allowed findings to emerge directly from the data obtained, rather than from tests of pre-existing hypotheses.14

    Based on previous studies showing that focus group thematic saturation is usually reached after 4–6 groups, CSI-EMS conducted five focus groups and at that point reached saturation.15 Prior to the first focus group, the research team created a discussion guide with open-ended questions to prompt discussion. In research situations involving subject matter expertise, such as CSI-EMS, two moderators will sometimes cofacilitate discussion, one with more expertise in moderating and the other more familiar with language and habits of discussion participants. In CSI-EMS, an experienced moderator was selected from the medical centre and another from the department of psychology of a local university. For each focus group, one of these experienced moderators paired with a paramedic who was a member of our research team to comoderate the group. We believed the presence of EMS staff would allow discussion group participants to feel comfortable with the moderators and would allow participants to use EMS jargon, while the presence of experienced non-EMS moderators would lessen the likelihood of biasing discussion towards researchers’ expectations.

    After coding by trained research team members, themes were reviewed and interpreted by the interdisciplinary research team across several sessions. The focus groups identified several domains of potential threats to patient safety, including lack of appropriately sized equipment, lack of standard medication doses, limited training, communication problems, difficulty in assessment and anxiety when caring for children. These were carried forward to the next phase of the study and were incorporated into the Delphi survey, which enabled quantitative evaluation of these domains. Table 2 describes the interaction of the qualitative and quantitative components of each element of CSI-EMS.

    View this table:
    Table 2

    Mixed methods study design examples, using notation by Morse26

    Delphi

    Delphi method overview

    The Delphi method entails several survey rounds performed on the same group of respondents in order to reach consensus on a topic.16 ,17 The first draft of a Delphi survey is based on prior hypotheses from the investigators, pilot data, results of qualitative research or a mixture of these. Successive rounds of the survey are refined based on the responses from previous rounds to gain greater insight into the responses and gain consensus. Another common method for consensus building is the nominal group technique often referred to as an expert panel.

    Delphi study example

    In CSI-EMS, we sought to find consensus among experienced EMS professionals regarding the most likely contributors to patient safety events in the out-of-hospital care of children.18 The Delphi method in this case followed the focus groups in a qual→QUAN mixed methods format. As part of questionnaire development, interviews were conducted with seven EMS professionals (who were not part of the focus groups or research team): three emergency medical technicians (EMTs), two physician/medical personnel and two children's EMS director/program coordinators. Interviewees gave input on wording, length and methods to encourage participation. The research team then finalised the Delphi round one questionnaire and carried out the design and refinement of each Delphi round (figure 2). Our interdisciplinary team included a social psychologist with expertise in survey methods and qualitative design. After each round of the Delphi survey, our team met to review and interpret the data over several sessions and prepare the next round of the survey. The Delphi method itself also incorporated a small qualitative component in the form of open-ended questions that served to provide context for the Likert-type questions, which our team used in refining each subsequent Delphi round.

    Figure 2
    Figure 2

    Progression of nationwide Delphi survey. EMS, emergency medical services.

    In responding to the Delphi, various groups of EMS providers—physicians, EMTs and so on—sometimes disagreed with other provider groups. In order to resolve such conflicts, we assigned more weight to the responses from groups with more experience in a given survey domain. For example, EMT paramedics’ responses were accorded higher weight than those of EMT basics in areas such as airway management where the EMT basic has a limited scope of practice. An interdisciplinary research team with people of various professional backgrounds, including qualitative methods, epidemiology, biostatistics and clinical medicine, can be helpful when interpreting survey results and preparing the next Delphi round.

    Medical record review

    Medical record review in mixed methods studies

    Best practices for designing medical record review studies have been described elsewhere.19 Medical record reviews themselves can incorporate mixed methods components often working in a QUAN+qual framework, with a major quantitative element and a concurrent smaller qualitative element. For example, reviewers may abstract objective data from the chart such as the number of attempts taken to successfully intubate a patient. In addition, reviewers could be asked to describe their overall impression about the nature or cause of an event that took place in the medical record, such as why the intubation took several attempts. Mixed methods studies also have potential benefit to investigators preparing to develop novel medical record assessment tools, with qualitative methods or surveys being used to inform the initial tool development.

    Medical record review example

    The objective of the medical record review in CSI-EMS was to have experts identify potential errors based on the medical record. The project developed and tested an out-of-hospital chart review tool, modelled after the landmark Harvard Medical Practice and Utah, Colorado studies.20 ,21 The structure and design of the review tool were based on the results of both the focus groups and the Delphi study, and domains included assessment and clinical decision making, medication, equipment and procedures. In CSI-EMS, a paramedic and an emergency physician reviewed approximately 500 paediatric charts that were high risk for significant errors as they were all patients transported with lights and sirens priority. Differences in reviews were arbitrated by one of two ‘gold standard’ reviewers who are paediatric emergency physicians with expertise in out-of-hospital paediatric care. CSI-EMS medical record review used a QUAN+qual framework (table 2). The intention of the smaller, concurrent qualitative component was to gain greater depth into the potential causes of safety events identified in the review.

    Simulation

    Simulation as a research tool

    Medical simulation is becoming a widely accepted part of medical education. Simulation has also been used for research. Much simulation research has focused on the efficacy of simulation as an educational or quality improvement tool; however, simulation can also help researchers to understand the practice or interaction among healthcare providers. Most aspects of assessment in simulation research would be considered quantitative and simulation researchers have developed validated performance assessment tools.22 Simulation can also have prominent, concurrent qualitative components through the analysis of debriefing, which is a structured but open-ended process. Simulation may be uniquely suited to accompany qualitative studies in mixed methods research. Qualitative components of simulation debriefing may help explain complex human interaction or systems of care in a QUAN+QUAL framework (table 2).

    Simulation study example

    Simulation complemented the other methods used in CSI-EMS, allowing the testing of hypotheses that were generated by earlier study components but that would have been difficult to assess using other methods. Through the focus group, Delphi and chart review processes, we identified hypotheses regarding the relative importance, frequency and potential mechanisms of errors and developed representative high-risk cases for in situ simulation. Two specific hypotheses we wished to test through simulation were that: (1) heightened anxiety increases the likelihood of errors and (2) younger patient age increases the likelihood of errors. In our study, simulation scenarios were developed by simulation experts on our research team and were refined after in situ pilot testing. We planned quantitative analysis of the scenarios and qualitative analysis of the debriefing in a QUAN+QUAL framework, both because the evaluation of scenario performance will happen via video review after the scenarios have all been completed, and because this review will not directly affect the debriefing process.

    Additional considerations

    Sample size in mixed methods studies

    Mixed methods investigators need to consider the sample size of the quantitative and qualitative components of their studies to determine if they will have enough participants and sufficient budget for all parts of the study. For the quantitative component, samples must be large enough to detect effects while limiting the likelihood of statistical error. In qualitative research, samples must be large enough to achieve thematic saturation. The quality of the data, the breadth of the topic, the nature of the topic and the ability of study participants to address the topic influence the ability to attain saturation.23

    Timing, weight and mixing

    In order for investigators to conduct a rigorous mixed methods study, particular attention needs to be paid to the weight, timing and mixing of the study elements. The timing refers to whether or not the quantitative and qualitative elements occur simultaneously or sequentially. Sequential designs are well suited when one method directly informs the other such as using focus groups to deepen the understanding of the results of a quantitative study. Simultaneous data collection would be well suited for studies where the results between qualitative and quantitative elements are being directly compared. The relative weights of the study elements are also important and in many studies one approach will be dominant, though in some studies the weights of the elements may be equal. Investigators will choose the dominant method based on the strengths and limitations of the method as well as the overall study objectives. Finally, investigators need to determine how the data from the various methods will be mixed. Typical frameworks include: (1) integration or merging the data when presenting or interpreting, (2) embedding one type of data within the other and (3) connecting the data such as the case when one method leads to the other. Table 2, previously referred to above, displays a common method to describe the weight and timing of elements in a mixed methods study.3

    Analysis

    Mixed methods studies require additional analytical expertise compared with single methods studies. The qualitative component of a mixed methods study often produces large quantities of data. The raw data are often text-based transcriptions of focus groups or interviews, or answers to other open-ended questions. Analysis of qualitative work often involves taking the text-based data and identifying specific themes within the data.24 Typically, at least two analysts from the team independently review the data and then meet to develop consensus. Finally, qualitative analysis takes place on a different timeline than quantitative analysis. In quantitative research, the instrument development is often labour intensive; in qualitative work, by contrast, time spent on analysis can be many times longer than time spent collecting data.

    Methodological appraisal and limitations

    Methods have been described to concomitantly grade the methodological quality of various elements of mixed methods studies and one common validated tool is the Mixed Methods Appraisal Tool (MMAT).25 Mixed methods research is subject to several limitations. Individual investigators may not have the skill set required to complete all aspects of the study. In addition, compared with single methods studies, mixed methods studies may be more costly and take longer to complete. Finally, as in all studies, mixed methods studies are subject to bias, though the sources of bias may be different and include more substantial bias from the investigative team members themselves.

    Conclusions

    EMS researchers are ultimately trying to improve care by uncovering truths in the midst of increasing complexity in the healthcare system. The interplay of qualitative and quantitative methods can enhance research where investigators desire to understand ‘the contexts and meaning of human lives and experiences’.4 Mixed methods approaches may be particularly well suited to EMS research due to complex systems where equipment, interpersonal interactions, societal norms, environment and other factors influence patient outcomes. By carefully planning the integration of study components, investigators will be able to answer new and challenging questions and ultimately improve care for patients in emergencies.

    References

    1. ↵
      1. Creswell J,
      2. Plano Clark V,
      3. Gutmann N, et al
      . Advanced mixed methods research designs. In: Handbook of mixed methods in social and behavioral research. Thousand Oaks, CA: Sage, 2003:209–40.
    2. ↵
      1. Cooper S,
      2. Porter J,
      3. Endacott R
      . Mixed methods research: a design for emergency care research? Emerg Med J 2011;28:682–5. doi:10.1136/emj.2010.096321
      OpenUrlAbstract/FREE Full Text
    3. ↵
      1. Creswell JW,
      2. Plano Clark VL
      . Designing and conducting mixed methods research. 2nd edn. Thousand Oaks, CA: Sage, 2007.
    4. ↵
      1. Curry LA,
      2. Nembhard IM,
      3. Bradley EH
      . Qualitative and mixed methods provide unique contributions to outcomes research. Circulation 2009;119:1442–52. doi:10.1161/CIRCULATIONAHA.107.742775
      OpenUrlFREE Full Text
    5. ↵
      NIH Office of Behavioral and Social Sciences Research (OBSSR)—Mixed Methods Research Index. http://obssr.od.nih.gov/mixed_methods_research/ (accessed 25 Apr 2015).
    6. ↵
      1. Goodacre S
      . Research methods: beyond the clinical trial. Ann Emerg Med 2003;42:56–65. doi:10.1067/mem.2003.235
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. McManamny T,
      2. Sheen J,
      3. Boyd L, et al
      . Mixed methods and its application in prehospital research a systematic review. J Mix Methods Res 2014;1558689813520408. doi:10.1177/1558689813520408
    8. ↵
      1. Hansen M,
      2. Meckler G,
      3. O'Brien K, et al
      . Pediatric Airway Management and Prehospital Patient Safety: Results of a National Delphi Survey by the Children's Safety Initiative-Emergency Medical Services for Children. Pediatr Emerg Care (in press).
    9. ↵
      1. Eccles M,
      2. Grimshaw J,
      3. Campbell M, et al
      . Research designs for studies evaluating the effectiveness of change and improvement strategies. Qual Saf Health Care 2003;12:47–52. doi:10.1136/qhc.12.1.47
      OpenUrlAbstract/FREE Full Text
    10. ↵
      1. Green J,
      2. Britten N
      . Qualitative research and evidence based medicine. BMJ 1998;316:1230–2. doi:10.1136/bmj.316.7139.1230
      OpenUrlFREE Full Text
    11. ↵
      1. Greenhalgh T
      . Integrating qualitative research into evidence based practice. Endocrinol Metab Clin North Am 2002;31:583–601, ix. doi:10.1016/S0889-8529(02)00009-9
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      1. Cottrell EK,
      2. O'Brien K,
      3. Curry M, et al
      . Understanding safety in prehospital emergency medical services for children. Prehosp Emerg Care 2014;18:350–8. doi:10.3109/10903127.2013.869640
      OpenUrlCrossRefPubMed
    13. ↵
      1. Kitzinger J
      . Qualitative research: introducing focus groups. BMJ 1995;311: 299–302. doi:10.1136/bmj.311.7000.299
      OpenUrlFREE Full Text
    14. ↵
      1. Corbin JM,
      2. Strauss A
      . Grounded theory research: procedures, canons, and evaluative criteria. Qual Sociol 1990;13:3–21. doi:10.1007/BF00988593
      OpenUrlCrossRef
    15. ↵
      1. Morgan DL
      . Focus groups. Annu Rev Sociol 1996;22:129–52. doi:10.1146/annurev.soc.22.1.129
      OpenUrlCrossRefWeb of Science
    16. ↵
      1. Dalkey NC
      . The Delphi method: an experimental study of group opinion. Santa Monica: Rand Corporation, Distributed by Clearinghouse for Federal Scientific and Technical Information, U.S. Dept. of Commerce, National Bureau of Standards, Institute for Applied Technology, 1969.
    17. ↵
      1. Lewis-Beck M,
      2. Bryman A,
      3. Liao T
      . Encyclopedia of social science research methods. Thousand Oaks, California, USA: SAGE Publications, Inc., 2004. http://knowledge.sagepub.com/view/socialscience/SAGE.xml (accessed 23 Jun 2015).
    18. ↵
      1. Hansen M,
      2. Meckler G,
      3. Dickinson C, et al
      . Children's safety initiative: a national assessment of pediatric educational needs among emergency medical services providers. Prehospital Emerg Care 2015;19:287–91. doi:10.3109/10903127.2014.959223
      OpenUrlCrossRef
    19. ↵
      1. Worster A,
      2. Haines T
      . Advanced statistics: understanding medical record review (MRR) studies. Acad Emerg Med 2004;11:187–92. doi:10.1111/j.1553-2712.2004.tb01433.x
      OpenUrlCrossRefPubMedWeb of Science
    20. ↵
      1. Brennan TA,
      2. Leape LL,
      3. Laird NM, et al
      . Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med 1991;324:370–6. doi:10.1056/NEJM199102073240604
      OpenUrlCrossRefPubMedWeb of Science
    21. ↵
      1. Leape LL,
      2. Brennan TA,
      3. Laird N, et al
      . The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med 1991;324:377–84. doi:10.1056/NEJM199102073240605
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Guise JM,
      2. Deering SH,
      3. Kanki BG, et al
      . Validation of a tool to measure and promote clinical teamwork. Simul Healthc 2008;3:217–23. doi:10.1097/SIH.0b013e31816fdd0a
      OpenUrlCrossRefPubMed
    23. ↵
      1. Morse JM
      . Determining sample size. Qual Health Res 2000;10:3–5. doi:10.1177/104973200129118183
      OpenUrlFREE Full Text
    24. ↵
      1. Pope C,
      2. Ziebland S,
      3. Mays N
      . Qualitative research in health care: analysing qualitative data. BMJ 2000;320:114. doi:10.1136/bmj.320.7227.114
      OpenUrlFREE Full Text
    25. ↵
      1. Pluye P,
      2. Hong QN
      . Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health 2014;35:29–45. doi:10.1146/annurev-publhealth-032013-182440
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      1. Morse JM
      . Handbook of Mixed Methods in Social & Behavioral Research Editors: Tashakkori and Teddlie. 2003. Chapter 7 Principles of Mixed Methods and Multimethod Research Design. Sage Publications. 2003.

    Footnotes

    • Contributors All of the authors have made substantial contributions in (1) the conception and design of the study, acquisition of the data and/or analysis and interpretation of the data, (2) drafting and revising the article for important content and (3) finally approving the submitted version. J-MG, KO and GM conceived the study, obtained research funding for the project, designed the focus groups and surveys and provided input on the Delphi analysis plan. J-MG, GM and MH contributed to design and analysis of the medical record review and simulation portions of the study. MH, KO, GM, AMC and J-MG contributed to writing, editing and critical content review of the manuscript. MH wrote the first draft and takes responsibility for the paper as a whole.

    • Funding This work is supported by the National Heart Lung and Blood Institute (NHLBI) grant number 5K12HL108974-03. This work is also supported by the National Institute of Child Health and Human Development grant: ‘Epidemiology of Preventable Safety Events in Pre-hospital EMS of Children’, grant number 1R01HD062478-04.

    • Competing interests None declared.

    • Ethics approval Oregon Health & Science University institutional review board approved all study components (IRB Number 00006942).

    • Provenance and peer review Not commissioned; externally peer reviewed.

    Linked Articles