Article Text
PDF
Abstract
The objective of this commentary is to develop a framework for assessing the rigour of qualitative approaches that identifies and distinguishes between the diverse objectives of qualitative health research, guided by a narrative review of the published literature on qualitative guidelines and standards from peer-reviewed journals and national funding organisations that support health services research, patient-centered outcomes research and other applied health research fields. In this framework, we identify and distinguish three objectives of qualitative studies in applied health research: exploratory, descriptive and comparative. For each objective, we propose methodological standards that may be used to assess and improve rigour across all study phases—from design to reporting. Similar to hierarchies of quality of evidence within quantitative studies, we argue that standards for qualitative rigour differ, appropriately, for studies with different objectives and should be evaluated as such. Distinguishing between different objectives of qualitative health research improves the ability to appreciate variation in qualitative studies and to develop appropriate evaluations of the rigour and success of qualitative studies in meeting their stated objectives. Researchers, funders and journal editors should consider how further developing and adopting the framework for assessing qualitative rigour outlined here may advance the rigour and potential impact of this important mode of inquiry.
- qualitative research
- patient-centered outcomes research
- health services research
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Article summary
Qualitative research in health services research (HSR) and allied fields has maintained steady, yet unsettled, interest and value over recent decades.
Qualitative methods are epistemologically and theoretically diverse, which is a strength. However, it also means that investigators do not necessarily approach qualitative research using a unified set of evidentiary rules. As such, assessing rigour and quality across studies can be challenging.
To help address these challenges, we propose a framework for assessing the rigour of qualitative approaches that identifies and distinguishes between three diverse study objectives. For each type of study, we propose preliminary methodological considerations to help improve rigour across all study phases. As is the case for quantitative studies, we argue that standards for qualitative rigourdiffer, appropriately, for different kinds of studies.
The objective of this commentary is not to resolve all potential conflicts between philosophical assumptions of different qualitative approaches, but rather help to advance a broader and richer understanding of qualitative rigour in relationship to other evidence hierarchies.
In recent decades, the role of qualitative research in health services research (HSR) and allied fields has maintained steady, yet unsettled, interest and value. Evidence of steady interest includes publication of qualitative reviews and guidelines by leading journals including Health Services Research,1 2Medical Care Research and Review3–5 and BMJ,6 7 and by funders including the Robert Wood Johnson Foundation,8 National Institutes of Health9 10 and National Science Foundation.11 12 In fields such as Patient-Centered Outcomes Research (PCOR) and implementation science, qualitative research has been embraced with particular enthusiasm for its ability to capture, advance and address questions meaningful to patients, clinicians and other healthcare system stakeholders.2 13 The majority (82%) of inaugural projects awarded by the Patient-Centered Outcomes Research Institute (PCORI) incorporated qualitative research methods.13 More recently, reflective of the continued prevalence of these approaches in the field, PCORI incorporated qualitative methods into their methodological standards.
Yet, despite this sustained interest, the status of qualitative health research remains unsettled, as illustrated by the BMJ’s changing engagement with the method. After championing qualitative methods in 2008,7 14–17BMJ editors in 2016 noted that they tended to assign low priority to qualitative studies because such studies are ‘usually exploratory by their very nature’.18 This statement came in response to an open letter from scholars arguing that BMJ should adopt formal policies and training for editorial staff on what distinguishes ‘good from poor qualitative research’ rather than de-emphasising the method in toto.19 In sum, despite sustained interest from the HSR community, the status of qualitative research remains contested. This status reflects debate over the purpose of qualitative research—is it a valuable tool to advance the field or a low-priority exercise in exploration? —and an ongoing desire for guidance on how best to distinguish high-quality from low-quality qualitative research.
Assessing rigour and quality in qualitative research is challenging because qualitative methods are epistemologically diverse.20–22 This diversity is a strength because it allows for the theoretical and methodological flexibility necessary to fully understand a specific topic from multiple perspectives.16 However, it also means that investigators do not necessarily approach qualitative research using a unified set of evidentiary rules.22 Thus, scholars may measure the quality of studies using different or even incompatible yardsticks.
The challenge of diverse epistemologies has become more acute as qualitative health research has expanded beyond its historical roots in phenomenological or grounded theory studies. Contemporary researchers may use qualitative data and methods to improve the descriptive accuracy of health-related phenomena that have already been characterised by exploratory work or are difficult to capture using other approaches.23 Researchers also use larger scale, comparative qualitative studies in ways that resemble quantitative efforts to identify explanatory pathways.24 Therefore, assessing the rigour of a specific qualitative study benefits from first identifying the analytic goals and objectives of the study—that is, identifying which yardstick investigators themselves have adopted—and then using this yardstick to examine how the study measures up.
To address these challenges, we propose a tailored framework for designing and informing assessments of different types of qualitative health research common within HSR. The framework recognises that qualitative investigators have different objectives and yardsticks in mind when undertaking studies and that rigour should be assessed accordingly. We distinguish three central types of qualitative study objectives common in applied health research: exploratory, descriptive and comparative. For each objective, we propose preliminary methodological considerations to help improve rigour across all study phases—from design to reporting. As is the case for quantitative studies, we argue that standards for qualitative rigour differ, appropriately, for different kinds of studies. The objective of this commentary is not to resolve all potential conflicts between philosophical assumptions of different qualitative approaches, but rather help to advance a broader and richer understanding of qualitative rigour in relationship to other evidence hierarchies. The proposed framework offers a nuanced set of categories by which to conduct and recognise high-quality qualitative research. The framework also supports efforts to shift debates over the value of qualitative research to discussions on how we can promote rigour across different types of valuable qualitative studies, and underscore how qualitative methods can advance clinical and applied health research.
Designing a tailored framework: methods and results
Our framework is based on a team-based review of published guidelines and standards discussing the scientific conduct of qualitative health research. Guided by expert consensus and a targeted literature scan, we identified and reviewed 17 peer-reviewed articles and expert reports published by journals widely read by the HSR community and by major funders or sponsors of qualitative health research (1–12, 21, 33–36). In contrast to previous reviews,25 we did not seek to synthesise these guidelines. Rather we drew on them to develop a conceptual framework for designing and informing formal assessments of rigorous qualitative research.
Range of approaches in qualitative research
Qualitative research incorporates a range of methods including in-depth interviews, focus groups, ethnography and many others.26 Even within a single method, accepted approaches and standards for rigour vary depending on disciplinary and theoretical orientations. Correspondingly, qualitative research cannot be defined by a single theoretical approach or data collection procedure. Rather many, often debated, approaches exist with distinct implications for appropriate standards for data collection, analysis and interpretation.
On one end of the spectrum, qualitative researchers guided by realism subscribe to the assumption that rigorous scientific research can provide an accurate and objective representation of reality, and that objectivity should be a primary goal of all scientific inquiries, including qualitative research.27 These qualitative researchers generally consider standards such as validity, reliability, reproducibility and generalisability as similarly legitimate yardsticks for qualitative research as they are in quantitative research.28 On the other end of the spectrum, relativist philosophical approaches to qualitative research typically argue that all research is inherently subjective and/or political,29 and some relativists criticise the scientific approach specifically because it claims to be objective.30 31 Much of applied qualitative health research falls somewhere between the two ends of the spectrum. For example, Mays and Pope consider themselves ‘subtle realists’.6 They acknowledge that all research involves subjectivity and includes political dimensions, but they also contend that qualitative research should, nevertheless, be assessed by a similar set of quality criteria as quantitative studies. Although we recognise the value strictly relativist approaches provide, the framework and design considerations we propose are largely guided by a realist (or subtle realist) orientation. However, in addition to resonating with those who operate under similar orientations, we hope this framework will serve to advance discussions of how best to communicate and assess qualitative research using different theoretical and epistemological standpoints.
Tailored framework for qualitative health research
Given the diversity of approaches, a foundational step to improving the assessment of rigour in qualitative research is to abandon the attempt to develop a single standard for the best practices regardless of study orientation and objective. Instead, standards must begin with an assessment of epistemological assumptions and corresponding study objectives, an approach that is similar to standards for quantitative PCOR research32 and mixed-methods research.33 In this vein, we identified and defined three general types of study objectives broadly used in applied qualitative health research (see figure 1). These three types reflect differences in primary study objectives and existing knowledge within a topic area.
Three broad types of qualitative health research.
In table 1, we provide preliminary distinctions on how exploratory, descriptive and comparative studies compare across a range of standards and guidelines that have been proposed for qualitative research (see table 1). Regardless of study type, researchers should report study details in clear, comprehensive ways, using standardised reporting guidelines whenever possible.34 35
Framework for designing different types of applied qualitative health research and developing evaluative instruments to assess their rigour
Compared with descriptive or comparative studies, exploratory studies approach the topic of study primarily in an inductive fashion to investigate the areas of potential research interest that remain mostly or wholly unexamined by the scientific community. Investigators undertaking exploratory studies typically have few expectations for what they might find, and their research design and approach may shift dramatically as they learn more about the phenomena of interest. An example of an exploratory study is a study that uses convenience sampling and unstructured interviews to explore what patients think about a new treatment in a single healthcare setting.
At the opposite end of this spectrum, investigators conducting comparative studies aim to use a primarily deductive approach designed to compare and document how well-defined qualitative phenomena are represented in different settings or populations. The qualitative methods employed in a comparative study are typically defined in advance, sampling should be systematic and structured by aims, and investigators enter the field with hypothesised ideas of what findings they may uncover and how to interpret those findings in light of previous research. An example of a comparative study is a multisite ethnography that seeks to compare how patient-provider communication varies by location, and uses random sampling of patient-provider interactions to collect data.
Descriptive studies occupy a middle position, building on previously conducted exploratory work so researchers will be able to proceed with more-focused inquiry. This should include well-defined procedures including sampling protocols and analytical plans, and investigators should usually articulate expected findings prior to beginning the study. However, as researchers investigate phenomena in new settings or patient populations, it is reasonable to expect descriptive studies to generate surprises. Thus, descriptive studies also feature inductive elements to detect unexpected findings, and must be flexible enough in design to accommodate shifts in research focus and methods based on empirical findings. An example of a descriptive study is a longitudinal study of patients with ovarian cancer that employs semistructured interviews and directed content analysis to examine decision-making across patients in a novel setting.
Discussion
Our review identified a number of qualitative standards and guidelines that have been published. The conceptual framework we present here draws on those extant guidelines through the recognition that qualitative health research includes studies of diverse theoretical and epistemological orientations, each of which has distinct understandings of scientific quality and rigour. Given this intellectual diversity, it is inappropriate to use a single yardstick for all qualitative research. Rather, assessments of qualitative quality must begin with an assessment of a study’s theoretical orientations and research objectives to ensure that rigour is assessed on a study’s own terms. This framework and suggested approaches may help to advance evaluations of qualitative rigour that acknowledge and differentiate between the studies that report exploratory, descriptive or comparative study objectives.
Existing standards for conducting health research and grading evidence, such as Grading of Recommendations Assessment, Development and Evaluation (GRADE),36 do not capture the diversity of qualitative studies—often designating all qualitative studies as providing weak levels of evidence. PCORI’s own methodological standards have been largely silent regarding qualitative methods until recently,32 leaving applicants without clear direction on how to conduct rigorous qualitative research. Incorporation of tailored qualitative standards could help to clarify and improve the rigour of proposal design, review and completion. The establishment and integration of such standards could also guide journal editors in developing transparent standards for deciding priorities for publication. For example, editors may decide against publication of exploratory or descriptive studies, but prioritise well-executed comparative studies that advance the field in ways quantitative studies could not.
In addition to these immediate applications, implementing standards that incorporate the diversity of objectives within applied qualitative research has the potential to address broader challenges facing qualitative health research. These include: (1) the need to educate broader audiences about the many goals of qualitative research, including but not limited to exploration; (2) the need to create rigorous standards for conducting and reporting various types of qualitative studies to help audiences, editors and funders evaluate studies on their own merits and (3) the challenges of publishing qualitative research in prestigious and high-impact journals that will reach a wide range of practitioners, researchers and lay audiences. We contend that these challenges can be reframed as opportunities to advance the science of qualitative research, and its potential for improving outcomes for patients, providers and communities.
Acknowledgments
Portions of this work were presented at the 2016 Academy Health Annual Research Meeting and 2017 Society of Behavioral Medicine Annual Meeting. The authors thank Katherine Gillespie and members of our Stakeholder Advisory Board for their invaluable contributions to the project.
References
Footnotes
Presented in Earlier version presented at the 2016 Academy Health Annual Research Meeting in Boston, MA, USA.
Contributors All authors (KAR, CMA, SBG, MCH and DD) helped to design and conceptualise this work including reviewing guidelines and conceptualising the proposed framework. KAR drafted the manuscript, and CMA, SBG, MCH and DD provided substantial review and writing to revisions.
Funding This work was funded by the Patient-Centered Outcomes Research Institute (PCORI) Award (ME-1409-22996).
Disclaimer The views presented in this article are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee. Funders had no role in the collection, analysis and interpretation of the data; in the writing of the report; and in the decision to submit the paper for publication.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.