QUALITY CRITERIA FOR QUALITATIVE RESEARCH

Recent decades have seen the emergence of many proposals for quality criteria for qualitative research. These have been useful on the one hand in identifying criteria that might be used, but have also led to an unhelpful proliferation and diversity with often little evidence of common ground.¹¹ For example, it has been shown¹² that guidelines proposed by Seale and Silverman¹³—which emphasise the need for detailed transcription, the support of generalisations with counts of events, and the use of computer software—differ significantly from those proposed by Popay et al⁶ which prioritise subjectivity, flexibility and adequate description and which argue that quasi-statistics and computer software are neither necessary nor sufficient for rigorous qualitative analysis. There are now over 100 sets of proposals on quality in qualitative research, some adopting non-reconcilable positions on a number of issues. For example, some recommend that qualitative studies should aim to be reproducible and that multiple coding is a good means of assessing the quality of qualitative research,^14,15 while others^16,17 deem such criteria meaningless when conducting research within a relativist paradigm involving multiple realities, subjectivity, and the negotiation of meaning. Attempts to produce consensus on criteria have proved difficult. The UK National Centre for Social Research has recently produced a framework for assessing qualitative research, drawing on 29 existing frameworks in the area as well as interviews with those active in the field and a workshop.¹⁸ This has proved useful in describing the tensions and diversity in the field but it is lengthy—involving over 18 separate domains—and potentially unwieldy.

QUALITATIVE RESEARCH IS NOT A UNIFIED FIELD

A key problem in much of the work on developing appraisal criteria for qualitative research has been the tendency to treat it as a unified field, both at the level of data collection (such as focus groups) and at the level of the methodological approach (such as grounded theory). Given the plurality of qualitative methodologies available, this is clearly a flawed approach destined to produce criteria that fit in certain cases but not in others. For example, Lincoln and Guba⁴ and others recommend respondent validation where participants are asked to check researchers’ interpretations. However, this may be an entirely unsuitable form of validity checking for some forms of qualitative research including, for example, discourse analysis, where its often anti-realist emphasis is on the multiple accounts that can be produced of any phenomenon rather than seeking a “single” verifiable account.

There is also a persistent failure to distinguish between the characteristics of a paper that are concerned with auditability and transparency of reporting, and those that are concerned with quality of process and analysis. This is not just a procedural problem or one of detail. It goes to the heart of what quality means in qualitative research. The dilemma is that some of the most important qualities of qualitative research can be the hardest to measure. For example, a study may be judged to have followed the appropriate procedures for a particular approach, to give information on selection of participants, and to provide clear details of the method followed. Yet the study may suffer from poor interpretation and offer little insight into the phenomenon at hand. On the other hand, a second study may be flawed in terms of the transparency of methodological procedures and yet offer a compelling, vivid and insightful narrative, grounded in the data. How the judgements of quality should be formed and informed in such cases is not at present clear. It does not follow from this that the project to address issues of quality in qualitative research should be abandoned—that is not helpful either intellectually or practically. However, it is crucial to take into account the particular features of qualitative research when considering the appropriate way of evaluating it.

UNIVERSAL AND SPECIFIC: WE NEED BOTH

We argue that there are universal features of all forms of qualitative research, but these are few in number, may well be universal to all forms of research, and are best formulated as prompts to sensitise appraisers to the various dimensions of articles that require evaluation. The prompts in box 1 have been developed by a project team in the ESRC Research Methods Programme, following an empirical evaluation of using two existing frameworks and extensive discussions within a multidisciplinary team. In the development of these prompts we have been explicitly attentive to the need to avoid commitments to particular methodological approaches, and have distinguished between aspects of reporting and aspects of study design and execution. In addition, we wish to make clear that these are proposed as prompts to cue the attentiveness to a specific set of issues. They are not criteria, do not prescribe how the reader makes the assessment, and do recognise that the assessment will inevitably involve subjective judgement on the part of the reader. Because these prompts are generic, it would also be possible for them to be complemented by prompts that are specific to different methods of data collection and qualitative methodologies. So far there have been few attempts to develop such methodology-specific approaches.

The need for such approaches is demonstrated by even the most cursory overview of currently published qualitative work in a wide range of journals. It is often difficult even to be sure about which study design—let alone which theoretical perspective—a published study has used. Researchers may describe something as qualitative research when, in the judgement of others, it is not. Similarly, studies may be described as using a specific approach such as grounded theory when, in the judgement of others, it has not been used.¹⁹

Indeed, grounded theory can be seen as an example of a particularly problematic case. Here the original text by Glaser and Strauss²⁰ is one of the most frequently cited social science texts. However, the initial vision is more frequently honoured in the breach than the observance.²¹ Inconsistencies, misappropriations, and mislabelling of studies purporting to use grounded theory are common.²² Some of the reasons for this lie in the practical difficulties of implementing grounded theory in its original form.²³ The problem is further exacerbated by the fissure that arose between Glaser and Strauss themselves, who went on to develop two quite distinctive approaches to grounded theory. Strauss’s later work is focused on the development of technique and procedure and could be seen as rather prescriptive in character, while Glaser’s work can appear to be lacking in a grounding in practicalities and has largely avoided the development of procedural techniques. It is, however, clear that some elements of grounded theory (particularly the constant comparative method of analysis) are especially useful, but the tendency to select some of these techniques to create ad hoc and “à la carte” approaches to qualitative analysis and still retain the label “grounded theory” is very unhelpful. The result is that, despite the tendency for researchers to assume that describing a study as using “grounded theory” is sufficient, it is far from self-evident what such a study might involve.

The situation in qualitative research contrasts with the development of methods for appraising quantitative research where it is recognised (in the work of the Critical Appraisal Skills Programme (CASP)²⁴ and the NHS Centre for Reviews and Dissemination (CRD) guidance,¹¹ for example) that different study types (such as randomised controlled trials, case control studies, studies to evaluate screening programmes) may demand different criteria. This allows the precise formulation of flaws that would be fatal or very damaging to the rigour of a particular study type. For example, failure to randomise properly in a randomised controlled trial, or to select appropriate controls in a case control study, could be deemed very serious problems. In qualitative research there is a need to recognise that focus groups, interviews, participant observation, and so on also constitute different study types and therefore may also need different criteria to allow their appraisal, and similarly have different types of fatal flaws. It is also important to recognise that the classification of minor errors and fatal flaws may not, however, be easily distinguished into simple binary judgements: it may be necessary to take a holistic view of a study that recognises the importance of context and what was feasible in that context.

It is also vitally important to recognise that, to a much greater extent than in quantitative research, the execution of a qualitative research study type is crucially related to the theoretical perspective in which the researchers have chosen to locate the study. An interview based study conducted within a grounded theory framework may therefore have very different characteristics from an interview based study conducted within a discourse analysis framework, and this clearly needs to be reflected in the framework for appraising a study. The key task then lies in defining what the expectations should be for a particular study design within a particular theoretical field. Unless we can move to these kinds of definitions and expectations, the diversity—indeed, near anarchy—in qualitative methodology means that it is very difficult to identify, or at least gain agreement, on what might constitute a fatal flaw in, for example, an interview based study using an opportunistic sample and the constant comparative method for analysis.

CONCLUSIONS

Qualitative research has an important and growing role in the study of quality and safety. The rationale for establishing the quality of qualitative research is clear. Problems occur when attempting to determine exactly how this task can proceed. It is evident that debate and discussion, together with systematic evaluation of the implications of adopting various methods for appraisal, are necessary to resolve many of the critical issues raised here. We have proposed a minimal set of prompts which have been designed to stimulate appraisal of different dimensions of qualitative research but are explicitly methodology neutral. We argue that there is a need for future development in this area to focus on the distinctive study designs and theoretical perspectives that qualitative research can adopt, to distinguish fatal flaws from more minor errors, and to recognise that many of the more important and interesting aspects of qualitative research may remain very difficult to measure except through the subjective judgement of experienced qualitative researchers.