Table 2

MMAT grading criteria (adapted from Pace et al31)

Types of mixed methods study components or primary studiesMethodological quality criteria
(See tutorial for definitions and examples)
Screening questions (for all types)Are there clear qualitative and quantitative research questions (or objectives), or a clear mixed methods question (or objective)?
Do the collected data allow address the research question (objective)? Eg. Consider whether the follow-up period is long enough for the outcome to occur (for longitudinal studies or study components).
Further appraisal may not be feasible or appropriate when the answer is ‘No’ or ‘Can't tell’ to one or both screening questions
1. Qualitative1.1 Are the sources of qualitative data (archives/documents/informants/observations) relevant to address the research question (objective)?
1.2 Is the process for analysing qualitative data relevant to address the research question (objective)?
1.3 I appropriate consideration given to how findings relate to the context, eg the setting, in which the data were collected?
1.4 Is appropriate consideration given to how findings relate to researchers’ influence, eg through their interactions with participants?
2. Quantitative randomised control (trials)2.1 Is there a clear description of the randomisation (or an appropriate sequence generation)?
2.2 Is there a clear description of the allocation concealment (or blinding where applicable)?
2.3 Are there complete outcome data?
2.4 Is there low withdrawal/drop-out (below 20%)?
3. Quantitative non-randomised3.1 Are participants (organisations) recruited in a way that minimises selection bias?
3.2 Are measurements appropriate (clear origin, or validity known, or standard instrument; and absence of contamination between groups when appropriate) regarding the exposure/intervention and outcomes?
3.3 In the groups being compared (exposed vs non-exposed; with intervention vs without; cases vs controls), are the participants comparable, or do researchers take into account (control for) the difference between these groups?
3.4 Are there complete outcome data (80% or above), and, when applicable, an acceptable response rate (60% or above), or an acceptable follow-up rate for cohort studies (depending on the duration of follow-up)?
4. Quantitative descriptive4.1 Is the sampling strategy relevant to address the quantitative research question (quantitative aspect of the mixed methods question)?
4.2 Is the sample representative of the population understudy?
4.3 Are measurements appropriate (clear origin, or validity known, or standard instrument)?
4.4 Is there an acceptable response rate (60% or above)?
5. Mixed methods5.1 Is the mixed methods research design relevant to address the qualitative and quantitative research questions (or objectives), or the qualitative and quantitative aspects of the mixed methods question (or objective)?
5.2 Is the integration of qualitative and quantitative data (or results) relevant to address the research question (objective)?
5.3 Is appropriate consideration given to the limitations associated with this integration, eg the divergence of qualitative and quantitative data (or results) in a triangulation design?
Criteria for the qualitative component (1.1 to 1.4), and appropriate criteria for the quantitative component (2.1 to 2.4, or 3.1 to 3.4, or 4.1 to 4.4), must be also applied).
  • MMAT, Mixed Methods Appraisal Tool.