A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed

Objectives: To evaluate critical appraisal tools (CATs) that have been through a peer-reviewed development process with the aim of analyzing well-designed, documented, and researched CATs that could be used to develop a comprehensive CAT.

Study design and setting: A critical review of the development of CATs was undertaken.

Results: Of the 44 CATs reviewed, 25 (57%) were applicable to more than one research design, 11 (25%) to true experimental studies, and the remaining 8 (18%) to individual research designs. Comprehensive explanation of how a CAT was developed and guidelines to use the CAT were available in five (11%) instances. There was no validation process reported in 11 CATs (25%) and 33 CATs (77%) had not been reliability tested. The questions and statements that made up each CAT were coded into 8 categories and 22 items such that each item was distinct from every other.

Conclusions: CATs are being developed while ignoring basic research techniques, the evidence available for design, and comprehensive validation and reliability testing. The basic structure for a comprehensive CAT is suggested that requires further study to verify its overall usefulness. Meanwhile, users of CATs should be careful about which CAT they use and how they use it.