Objectives Randomised controlled trials in healthcare increasingly include economic evaluations. Some show small differences which are not statistically significant. Yet these sometimes come to paradoxical conclusions such as: ‘the intervention is not clinically effective’ but ‘is probably cost-effective’. This study aims to quantify the extent of non-significant results and the types of conclusions drawn from them.
Design Cross-sectional retrospective analysis of randomised trials published by the UK’s National Institute for Health Research (NIHR) Health Technology Assessment programme. We defined as ‘doubly null’ those trials that found non-statistically significant differences in both primary outcome and cost per patient. Paradoxical was defined as concluding in favour of an intervention, usually compared with placebo or usual care. No human participants were involved. Our sample was 226 randomised trial projects published by the Health Technology Assessment programme 2004 to 2017. All are available free online.
Results The 226 projects contained 193 trials with a full economic evaluation. Of these 76 (39%) had at least one ‘doubly null’ comparison. These 76 trials contained 94 comparisons. In these 30 (32%) drew economic conclusions in favour of an intervention. Overall report conclusions split roughly equally between those favouring the intervention (14), and those favouring either the control (7) or uncertainty (9).
Discussion Trials with ‘doubly null’ results and paradoxical conclusions are not uncommon. The differences observed in cost and quality-adjustedlife year were small and non-statistically significant. Almost all these trials were also published in leading peer-reviewed journals. Although some guidelines for reporting economic results require cost-effectiveness estimates regardless of statistical significance, the interpretability of paradoxical results has nowhere been addressed.
Conclusions Reconsideration is required of the interpretation of cost-effectiveness analyses in randomised controlled trials with ‘doubly null’ results, particularly when economics favours a novel intervention.
- randomised controlled trials
- doubly null
- cost effectiveness
- economic evaluation
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Contributors JR initiated the project, carried out the research and wrote successive drafts. HW helped to initiate the project and made helpful comments on draft manuscripts. AC made helpful comments on draft manuscripts. JT made helpful comments on draft manuscripts. JN made helpful comments on draft manuscripts. HS made helpful comments on draft manuscripts. KS helped to initiate the project and commented on draft manuscripts.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests All authors work part-time for the NIHR HTA programme as well as being independent academics. All but HW and AC are current members of the editorial board of the HTA monograph series. HW is director of the HTA programme. AC is a past member of the editorial board of the HTA monograph series. JR is employed part-time as a researcher by NETSCC, the secretariat which manages the NIHR HTA programme. Neither NIHR nor the HTA programme had any role in interpreting the results nor in preparing this paper for publication. NETSCC has taken responsibility for payment of the publication fee.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement All data relevant to the study are included in the article or uploaded as supplementary information.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.