Dear editor,
According to the authors, the "study lends support to the hypothesis that a large-scale naloxone programme in a region may decrease rates of overdose deaths in the population, in comparison to a historic control period when naloxone distribution was unavailable".
They found that the rate of opioid overdoses in Skåne, the southernmost Swedish region, dropped significantly in three years ensuing the implementation of a Naloxone take home program (2019–2021) for opioid addicts, compared to "a historic control period” (2013–2017).
The authors correctly state the risk of biases in observational retrospective studies. Their results should be interpreted with caution. However, even with such assurances, their results may be questioned. The years 2013-2017 happens to represent the peak of the Swedish opioid epidemic when Sweden as a whole experienced high rates of overdose deaths in the population (1).
The years 2019-2021, on the other hand, represents a period when Sweden experienced a significant drop in overdose deaths (2). The present study specifically evaluates a local Naloxone take home programme, the first of its kind in Sweden. The observed decline in death rates may be part of a national trend that in turn may have many different explanations. For instance, in 2018 two Swedish brothers were convicted of manslaughter for selling Fentanyl online, in an unprecedented verdict. Fentanyl associated deaths dropped sharply thereafter and contributed to the overall decline in opioid associated overdoses in Skåne as well as in Sweden as a whole (3).
Considering that Skåne have had a similar trend in opioid overdose deaths as the rest of Sweden, it seems unfair to claim that Naloxone take home programmes may decrease overdose deaths in the population, based on the findings by Håkansson et al.

1) Andersson, L., Håkansson, A., Krantz, P. et al. Investigating opioid-related fatalities in southern Sweden: contact with care-providing authorities and comparison of substances. Harm Reduct J 17, 5 (2020). https://doi.org/10.1186/s12954-019-0354-y
2) Swedish health board (Folkhälsomyndigheten), 2024. Drug induced deaths. Accessed 2024 january. Weblink: https://www.folkhalsomyndigheten.se/livsvillkor-levnadsvanor/andts/utvec...
3) Swedish State Television (SVT) 2019. Sharp reduction in deaths caused by Fentanyl. Accessed 2024 January. Weblink: https://www.svt.se/nyheter/lokalt/ost/farre-dodsfall-med-natdroger-dom-m...

This article [1] reports three evidence syntheses which investigated outcomes of e-cigarette use among non-smokers: an umbrella review of three systematic reviews [2,3,4]; a “top-up” systematic review that included additional studies; and a quantitative meta-analysis of studies from the umbrella review and the top-up systematic review. Unfortunately, the approach for assessing the risks of bias of the individual research studies suffers from serious flaws: it is inconsistent, uncritical, lacks transparency, and does not inform any of the data syntheses. We are therefore uncertain whether the results and conclusions of the umbrella review, systematic review, and meta-analysis could be biased. This appears likely, since each of the three systematic reviews included in the umbrella review [2,3,4] had identified risks of bias or other “quality” issues in the evidence. Two of those reviews had identified moderate [3] or high [4] risks of bias due to confounding in their included studies, according to the ROBINS-I tool. The third review [2] (which did not use the ROBINS-I tool) found that outcomes of e-cigarette use differed between studies that were rated “fair quality” or “poor quality” on the Newcastle-Ottawa Scale (NOS) compared to those rated “good quality”. None of these bias or “quality” concerns are considered in the methods, results or conclusions of the umbrella review or meta-analysis. The authors did conduct a “quality assessment” of the studies included in their top-up review, using the NOS [1]. The outcome of this assessment is reported as the number of “stars” achieved out of a maximum of 10, with no explanation of whether or how this output from the NOS can be interpreted in terms of the risk of bias (and as noted above, these NOS results do not inform the data synthesis).

Inadequate methodological and reporting standards have been reported for other systematic reviews of e-cigarette use outcomes [5], and we question why research in this area is not conducted more critically and rigorously. For a risk of bias assessment (and hence its parent systematic review) to be fit-for-purpose, four core principles must be met: (i) the assessment must assess the risk of bias, i.e. assess the risk of systematic error in estimation of outcomes (as opposed to assessing other “quality” constructs which may have no bearing on systematic error); (ii) all risks of bias relevant to the study design(s) of interest should be identified; (iii) the risk of bias assessment must inform the data synthesis in an appropriate way; and (iv) the assessment process should be transparently reported and explained [6]. The present article falls short on all four of these principles.

Systematic reviews, umbrella reviews and meta-analyses are high-profile methods of evidence synthesis that should provide reliable answers to research questions, or at least indicate the extent of uncertainty in the answers. But if these evidence syntheses fail to account for biases or other errors in the primary research studies then their results may be inaccurate or wrong, leading to the propagation of flawed evidence, with potential for negative impacts on end-users of the evidence syntheses including policymakers and researchers. It is important that systematic review teams and peer reviewers of systematic reviews understand the basic requirements of risk of bias assessments, particularly the need for risk of bias assessments to inform the data synthesis and conclusions of a review, and the importance of clearly differentiating between the risk of bias (i.e. systematic error) and other “quality” constructs [6].

REFERENCES

1. Baezinger ON, Ford L, Yazidjoglou A, Joshy G, Banks E. E-cigarette use and combustible tobacco cigarette smoking uptake among non-smokers, including relapse in former smokers: umbrella review, systematic review and meta-analysis. BMJ Open 2012;11e045603.

2. Khouja JN, Suddell SF, Peters SE, Taylor AE, Munafò MR. Is e-cigarette use in non-smoking young adults associated with later smoking? A systematic review and meta-analysis. Tobacco Control 2021; 30:8-15.

3. Soneji S, Barrington-Trimis JL, Wills TA, Association between initial use of e-cigarettes and subsequent cigarette smoking among adolescents and young adults. JAMA Pediatrics 2017;171(8):788-797.

4. Aladeokin A, Haighton C. Is adolescent e-cigarette use associated with smoking in the United Kingdom?: A systematic review with meta-analysis. Tobacco Prevention and Cessation 2019;5(15).

5. Kim MM, Pound L, Steffensen I, Curtin GM. Reporting and methodological quality of systematic literature reviews evaluating the associations between e-cigarette use and cigarette smoking behaviors: a systematic quality review. Harm Reduction Journal 2021;18(1):121.

6. Frampton G, Whaley P, Bennett M, Bilotta G, Dorne J-LCM, Eales J, James K, Kohl C, Land M, Livoreil B, Makowski D, Muchiri E, Petrokofsky G. Principles and framework for assessing the risk of bias for studies included in comparative quantitative environmental systematic reviews. Environmental Evidence 2022;11:12.

Dear Editor,
Thank you for drawing our attention to this rapid response by Condon L, which was published on the 6th December 2023.
In response to the author’s comments on the inclusion of one of her studies - (Reference #52: Condon L, Rhodes C, Warren S, et al. ‘But is it a normal thing?’ Teenage mothers’ experiences of breastfeeding promotion and support. Health Education Journal 2013; 72: 156–62) in our review. We will take each point in turn.

Comment: It is unclear why our article has been included in this systematic review. In their Methods section, Malouf et al. describe their study selection criteria as follows: “‘Studies were eligible for inclusion if they involved women with low-risk pregnancies [...] and gave birth in hospitals or birth centres in the UK’. Our study explored experiences of breastfeeding promotion and support among pregnant teenagers and teenage mothers, and does not provide information about risk in pregnancy or place of birth. Our study therefore does not meet the eligibility criteria and should not have been included.

Response: We have clearly stated our approach to the eligibility criteria highlighted by Condon in our strengths and limitations sections:
‘Although we set out to review the literature relating to postnatal care for women at low risk of complications to explore routine practice, this was not always possible. Most of the studies reported results undifferentiated by risk and without excluding those women at high risk. Similarly, this review has focused on postnatal care in hospital, but for some outcomes, particularly those relating to infant feeding, it was not possible to separate hospital from community care. These studies were included for completeness.’ Malouf et al 2019

The inclusion of the study in question was one that was discussed and decided on by the whole research team as the relevance of the study in regard to sample was unclear. We decided to include because one of the themes highlighted health professional care in hospital just after birth. For example under this theme the following statement was made:
‘Feeding decisions made in pregnancy were not necessarily followed through when the baby was born, demonstrating the influence of health professional promotion and support at birth. Despite wishing to breastfeed, one mother was directed towards the hospital stocks of formula milk and subsequently bottle fed her baby.’ Condon et al 2013
This theme was deemed to be relevant to the review objectives to comprehensively report on women’s experiences of the immediate postnatal care received in hospitals and birth centres. Based on this information and in the absence of risk data or evidence of specialist services related to breast feeding around birth we decided to include.

Comment: The erroneous inclusion of our article leads the authors to make inaccurate claims about the quality of our study. In Table 2, Malouf et al. list the characteristics of qualitative studies included in their review. For our study, the ‘Sample characteristics’ column states that details were not reported for the postnatal sample. This is misleading, as Table 1 of our article presents the characteristics for our overall sample of teenage mothers and pregnant teenagers; however, as laid out above, our study did not include a separate postnatal sample Furthermore, in their Results section, under the heading ‘Risk of bias in qualitative studies’, Malouf et al. have claimed that the study population was “not described” in our article, “limiting transferability”. This gives the impression that our article fails to appropriately disclose the sample characteristics, which is untrue.

Response: These are statements of fact in regard to the description of the 23 teenage mothers in the sample. Such descriptions are common in systematic reviews and are not a judgement on the paper. They are included for complete transparency on what data could be extracted from the paper using our data extraction protocol.

Comment: Malouf et al. provide a quality assessment of the qualitative studies included in their review. In final column of this table, titled ‘Is the research valuable?’, it is unclear whether the authors are referring to the value of the research for their review, or to the field in general. Our study has been deemed not valuable, but the authors do not disclose the methodology used to reach this apparently subjective judgement. Of note, our study meets almost all of the quality assessment criteria listed in Table 4, while other studies considered valuable meet fewer of these criteria

Response: To assess the study quality we used the CASP risk of bias assessment for qualitative studies. The CASP tool is widely used to quality assess qualitative studies in qualitative reviews and the reporting that we provided for each domain is standard. Condon’s study has performed well using this tool, with seven ‘yes’ ratings out of ten questions. This was a reflection on the clarity of the study aims, the appropriate qualitative methodology and design to achieve the study aims, the transparency of the study recruitment strategy, the adequacy of the data collection (focus group and interviews), sufficient details on the ethical approval and the explicit findings of Condon’s study. The study was rated ‘unclear’ on one question – ‘Was the data analysis sufficiently rigorous?’. We could not find enough information in the paper on the analysis process or sufficient details on how the coding for the thematic analysis was done and by whom (whether done by e.g. two researchers or independently). The final question on the CASP is ‘is the research valuable?’. The terminology used is that of CASP’s and not ours and we completely understand how this could cause offence and we agree that many of the questions can be subjective. A key component of this final question is whether the researchers findings can be transferred to the local/other populations. In this study a rating of ‘no’ was given as the study findings cannot be transferred to other similar populations. This was stated by the authors in their study’s limitations section– ‘rates of breastfeeding initiation and continuation were higher among the study sample than nationally’ and ‘study sample may have been highly atypical of pregnant teenagers and teenage mothers in other UK cities’. The ‘no’ rating is for this one domain and is not a global assessment of study quality which is why other studies could be considered ‘valuable’ but did not perform well on other domains.

Comment: We recognise that this systematic review has been published for some time; however, it has recently been brought to our attention, and based on the points laid out above we would consider it appropriate for the article to be corrected to remove the misleading claims regarding our work. Furthermore, we strongly recommend that the authors consider whether the study selection and eligibility criteria have been correctly applied across all studies included in their review, and how this might affect the credibility of their findings.

Response: This review followed a rigorous methodology, decision making and reporting process. We greatly value the contribution of Condon et al’s paper in this review on this particularly under researched group. We do not believe that the data included in the review misrepresents the study within the context of the review question. We stand over our decision to include the paper in the review, the data extracted from the paper and quality assessment made.

The author team would like to inform the readers of a recent amendment to the original protocol as published here.

The ARON trial was planned before the COVID-19 pandemic. Though, the trial actually started in the initial part of the pandemic, just as the Delta variant (second wave) was first emerging in early 2021. The ARON trial kept going through all subsequent COVID-19 waves and is currently still ongoing. The COVID-19 pandemic has shown that antibiotic prescribing has (for the better) declined during the pandemic, as shown by several studies and reports:
- Gillies, MB, Burgner, DP, Ivancic, L, et al. Changes in antibiotic prescribing following COVID-19 restrictions: Lessons for post-pandemic antibiotic stewardship. Br J Clin Pharmacol. 2022; 88( 3): 1143- 1151.
- https://www.ecdc.europa.eu/en/news-events/reported-decrease-antibiotic-c...
- Colliers, A.; De Man, J.; Adriaenssens, N.; Verhoeven, V.; Anthierens, S.; De Loof, H.; Philips, H.; Coenen, S.; Morreel, S. Antibiotic Prescribing Trends in Belgian Out-of-Hours Primary Care during the COVID-19 Pandemic: Observational Study Using Routinely Collected Health Data. Antibiotics 2021, 10, 1488. https://doi.org/ 10.3390/antibiotics10121488

This has urged us to revise our sample size calculation, based on the overall prevalence found in the first patients recruited, given the start of our trial during the pandemic.

This amendment concerns a clarification of the impact the COVID-19 pandemic has had on recruitment as well as the overall prevalence of our primary outcome, the antibiotic prescribing rate in children.

With this amendment, we aim to:
- (1) investigate the influence of the COVID-19 pandemic on antibiotic prescribing rates in practices included in the ARON trial and (2) redetermine the required sample size for the ARON trial.
- (3) monitor whether inclusions were done consecutively, as per the study protocol, in order to minimize selection bias.

The following methods are described in this amendment to achieve the above aims:

1. Audit of antibiotic prescribing rate before and during the COVID-19 pandemic:
An audit is an official methodical examination and review, typically of an organisation’s or individual’s accounts or financial situation . In the context of this appendix, we aim to audit practices’ antibiotic prescribing rates in a time-matched before-after manner. We will compare the rate before the COVID-19 pandemic and during the COVID-19 pandemic, in an identical period during the year. A statistically significant change from baseline quantified by odds ratios will indicate a change in antibiotic prescribing behaviour due to COVID-19. This audit will be executed by the study coordinator (SC) that manages the practice.

2. Adjusted sample size calculation:
The original sample size calculation was based on previous data, assuming an overall prescribing rate of 26.5% in those children recruited for our trial.
The overall prevalence was found to be 18% overall in children recruited in the first 4938 patients recruited in the ARON trial.
If we were to assume a reduction of 5.3% (proportionate to our original reduction) between the usual care group and the intervention group, using a 5% significance level (alpha 0.05), an intracluster correlation coefficient (ICC) of 0.063 (based on data from the ERNIE2 study in the exact same population), and power of 90% (beta 0.1), this would require 63 clusters of 50 patients in both arms, resulting in 6300 children.
R code:
We used the n4props-function in the R package CRTSize, using the following command:
n4props(0.2065,0.1435,50,0.063, alpha = 0.05, power=0.9, AR=1, two.tailed=TRUE, digits=3)
Considering the pragmatic nature of this trial and in correspondence with the 10% of practices performing non-consecutive inclusion of patients (high risk of selection bias) during the ERNIE2 trial, we will perform sensitivity analyses only considering physicians who have recruited in a consecutive way. Taking into account the required sample size for this analysis of the primary study outcome, these assumptions result in a total sample size of 7000 patients for the primary study outcome.

3. Explore non-consecutive recruitment:
In the protocol, we have specified: “The participating physicians will be asked to consecutively recruit children with an acute illness over the recruitment period covering two winter seasons.”
We wanted to further clarify what is meant by non-consecutive recruitment, by stating that GPs are likely to breach this assumption if:
- They recruit less than 10 patients per year
- They perform a point-of-care CRP test on nearly all (>90%) of the included children
- They deliberately only include children (>90%) that do not need antibiotics and exclude those that might need antibiotic treatment.

All of the above circumstances introduce significant selection bias and reduce the generalisability of our findings. Although we plan to perform an intention-to-treat analysis given the pragmatic nature of our trial, protocol violations caused by the inappropriate inclusion/exclusion of patients will be far more detrimental for the validity of our trial. Apart from the intention-to-treat analysis a per protocol sensitivity analysis will be performed.

The above-mentioned assumptions can be tested, based on:
- the qualitative sub-study (semi-structured interviews with parents and physicians) in a selection of the recruiting practices as described in the original protocol
- the audit as described above
- monitoring visits by the clinical trial centre supporting data monitoring of the ARON trial

The above changes as part of the amendment have been approved by the Funder (KCE), the Ethical Review Board of UZ/KU Leuven (Belgium) and adapted in the registered protocol on Clinicaltrials.gov (NCT04470518).

Yours sincerely,

Prof Dr Jan Y Verbakel, on behalf of the author team of the ARON trial protocol.