Translating new evidence into clinical practice: a quasi-experimental controlled before–after study evaluating the effect of a novel outreach mentoring approach on knowledge, attitudes and confidence of health workers providing HIV and infant feeding counselling in South Africa

Objectives We report the effectiveness of a mentoring approach to improve health workers’ (HWs’) knowledge, attitudes and confidence with counselling on HIV and infant feeding. Design Quasi-experimental controlled before–after study. Setting Randomly selected primary healthcare clinics (n=24 intervention, n=12 comparison); two districts, South Africa. Participants All HWs providing infant feeding counselling in selected facilities were invited. Interventions Three 1–2 hours, on-site workshops over 3–6 weeks. Primary outcome measures Knowledge (22 binary questions), attitude (21 questions—5-point Likert Scale) and confidence (19 questions—3-point Likert Scale). Individual item responses were added within each of the attitude and confidence domains. The respective sums were taken to be the domain composite index and used as a dependent variable to evaluate intervention effect. Linear regression models were used to estimate the mean score difference between intervention and comparison groups postintervention, adjusting for the mean score difference between them at baseline. Analyses were adjusted for participant baseline characteristics and clustering at health facility level. Results In intervention and comparison sites, respectively: 289 and 131 baseline and 253 and 114 follow-up interviews were conducted (August–December 2017). At baseline there was no difference in mean number of correctly answered knowledge questions; this differed significantly at follow-up (15.2 in comparison; 17.2 in intervention sites (p<0.001)). At follow-up, the mean attitude and confidence scores towards breast feeding were better in intervention versus comparison sites (p<0.001 and p=0.05, respectively). Controlling for confounders, interactions between time and intervention group and preintervention values, the attitude score was 5.1 points significantly higher in intervention versus comparison groups. Conclusion A participatory, low-intensity on-site mentoring approach to disseminating updated infant feeding guidelines improved HWs’ knowledge, attitudes and confidence more than standard dissemination via a circular. Further research is required to evaluate the effectiveness, feasibility and sustainability of this approach at scale.

The limitations suggest that you were seeking to change "actual infant feeding practices" while the theory and constructs indicate you were focusing on HW infant feeding counselling practices (also not measured). This is more in line with the study population of interest (p. 1, line 49). Similarly, clarify what the study was intending to change knowledge, attitudes and confidence about, e.g. counselling based on the latest evidence, on p.4, line 10 in the Background.

BACKGROUND
Add a reference to the frequent changes to infant feeding guidelines, e.g. Jackson et al, 2019 in BMJ Open or Nieuwoudt et al, 2019 in PLOS One. Alternatively, reference the actual guidelines that changed over the past decade. Poor dissemination of past/changing guidelines to HWs has also been written about in the South African literature and could be referenced. (p.3, line 23) Add primary reference for the PMTCT guidelines (p3, line 40) and reference the policy shifts in South Africa that resulted. There is no mention of the guideline changes after the Tshwane Declaration, which was arguably one of the more dramatic shifts in PMTCT counselling for HIV exposed infants in the public health setting. On p4, lines 20-21, the wording suggests that the 2017 change was the only (or most significant) that the two provinces have experienced, which may not be the case.

METHODS
Specify and reference the technique(s) used to adjust for clustering on p.5, line 6 and/or p.6, line 52.
The authors need to engage with the possibility that the Horwood et al article will not be published by the time this is accepted (see p5, line 17 and line 55). If this happens, more description of the intervention will be needed.
The question of whether and how the 2017 circular was disseminated to clinics (p5, lines 46-8) through routine systems is important and should feature more strongly in analysis of intervention results, beyond Supplemental Figure 1. Specific suggestions are made in results section.
Add references to existing tools you used/adapted and note whether or not they are validated (p. 6, line 8 or in section starting on line 23). Please indicate if the study team conducted any factor analysis or validation exercises on the three scales? In reviewing the tables, I am concerned that several questions are doublebarreled or overly complex so as distort findings. For example, the first part of the knowledge question about sterilization of bottles is True and the second part is False. The low levels of correct answers may relate to the nature of the way the question was framed. This is a potential limitation that should be acknowledged.

RESULTS
p.7, lines 30-9: Were the three statistically significant differences in training between comparison and intervention sites explored and/or integrated into the models measuring differences in attitudes and confidence (Tables 4 & 5)? If so, this could strengthen the argument for an intervention effect. If not, please explain their exclusion or consider adding this to your analysis.
p.9, from line 21. For knowledge interpretation, it may be worth highlighting the questions in Table 3 that contradicted prior PMTCT guidelines to support interpretation and deepen discussion.
What was the rationale for not analysing knowledge using the same robust statistical techniques to those applied for attitudes and confidence?
Supplementary Tables 1 and 2: The p-values for individual items presented in the table seem to be incorrect or inconsistent. For example, in line 16 of the Table 1, 90% vs. 83% has a p-value of 0.14, while on line 19, 88.6% vs. 88.4% has a p-value of 0.06. I recommend that these are checked and that the results (p.12, lines 5-9 for Supp Table 1 and p.13, lines 5-6 for Supp Table 2) are updated if needed. This issue was noted for both tables.
The "three analyses" conducted to create Tables 4 and 5 are not specified. Particularly given that only two models are presented (see p.12, line 9 and 19; p.13, line 9), the test(s) used to generate the first model in both tables needs to be specified and differentiated. On p.12, line 20, "ANCOVA or linear regression analyses" are mentioned. One or the other should be specified as the basis for the tables (or explain that they had identical results if that was the case).

DISCUSSION
While logically presented, the discussion ventures beyond the study findings with several arguments with limited empirical evidence. I recommend that the authors identify studies from the global literature that can bolster or support the following arguments that are made: 1. That improvements in knowledge, attitudes and/or confidence will result in HWs improving their counselling practices and that improved HW counselling can lead to improved infant feeding practices (p.14, lines 24-26) 2. The intervention is a sustainable model (p.14, line 38). [NB: My understanding is that a dedicated mentor/facilitator was required to run a series of workshops across clinics. Without a clear dose response, is this sufficient evidence for DOH to invest in mentors at scale?] 3. On-site mentorship is more cost-effective than off-site (assuming DOH is willing to invest in any mentorship scheme) (p.15, lines 25-30) LIMITATIONS p.15, line 37: Are you referring to the personal breastfeeding experiences of HWs or their previous experiences of providing infant feeding counselling? This is unclear and both limitations could be argued The validity of the scales used to measure the concepts of knowledge, attitudes and confidence should be addressed in the limitations or explained in greater detail in methods. p.12, line 23: Consider using the word "better" to describe attitude rather than "higher" for easier interpretation Table 3, p.11. The = sign used before some statements is confusing and I thought it was a typo at first. I suggest rather using a more common signifier, like **, to indicate that the reader should look for the explanation below the table. p12, line 9: Remove the comma before (Supplementary Table 1 This has now been added.

MINOR
The limitations suggest that you were seeking to change "actual infant feeding practices" while the theory and constructs indicate you were focusing on HW infant feeding counselling practices (also not measured). This is more in line with the study population of interest (p.1, line 49). Similarly, clarify what the study was intending to change knowledge, attitudes and confidence about, e.g. counselling based on the latest evidence, on p.4, line 10 in the Background. Thank you for this comment -this has now been clarified. We were not seeking to measure changes in mothers' practices: we sought to improve health care workers' knowledge of, attitudes towards and confidence with counselling on HIV and infant feeding. However, we conducted 12 focus groups amongst mothers attending these facilities approximately 3 months after the health worker training. Add primary reference for the PMTCT guidelines (p3, line 40) and reference the policy shifts in South Africa that resulted. There is no mention of the guideline changes after the Tshwane Declaration, which was arguably one of the more dramatic shifts in PMTCT counselling for HIV exposed infants in the public health setting. On p4, lines 20-21, the wording suggests that the 2017 change was the only (or most significant) that the two provinces have experienced, which may not be the case. Additional references have been added.
Our statement says that: 'In June 2017, these recommendations were adopted in South Africa, thus necessitating updates for health workers.' It does not specifically say that only these two provinces were affected.

METHODS
Specify and reference the technique(s) used to adjust for clustering on p.5, line 6 and/or p.6, line 52. A reference has now been added to provide clarity.
The authors need to engage with the possibility that the Horwood et al article will not be published by the time this is accepted (see p5, line 17 and line 55). If this happens, more description of the intervention will be needed. The paper has been published and added as a reference.
The question of whether and how the 2017 circular was disseminated to clinics (p5, lines 46-8) through routine systems is important and should feature more strongly in analysis of intervention results, beyond Supplemental Figure 1. Specific suggestions are made in results section. The district office sent the circular via e-mail and whats app to all health facilities. This has now been added on pg. 6. The study team documented that the June 2017 circular issued by the National Department of Health, informing health facilities of the change in Infant and Young Child feeding policy, was disseminated to comparison clinics as an announcement via e-mail and other electronic communication as well as during meetings or trainings. We documented that in Tshwane, 15 of the 18 clinics had received the circular; 11 via e-mail and three at a meeting. In Ugu nine of 17 clinics had received the circular; 8 received it via hand delivery and one via e-mail.
Add references to existing tools you used/adapted and note whether or not they are validated (p. 6, line 8 or in section starting on line 23). Please indicate if the study team conducted any factor analysis or validation exercises on the three scales? References have been added. Factor analyses were not conducted on the tools and they were not validated; however they were reviewed by an education specialist and were piloted amongst health care providers. This is explained in the intervention paper that has been published: Christiane  It is also explained more clearly in the data collection section of this paper.
In reviewing the tables, I am concerned that several questions are double-barreled or overly complex so as distort findings. For example, the first part of the knowledge question about sterilization of bottles is True and the second part is False. The low levels of correct answers may relate to the nature of the way the question was framed. This is a potential limitation that should be acknowledged. Thanks for thisit has been added as a limitation.

RESULTS
p.7, lines 30-9: Were the three statistically significant differences in training between comparison and intervention sites explored and/or integrated into the models measuring differences in attitudes and confidence (Tables 4 & 5)? If so, this could strengthen the argument for an intervention effect. If not, please explain their exclusion or consider adding this to your analysis. Thank you for raising this. The analysis has been re-done to control for these findings and Tables 4 and 5 have been updated.
p.9, from line 21. For knowledge interpretation, it may be worth highlighting the questions in Table 3 that contradicted prior PMTCT guidelines to support interpretation and deepen discussion.
The questions really measuring the main change in the 2017 guidelines are: "Continued breastfeeding for 2 years is the recommended infant method in SA for ALL children, regardless of mother's HIV status (True)" and "In South Africa, HIV-infected women who are breastfeeding should be supported to adhere to antiretroviral treatment and should introduce complementary foods around 6 months and be supported to continue breastfeeding for at least two years. (True)" For the first question in the intervention group there was a 36% improvement in knowledge whilst in the comparison group there was a 13% increase in knowledge.
For the second question in the intervention group there was a 15% increase in correct knowledge for this question while for the comparison group knowledge decreased from 89-81%. So, these findings supports the conclusion that knowledge increased and especially for the main change in the guideline i.e. duration of breastfeeding. These findings have been flagged in the results, in the table and in the discussion.
What was the rationale for not analyzing knowledge using the same robust statistical techniques to those applied for attitudes and confidence?
We were more interested in understanding changes in attitudes and confidence, as knowledge can easily be improved through studying/ training, whilst attitude and confidence are more difficult to change; hence the more advance analyses for attitudes and confidence.
Supplementary Tables 1 and 2: The p-values for individual items presented in the table seem to be incorrect or inconsistent. For example, in line 16 of the Table 1, 90% vs. 83% has a p-value of 0.14, while on line 19, 88.6% vs. 88.4% has a p-value of 0.06. I recommend that these are checked and that the results (p.12, lines 5-9 for Supp Table 1 and p.13, lines 5-6 for Supp Table 2) are updated if needed. This issue was noted for both tables. Apologies for this and thank you for pointing this outthere were incorrect transcriptions in this table These have now been rectified.
The "three analyses" conducted to create Tables 4 and 5 are not specified. Particularly given that only two models are presented (see p.12, line 9 and 19; p.13, line 9), the test(s) used to generate the first model in both tables needs to be specified and differentiated. On p.12, line 20, "ANCOVA or linear regression analyses" are mentioned. One or the other should be specified as the basis for the tables (or explain that they had identical results if that was the case 12, line 23: Consider using the word "better" to describe attitude rather than "higher" for easier interpretation Done Table 3, p.11. The = sign used before some statements is confusing and I thought it was a typo at first. I suggest rather using a more common signifier, like **, to indicate that the reader should look for the explanation below the  Table 3. Show within the group differences in addition to between the group differences.
The study was not designed to show differences by site and province; thus we are not comfortable reporting the data by the two sites. However, we have reviewed the data by site, and decided to combine the data because there were few differences between the sites.
Strengths and limitations: include something on the differences between provinces, and that the knowledge score sometimes increased at comparison sites. The study and sample size was not designed to show differences by site and province; thus we are not comfortable reporting the data by the two sites. Formatting/grammar: extra space (page 3, line 46), delete 'of' (page 3, line 53), incorrect punctuation (page 15, line 46). These have been corrected.
Reviewer: 3 Please leave your comments for the authors below This manuscript is tantalising in that the approach to participatory mentoring of the health workers concerned is to be congratulated. However the huge elephant in the room is the apparent total lack of involvement of women living with HIV, who are supposed to be the beneficiaries of this training, or their values and preferences in the process; and no commentary on this. Given the growing body of research, practice and guidance in this complex area, this is a considerable and disappointing omission. For example, there is clear guidance in the WHO 2017 Consolidated Guideline on the SRHR of women living with HIV that women living with HIV should be meaningfully involved in all research that affects their lives (section 6.2.1 especially); and that "peer support, provided by, with, and for women living with HIV, should be included in HIV care" (GPS A.1). This guideline was published before the research baseline was conducted in August 2017. There is also an RCT (Richter et al 2014) that describes the advantage of a peer mentor mother process -yet none seemed to be in place -or was mentioned -in this manuscript. Documents published by WHO more recently have clearly described the huge added value of the meaningful involvement of women living with HIV (see eg WHO 2019a (https://www.who.int/reproductivehealth/publications/srhr-women-hivimplementation/en/) and WHO 2019b (https://apps.who.int/iris/bitstream/handle/10665/330034/WHO-RHR-17.33-eng.pdf?ua=1). Amongst other benefits, the use of women living with HIV to inform health workers directly of the many and complex issues they face, including in relation to infant feeding, have been documented. It is also important for health workers to be trained in trauma-informed care, so that they are aware of the full range of potential barriers to access to care and treatment experienced by women living with HIV. Thank you for this comment. We agree that it is important to measure the effect of interventions on mothers and their children. However, this paper and study specifically and consciously : sought to assess the effect of a participatory intervention for health workers on their knowledge, attitudes and confidence. Given the study timelines we did not seek to measure the effect of the intervention on mothers' infant feeding practices. This is definitely being included in our next pieces of work, which we are seeking funding for. However, women were involved in qualitative interviews : we interviewed women who had received care from the participating clinics to explore their perceptions of care and this has been published separately: Doherty  We did not gather this information. It may have been a missed opportunity but the primary focus of our manuscript was not to assess the relationship between health care workers' HIV status and knowledge, It was to assess whether a participatory mentoring approach improves health care workers' knowledge of, attitudes towards and confidence with counselling on HIV and infant feeding.  Table 2 is not discussed at all in the text, but is referred to on p.11, line 7, to argue for a dose response (data not shown) and p15, line 33 in noting there was not a significant dose response.
p.13, Table 3, line 33: Percentage in the baseline comparison column has 2 decimal places p.14, Table 4 and accompanying text: As Model 2 is not presented in the article, the reference to three approaches and presentation of Model 1 and Model 3 are somewhat confusing to the reader. As with comments in data analysis, I'm wondering if this isn't more of a distraction than necessary. If all three will be reference, maybe indicate to the reader where Model 2 is shown.
p.15, Table 5 and accompanying text: Same comment about missing Model 2 in the presentation as noted for Table 4.

Discussion
The first paragraph does not discuss findings, but synthesizes the results. As such, it may be better placed as a concluding paragraph in the Results section. There is also a lot of similarity between this content and what is discussed in the third paragraph of the discussion. Can you synthesize? Later on in the paper you also mention: "poor alignment between community views/ practice and health programmes.
[29]'. No matter how good a training may be, unless it produces the intended outcome amongst those who are supposed to be the end beneficiaries, it is not a complete study. This omission therefore continues to be a limitation of this study and should be identified clearly as such, in line with current standards. It should also be identified as an area for future development.
4. You rightly describe the ongoing health worker crisis in limited settings -here is an opportunity also to highlight the possibility of developing a peer mentor mother programme, so that women living with HIV themselves can feel supported to share this information with each other. Recent examples from eg UNICEF's programme in Southern Africa suggests early positive results from peer mentor mother work. And I cited previously other examples. It is also in line with the WHO 2017 SRHR Guideline. This should be highlighted also for further research. PVT is not a known or globally acceptable term; thus, for ease of sharing our article through MESH headings we have reverted to using the term preventing mother-to-child transmission of HIV (PMTCT) in the manuscript and have thus reverted to PMTCT. We hope this is OK. We agree that this does not and should not apportion blame on the mother. There needs to be an international dialogue about the term PMTCT and an agreement on whether to change to PVT instead of PMTCT .
2. Unfortunately, the PPI statement about why you did not include women living with HIV in this study still does not align with the WHO 2017 Guideline on SRHR of women living with HIV around their meaningful involvement in any research which affects their lives. See also this article: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6803394/ about the value of involving patients for quality improvement in healthcare. Later on in the paper you also mention: "poor alignment between community views/ practice and health programmes.
[29]'. No matter how good a training may be, unless it produces the intended outcome amongst those who are supposed to be the end beneficiaries, it is not a complete study. This omission therefore continues to be a limitation of this study and should be identified clearly as such, in line with current standards. It should also be identified as an area for future development. We completely agree that the ultimate aim is to improve feeding practices amongst mothers, and that in any work going forward we will need to collaborate with women who are HIV negative and women who are living with HIV to co-design interventions that aim to improve health workers' knowledge, attitudes, skills and practices. However, this study did not set out to do this. It was a very limited piece of work and the aims captured in the study protocol, stated that: 'This study aims to assess the feasibility and impact of a novel outreach-based mentorship approach, implemented through the primary health care system, for disseminating the 2016 updated WHO HIV and infant feeding guidelines on infant feeding attitudes and practices of front-line health workers in South Africa.' Thus it completely focused only on health workers, and we co-designed the intervention with health workers. At this late stage we cannot change the aims of the study, nor can we not report significant results because the study did not co-design the intervention with women living with HIVit was unfortunately not set up to do the latter; nor was it set up to measure changes in feeding practices as the time frame was limited, the funder needed the work to be completed by a specific date. We did however include women, including those living with HIV, in a qualitative sub-study to explore their perspectives of the infant feeding counselling received at the intervention facilities. This has been published as a separate paper (Doherty T, Horwood C, Haskins L, Magasana V, Goga A, Feucht U, et al. Breastfeeding advice for reality: Women's perspectives on primary care support in South Africa. Maternal & Child Nutrition 2020; 16(1):e12877 DOI: 10.1111/mcn.12877.) We had already included this as a limitation; however we have now expanded on this and made this a separate point as it is an important limitation. We have also included this as an area for future research.
3. Thank you again for adjusting the language in places. However, It is not ideal to keep mentioning 'mothers living with HIV' -in more places than currently, they could be referred to as women living with HIV, since eg breastfeeding women are most likely to be mothers also. This is all about looking at women holistically, rather than just as reproductive machines, which feels dehumanising.
We disagree with the view that using the word 'mother/mothers' views women as reproductive machines and is thus dehumanizing, because the word mother in this context denotes someone with a child, and so it specifically indicates a particular group of women, and as a group we have respect for the position of 'mother'.. However, we have amended the term to women living with HIV. This is an important point, and we agree that peer-peer mentors or feeding buddies can influence feeding practices, and we have added this point in. However the purpose of this study (and thus the purpose of this paper) was to investigate the effect of a novel outreach-based mentorship approach, implemented through the primary health care system, for disseminating the 2016 updated WHO HIV and infant feeding guidelines on infant feeding attitudes and practices of front-line health workers in South Africa. Thus it was outside the scope of this study to investigate the effect of peer counsellors / supporters / mentors / feeding buddies on infant feeding practices.

You rightly describe the ongoing health worker crisis in limited settings
Reviewer: 1 Reviewer Name: Sara Jewett Nieuwoudt Institution and Country: University of the Witwatersrand Please state any competing interests or state 'None declared': None declared Please leave your comments for the authors below I was pleased to see the revisions to this manuscript, which have strengthened the overall presentation and rigour of this work. This contributes to how we think about supporting HWs who are involved in infant feeding counselling. The Discussion, while improved, still could provide more critical engagement with the results. Thank youthis has been corrected p7, line 8: Change "altitude" to "attitude" Changedthank you p7, line 10: Use past tense for all descriptions of analysis for consistency (see also line 16 and 19). Model 2 is discussed here, but not presented in results. As such, as it necessary to discuss Model 2 at all?
The tense has been revised where appropriate, and past tense has been used when appropriate.
Results p.10, Table 2: The statistics on how many workshops HWs attended was 249 (out of 303 eligible), but the percentage calculations treat 249 as 100%, which seems misleading. What happened to the ~18% missing data? This part of Table 2 is not discussed at all in the text, but is referred to on p.11, line 7, to argue for a dose response (data not shown) and p15, line 33 in noting there was not a significant dose response. Table 2. Table 2 is descriptive, and the role of dose is referred to later on in the results. This table was added in so that the reader understands a bit more about the workshops.

p.13, Table 3, line 33: Percentage in the baseline comparison column has 2 decimal places
Thank youthis has been corrected p.14, Table 4 and accompanying text: As Model 2 is not presented in the article, the reference to three approaches and presentation of Model 1 and Model 3 are somewhat confusing to the reader. As with comments in data analysis, I'm wondering if this isn't more of a distraction than necessary. If all three will be reference, maybe indicate to the reader where Model 2 is shown.
We were keen to present all three methods in the data analysis section, to the reader to illustrate that the findings are robust, regardless of the analytical methods: However, we have amended the wording to include three methods but present 2 models because methods 1 and 2 yielded very similar responses. we have amended the wording as follows: • Method 1 used the post-treatment measurements as the outcome variable, but adjusted for the pre-treatment values; • Method 2 analysed the change score as an outcome variable adjusting for pre-treatment values; • Method 3 analysed all the pre-and post-measurements as the outcome variable, and used time (coded : 1 at follow-up and 0 at baseline) as a covariate with an interaction term for time and treatment, in addition to an adjustment for pre-treatment values).
p.15, Table 5 and accompanying text: Same comment about missing Model 2 in the presentation as noted for Table 4.
We have amended the wording -hope it's clearer now.

Discussion
The first paragraph does not discuss findings, but synthesizes the results. As such, it may be better placed as a concluding paragraph in the Results section. There is also a lot of similarity between this content and what is discussed in the third paragraph of the discussion. Can you synthesize?
Thanks for this commentsthe paragraphs have been synthesised p.16, lines 28-29: Suggest responding confidence in a separate sentence, as improvements were not significant in regression analysis for Model 3.

p.16, line 33: Mentions 2017 guidelines while on line 50 mentions 2016 guidelines. Correct year.
We have clarified that the WHO guidelines were updated in 2016 and the SA policy was revised in June 2017. The reference to the two guidelines have now been clarified in the paper.
The second paragraph does not discuss findings. It repeats background information on the intervention. This seems misplaced unless integrated into discussion.
We have now included description of the features of the mentoring approach into the methods section.  2) Thank you for expanding on this limitation. The challenges experienced with developing the confidence of health workers to support women with the issues described at the bottom of page 14 would be reduced if the programme had consulted with women living with HIV themselves about how to address these issues with their peers. Indeed the whole knowledge dissemination issue would be more effective if the women themselves were actively engaged in discussions around their own healthcare and that of their babies.
3) Thank you for referring to women rather than mothers here. 4) Thank you for adding this point in. However, in your tracked change on p17, it looks as if there is an either/or between mentoring for HW or peer mentoring by women living with HIV. Whilst I appreciate you were only exploring the former, it would surely be preferable to be promoting both here, given point 2 above especially.    Tables 4 and 5 were based on Intention to treat (ITT) analysis (as opposed to a per-protocol (PP) analysis). Thus, HWs who did not attend any workshop were not excluded from the treatment group in estimating the mentorship program effect. We have removed results concerning dose response effect from the narrative pertaining to Table 4 and 5, and instead indicated these at the end of the results section. The dose response analysis included attendance at 0, 1 or 2, or 3 workshops.