Article Text

Download PDFPDF

Effect of a multispecies probiotic on reducing the incidence of antibiotic-associated diarrhoea in children: a protocol for a randomised controlled trial
  1. Jan Łukasik,
  2. Hania Szajewska
  1. Department of Paediatrics, The Medical University of Warsaw, Warsaw, Poland
  1. Correspondence to Professor Hania Szajewska; hania{at}


Introduction Certain individual probiotic strains have been proven to be effective in reducing the risk of antibiotic-associated diarrhoea (AAD). However, the effects of using multispecies probiotics (MPs) remain unclear. We aim to assess the effectiveness of a specific MP preparation (Winclove 612) in reducing the incidence of AAD in children.

Methods and analysis A total of 350 children aged 6 months to 18 years, undergoing antibiotic treatment, will be randomly allocated to receive either a MP consisting of two strains of Bifidobacterium (B. bifidum W23 and B. lactis W51) and six strains of Lactobacillus (L. acidophilus W37, L. acidophilus W55, L. paracasei W20, L. plantarum W62, L. rhamnosus W71 and L. salivarius W24) at a total dose of 1010 colony-forming units daily, or a placebo, from the first day of antibiotic treatment until 7 days after antibiotic cessation, up to a maximum of 17 days. The primary outcome will be the incidence of AAD, defined as ≥3 loose or watery stools (a score of A on the Amsterdam Infant Stool Scale or a score of 5–7 on the Bristol Stool Form scale) in 24 hours, caused either by Clostridium difficile or of otherwise unexplained aetiology, occurring during the intervention period. The secondary outcomes will include the incidence of AAD according to alternative definitions; the incidence of any kind of diarrhoea; the duration of diarrhoea; the need for hospitalisation; intravenous rehydration or discontinuation of antibiotic treatment due to diarrhoea; adverse events; and the intestinal microbiota composition.

Ethics and dissemination The study protocol is approved by the Ethics Committee of the Medical University of Warsaw. The findings will be published in a peer-reviewed journal and submitted to relevant conferences.

Date and protocol version identifier 14/10/2017.

Trial registration number NCT03334604; Pre-results.

  • probiotics
  • antibiotics
  • diarrhoea
  • rct

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • This study’s design is simple, with the intent to answer a precise and unambiguous clinical question.

  • The study protocol closely follows the rules included in the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) statement.

  • This will be the first trial of this specific probiotic formulation in the paediatric population.

  • The incidence of antibiotic-associated diarrhoea (AAD) in specific populations is difficult to predict and may turn out to be lower than expected, limiting the trial’s statistical power.

  • Since AAD may occur up to 8 weeks after antibiotic treatment, some cases may be missed in this study.


Antibiotics are well known to cause disturbances in the composition of the intestinal microbiota, leading to the development of gastrointestinal (GI) symptoms.1 Antibiotic-associated diarrhoea (AAD), which may be defined as diarrhoea that occurs in relation to antibiotic treatment with the exclusion of other aetiologies, is a common complication of antibiotic use in children.2 Based on the analysis of data from randomised controlled trials (RCTs), the pooled risk of AAD in children was 19%.3 However, the risk varies greatly from study to study, ranging from 2.1%4 to 80%,5 depending on factors such as the adopted definition of diarrhoea, the study population and the type of antibiotic treatment.6 The underlying mechanism of AAD is not fully understood. It may be caused by a specific enteric pathogen (eg, Clostridium difficile, C. perfringens, Staphylococcus aureus or Candida albicans), metabolic consequences of altered intestinal microbiota or a direct effect of antibiotics on the mucosa.7 AAD may vary both in severity (from uncomplicated diarrhoea to pseudomembranous colitis) and in incubation period (from the first day of antibiotic treatment to 8 weeks after discontinuation).8

The impact of antibiotics on commensal micro-organisms of the gut justifies the idea of using probiotics to reduce the incidence of AAD. According to a consensus definition, probiotics are ‘live micro-organisms that, when administered in adequate amounts, confer a health benefit on the host’.9 There are a number of potential mechanisms of their action, including activity in the intestinal lumen (eg, competition with, or direct suppression of, pathogenic micro-organisms), interaction with the mucosal barrier (eg, upregulation of tight junctions, modulation of water and ion channels) and influence on the intestinal immune system.10

Probiotic properties are species-specific and strain-specific, so each strain or their combinations should be examined separately.2 11 In children, two probiotic strains with proven efficacy in the prevention of AAD are Lactobacillus rhamnosus GG and Saccharomyces boulardii.12 13 Both are currently recommended to reduce the incidence of AAD in children, if the use of probiotics is considered.2 Probiotic preparations consisting of more than one strain are not yet recommended for reducing the incidence of AAD in children, despite some evidence of their effectiveness.3 14

In this trial, a preparation consisting of eight probiotic strains (Winclove 612, Winclove Probiotics, the Netherlands), including two strains of Bifidobacterium (B. bifidum W23, B. lactis W51) and six strains of Lactobacillus (L. acidophilus W37, L. acidophilus W55, L. paracasei W20, L. plantarum W62, L. rhamnosus W71 and L. salivarius W24), will be used. Hereafter, this probiotic strain combination is referred to as ‘multispecies probiotic’ (MP). None of the individual strains included in MP have been proven to be effective in reducing the incidence of AAD. However, studies on the effectiveness of a comparable preparation, Ecologic AAD, in reducing diarrhoeal symptoms have been performed.15 16 The aforementioned preparation has a similar composition to MP; however, it additionally contains Enterococcus faecium W54. The species E. faecium is not recommended for use in children by  the European Society for Paediatric Gastroenterology Hepatology and Nutrition (ESPGHAN) due to unclear safety issues17 and, therefore, is excluded from the current formulation. In one RCT conducted in 41 healthy adult volunteers receiving amoxicillin with either Ecologic AAD or placebo, subjects in the experimental group had a significantly lower rate of diarrhoea-like bowel movements compared with those in the placebo group (48% vs 79%, respectively, relative risk (RR)=0.61, p<0.05).15 Another RCT conducted in 45 adult patients with a chronic obstructive pulmonary disease exacerbation who were treated with antibiotics did not reveal a difference the in rate of diarrhoea-like bowel movements between the Ecologic AAD and placebo groups (77% vs 70%, respectively, RR=1.1, p>0.05).16 However, this study was carried out in a very specific group of patients, that is, those with a history of frequent and prolonged antibiotic use. So far, there have been no RCTs using this probiotic preparation carried out in larger groups of participants or in children.

Methods and analysis


The primary objective of this study is to test the hypothesis that the MP reduces the risk of AAD in children undergoing antibiotic treatment. Other objectives include investigating the MP’s influence on the incidence of other types of diarrhoea, diarrhoea duration, intestinal microbiota composition and potential adverse events associated with the MP’s use.

Trial design

The study is a randomised, double-blind, placebo-controlled, parallel group trial with an allocation ratio of 1:1.

Study setting

Participants in this study will be recruited among both the inpatients and outpatients of the Paediatric Hospital of the Medical University of Warsaw, Poland. In case of a low recruitment rate (defined as described in the ‘Monitoring’ section of this document), other hospitals and medical care centres would also be plausible sources of participants, providing the presence of adequately trained personnel. In case of the inclusion of additional recruitment centres, adequate information will be added to the protocol registry site, and the bioethics committee will be informed.

Eligibility criteria

Eligibility criteria will be as follows: (1) age between 6 months and 18 years, (2) therapy with oral or intravenous antibiotics for common infections, (3) ability to start the probiotic intervention within 24 hours after the start of antibiotic intake, (4) therapy with broad-spectrum antibiotics (broad-spectrum penicillins, cephalosporins, fluoroquinolones, clindamycin) and (5) signed informed consent.

The exclusion criteria will include the following: prior use of antibiotics within the previous 4 weeks, presence of a severe or generalised infection, history of severe chronic disease (eg, cancer, inflammatory bowel disease, tuberculosis), critical/life-threatening illness, immunodeficiency, history of pre-existing diarrhoea within the previous 4 weeks, exclusive breastfeeding, allergy or hypersensitivity to any component of the study product, tube-feeding, use of proton-pump inhibitors, laxatives, antidiarrhoeal drugs or any probiotics 14 days before or during the study,


The experimental group will receive MP at a dose of 1010 colony-forming units (CFU) daily. This food supplement consists of the eight following bacterial strains:

  • B. bifidum W23

  • B. lactis W51

  • L. acidophilus W37

  • L. acidophilus W55

  • L. paracasei W20

  • L. plantarum W62

  • L. rhamnosus W71

  • L. salivarius W24

Apart from the probiotic strains, the active product consists of maize starch, maltodextrin, fructo-oligosaccharides P6, maize dextrin P9, potassium chloride, hydrolysed rice protein, magnesium sulfate, amylase and manganese sulfate. The dosage of MP to be used in this study is based on the aforementioned human studies with a comparable preparation.15 16 The control group will receive a placebo product that is indistinguishable in colour, smell and taste from MP, and will have the same composition but without the live bacteria, fructo-oligosaccharides and maize dextrin. Both MP and placebo will be a powder, which has to be dissolved in water or milk before use. The interval between antibiotic intake and probiotic consumption has to be at least 2 hours. The study products (MP and placebo) will be manufactured and supplied by Winclove Probiotics B.V (Amsterdam, The Netherlands) free of charge.

The products will be transferred to the study site with a temperature control system, and the readings from a thermometer will be verified after their delivery. The study products will be stored at the study site in a locked, dark, and dry place, at room temperature.

Explanation for choice of comparators

To enable assessment of the occurrence of AAD in this study’s population, a placebo will be used as a comparator. Contrary to the ‘best available therapy’ model, use of a placebo may lead to the development of a number of cases of theoretically avoidable AAD in the placebo group. However, overestimation of the MP’s effectiveness will be avoided.18 One may argue that probiotics with proven efficacy such as LGG or S. boulardii should be used in the control group. However, it is noteworthy that they are only recommended if the use of probiotics for preventing AAD is considered because of the existence of risk factors such as class of antibiotic(s), duration of antibiotic treatment, age, need for hospitalisation, comorbidities or previous episodes of AAD diarrhoea.2 19 Due to these factors, no universal standard of care to reduce the risk of AAD in the paediatric population is defined.

Study procedure

The recruiting physician who is familiar with the study protocol will perform an eligibility screen on the prospective patients, who began therapy with antibiotics in the preceding 24 hours, based on their medical records. Then, during a face-to-face meeting with the patient’s caregivers, the recruiter will obtain missing information concerning the inclusion and exclusion criteria, explain the study procedures, risks and benefits and supply them with a leaflet containing the study’s description. After that, written informed consent in two copies will be obtained from the participant’s caregivers. Consent will be also obtained from participants themselves if they are 15 years of age or older. Subsequently, the patient’s case report form (CRF) will be created and archived along with one copy of the informed consent. Participants will be randomised to receive orally two times per day either MP at a dose of 5×109 CFU (total daily dosage of 1×1010 CFU) or a placebo during the antibiotic treatment and until 7 days after antibiotic cessation, up to a maximum of 17 days. This period is referred to as the intervention period later in the document. Data from earlier studies suggest that doses of >5×109 CFU of probiotic micro-organisms are more effective than doses <5×109 CFU in preventing AAD.20

During the intervention period (ie, the whole MP/placebo administration period), stool number and consistency will be recorded in a study diary, based on the Amsterdam Infant Stool Scale (AISS)21 for children younger than 1 year and the Bristol Stool Form (BSF) scale22 for children older than 1 year. The study diaries will be filled in by caregivers of participants younger than 14 years or by participants themselves, provided they are older than 14 years. A score of A on the AISS or 5–7 on the BSF scale will be considered as loose or watery stool. Caregivers also will be instructed to record any other observations concerning the health of the participants, including all adverse events involving the GI tract (such as vomiting, decreased appetite or abdominal pain) or other systems as well as information regarding compliance with treatment (ie, if the participant has taken the MP or not) in the study diary. The diary will be returned to the study site at the end of the intervention period. The outcome data for inpatients (eg, the occurrence of diarrhoea) will be verified using hospital charts.

The participants will be reminded not to use other treatments during the intervention period that may affect the incidence or course of the diarrhoea, namely other probiotics, diosmectite, loperamide, proton pump inhibitors or laxatives. Usage of any of the aforementioned preparations will be treated as a protocol violation, and such patients will not be included in the per protocol analysis. Caregivers will be asked to write down in the study diary any other medications or dietary supplements taken by the participants during the intervention period. Withdrawal of consent for participation in the study will be possible at any moment, with no consequences, and without an obligation to give reasons for the decision. In case of the occurrence of serious adverse events or new circumstances affecting the safety of the participants (eg, difficulty in swallowing, a new diagnosis of immunodeficiency), the intervention will be discontinued.

In cases of the occurrence of diarrhoea, stool samples will be obtained and examined for presence of common diarrhoeal pathogens—rotavirus, adenovirus, norovirus, Campylobacter spp, Salmonella spp, Shigella spp, and Yersinia spp—via chromatographic immunoassay (for viruses) or isolation from stool culture (for bacteria). Additionally, C. difficile toxins A and B will be identified in the stool using immunoassay in cases involving children older than 1 year.

Additionally, as a part of an independent study, participants’ microbiota composition will be tested in the stool at four time points: at baseline, at the day of antibiotic cessation, at the end of intervention and 1 month after the intervention’s cessation. The tests will be performed by analysing microbial gene sequences with 16S rRNA-based diversity methods. DNA will be extracted from the faecal samples by state-of-the art methods in a specialised laboratory. PCR amplificated 16S rRNA gene fragments will be analysed with the use of Illumina HiSeq Sequencer, and subsequent bioinformatic analyses will be performed by standardised pipelines within this laboratory. Next to this, microbial biomass will be measured with quantitative PCR and/or flow cytometry. Microbial functionality (metabolites produced) can be performed in addition to the composition analyses, and will be done by proteome analyses.


The primary and secondary outcomes (for details, see below) will be assessed during the intervention period. There will be no follow-up period. In cases of inpatients discharged before the end of the intervention period as well as in outpatients, the caregivers will be asked to bring the remaining product along with the study diary to the study site at the end of the 7-day intervention period.


Compliance with the study protocol will be assessed by direct interview with the patient and/or caregiver, by analysing information from the study diary and by checking the number of returned non-consumed study products. Participants who receive <75% of the recommended dose of MP/placebo will be considered as non-compliant.

Outcome measures

The primary outcome measure will be AAD, defined as three or more loose or watery stools (a score of A on the AISS or 5–7 on the BSF scale) per day in a 24-hour period, caused by C. difficile infection or of otherwise unexplained aetiology (after testing for common diarrhoeal pathogens), occurring during the intervention period.

Secondary outcomes assessed during the intervention period will include AAD based on two other definitions of diarrhoea used in previous studies:

  • ≥3 loose or watery stools per day for a minimum of a 48-hour period caused by C. difficile infection or of otherwise unexplained aetiology.

  • ≥2 loose or watery stools per day for a minimum of a 24-hour period caused by C. difficile infection or of otherwise unexplained aetiology.

For both definitions, loose or watery stools will correspond to a score of A on the AISS or 5–7 on the BSF scale. AAD needs to be caused by C. difficile infection or of unexplained aetiology (after testing for common diarrhoeal pathogens), and it must occur during the intervention period.

Other secondary outcome measures will be as follows:

  • Any diarrhoea (defined as ≥3 loose or watery stools per day for a minimum of 24 hours regardless of its aetiology).

  • C. difficile-associated diarrhoea (diarrhoea defined as above caused by C. difficile confirmed by the presence of toxin-producing C. difficile in stools (positive toxin tests)).

  • The duration of diarrhoea (defined as the time until the normalisation of stool consistency according to the BSF or AISS scale (on BSF scale, numbers 1, 2, 3 and 4; on AISS scale, letters B or C), and the presence of normal stools for 48 hours).

  • Discontinuation of the antibiotic treatment due to severity of diarrhoea.

  • Hospitalisation caused by diarrhoea in outpatients.

  • Need for intravenous rehydration in any of the study groups.

  • Adverse events.

  • Intestinal microbiota composition, tested in stool samples as described above at four time points: at baseline, at the day of antibiotic cessation, at the end of intervention and 1 month after the intervention’s cessation.

The timeline of the study is presented in table 1.

Table 1

The timeline of the study

Sample size

The pooled risk of AAD determined from previous studies conducted at the Medical University of Warsaw23 24 is 12.4%. However, in those studies, the definition of diarrhoea was more strict—loose or watery stools had to last for at least 48 hours, so AAD is expected to be more frequent in our proposed study. Consequently, we have chosen to perform a sample size calculation based on an expected AAD risk of 16%, which is a compromise between the results from the Medical University of Warsaw and the pooled AAD risk of 19% as reported in the Cochrane meta-analysis.3 To show a difference of 11% in the treatment effect in the study groups with α=0.05% and 80% power (unpaired Student’s t-test), and assuming a 20% withdrawal rate, a total of 337 participants will be needed. Sample size calculations were performed with StatsDirect (V.3.1.4, StatsDirect statistical software; StatsDirect, Chesire, UK).

Random sequence generation and allocation concealment

The randomisation will be performed centrally by Winclove Probiotics B.V. by a person not involved in the study. Blocked randomisation (blocks of four) will be used to ensure a good balance of participant characteristics in each group. Allocation will be determined by using a computerised random number generation process. All study products will be sequentially numbered. Coded study products will be handed over to the researchers. When the study has ended, participants will be divided into two blinded groups, which will be used in the statistical analyses. After performing the analyses, code numbers will be opened by the coordinating and principal investigators. Sealed envelopes containing the allocation of each number will be handed to the principal investigator ensuring that if a medical problem occurs for which treatment allocation is needed, the code can at all times be broken.


The probiotic preparation and placebo will be stored in identical packages. The contents will look, smell and taste the same. Researchers, caregivers, participants, medical personnel and outcome assessors will all be blinded to the intervention until the study is completed and the data analysed.

Data collection and management

All study participants will receive a study identification number. CRFs containing each participant’s identification number and baseline data will be filled in electronically and printed. Outcome data will be added to both the paper and electronic copies of the CRF after the reception of the study diary. Electronic data will be stored in a password-protected electronic database. The original paper copies of the CRFs and all study data will be stored in a locker within the study site. Both versions of the CRFs will be accessible to the involved researchers only. Overall, only the involved researchers will have access to the participant’s personal information, and no personal data will be shared with the company performing the randomisation or with any other outside party.

Statistical analysis

Descriptive statistics will be used to summarise baseline characteristics. For continuous variables, comparison between groups will be done using the Student’s t-test or Mann-Whitney U test, depending on whether or not the variables are distributed normally. The normality of the distribution will be checked using the Shapiro-Wilk test. The χ2 test or Fisher’s exact test will be used, as appropriate, to compare dichotomous variables. Differences between groups will be presented for continuous outcomes as differences in means or differences in medians (for normal or non-normal distribution, respectively) along with a 95% CI. For dichotomous outcomes, the RR and number needed to treat, calculated as the inverse of the absolute risk reduction, will be determined along with a 95% CI. In the second stage of analysis, the primary outcome will be analysed by logistic regression, controlling for five prespecified potential risk factors for AAD (age, sex, antibiotic class, duration of antibiotic treatment and duration of hospital stay). The difference between study groups will be considered significant when the p value is <0.05, when the 95% CI for RR (or OR) does not include 1.0 or when the 95% CI for mean difference does not include 0. All statistical tests will be two-tailed and performed at the 5% level of significance.

An intention-to-treat model will be applied—data from all randomised participants will be used in the analysis, including those with low compliance or those who drop out or withdraw their consent. Per-protocol analysis will be performed as well, and it will include all participants who finish the study according to the protocol.


The study will be carried out in accordance with the protocol, as it will be registered. No changes in the study protocol are expected to be made after the study starts. However, in case of any unexpected circumstances requiring alterations of the protocol, changes will be immediately applied to the protocol registry site at, and, if relevant enough, reported to the Bioethics committee. An independent Data and Safety Monitoring Board (DSMB) will be created before the start of the study. The DSMB will review data after recruitment from 25%, 50% and 75% of participants to assess the study progress (including rate of recruitment, completeness of data and their appropriate collection) and all of the adverse events. The number of recruited patients will be monitored and kept up to date; appropriate changes (ie, training of the recruiting physicians, study leaflets, addition of new recruitment centres) will be applied to the study procedure and protocol if the pace of recruitment is not high enough to finish the study within the established time, which is 2 years.


All eight of the probiotic strains to be used in the study have the Qualified Presumption of Safety status established by the European Food Safety Authority.25 The occurrence of serious adverse events in immunocompetent populations during oral use of probiotics is unlikely.26

The exact same product has not been assessed in previous studies. However, several clinical studies have been performed with a comparable product, in different populations (healthy volunteers and chronic obstructive pulmonary disease patients) in the Netherlands and Austria without any reported serious side effects.15 16 27 Moreover, currently, a study is being performed with Ecologic AAD in patients with spinal cord injury who require antibiotic treatment during their inpatient rehabilitation (trial number: NTR5831).

In addition, the preparation is commercially available in several countries (Austria, Germany, Greece, Norway, Russia, Slovenia, Ukraine and the Netherlands) and since the market introduction in 2007, no serious adverse effects have been reported. In the Netherlands, probiotics are considered to be food or food supplements and, therefore, have to be produced under Hazard Analysis and Critical Control Point regulations, which is the Dutch regulation system for safety and hygiene in food and food supplements. All components are legally admitted as food additives or food components. Winclove is an NSF International Certified Good Manufacturing Practices Facility for manufacturing dietary supplements and works with the food safety management system ISO 22000:2005.

Overall, based on the literature and manufacturer’s data, we assume that receiving the study product poses only a marginal risk to the participants. Nevertheless, during the whole study period, the participants will benefit from telephone and email contact with the primary investigator, so all the potential adverse events will be reported to and consulted by a physician. Moreover, patients at higher likelihood of experiencing severe adverse events (eg, critical/life-threatening illness, immunodeficiency or severe chronic illness) will not be recruited, as stated in the exclusion criteria.

Since adverse events of probiotic use are unlikely, no prespecified list will be a part of the study diary or CRF. Instead, a section entitled ‘other symptoms’ will be included, in which caregivers of the participants will be able to write down any other symptoms that occur during the intervention. Additionally, at the time of study diary reception, a physician will personally ask the caregiver about the occurrence of any symptoms during the study. As indicated in the Consolidated Standards of Reporting Trials (CONSORT) extension on harms document,28 all of those symptoms will be reported for all of the randomised participants, including those who withdraw from the study. The data on adverse events will be presented for each study arm and each type of adverse event separately, with an exact count of each event, and distinction between patients with single and multiple events.

In case of suspected serious adverse events, the project leader will immediately notify the Ethics Committee, DSMB, all study personnel and the manufacturer of the product about the nature of the event. The decision regarding continuation or discontinuation of the trial will be made by the project leader in agreement with the Ethics Committee and DSMB. All adverse events also will be noted in the CRFs.

Patient and public involvement

Patients and public were not involved in the design of the study.

Ethics and dissemination

The protocol of the study was reviewed and approved by the Ethics Committee of the Medical University of Warsaw. Participants (or their legal representatives) will be fully informed about the study, and informed consent will be obtained. The manufacturer of the study products commented on the first draft of the protocol; however, all final decisions were made by the study team who also will be in charge of all study data.

The manufacturer will have no role in the conduct of the study, or in the analysis or interpretation of the data. The findings of this study, whether positive or negative, will be published in a peer-reviewed journal in accordance with CONSORT. Abstracts will be submitted to relevant national and international conferences.



  • Contributors HS conceptualised the study. JŁ developed the first draft of the manuscript. Both authors contributed to and approved the final manuscript. HS is the guarantor.

  • Funding This study will be funded by the Medical University of Warsaw. Both the placebo and the probiotic preparation will be manufactured and kindly provided for study purposes by Winclove Probiotics B.V. (Amsterdam, The Netherlands). Allocation concealment and randomisation procedures will also be performed by the product’s manufacturer, as described, free of charge. At the same time, the manufacturer will have no access to the patient’s individual information and no role in the conduct of the study, management, analysis and interpretation of the data or dissemination of the findings.

  • Competing interests None declared.

  • Patient consent Not required.

  • Ethics approval Ethics Committee of the Medical University of Warsaw.

  • Provenance and peer review Not commissioned; externally peer reviewed.