Article Text
PDF
Abstract
Objective The Bayley Scales of Infant Development (BSID) is the most used diagnostic tool to identify neurodevelopmental disorders in children under age 3 but is challenging to use in low-resource countries. The Ages and Stages Questionnaire (ASQ) is an easy-to-use, low-cost clinical tool completed by parents/caregivers that screens children for developmental delay. The objective was to determine the performance of ASQ as a screening tool for neurodevelopmental impairment when compared with BSID second edition (BSID-II) for the diagnosis of moderate-to-severe neurodevelopmental impairment among infants at 12 and 18 months of age in low-resource countries.
Methods Study participants were recruited as part of the First Bites Complementary Feeding trial from the Democratic Republic of Congo, Zambia, Guatemala and Pakistan between October 2008 and January 2011. Study participants underwent neurodevelopmental assessment by trained personnel using the ASQ and BSID-II at 12 and 18 months of age.
Results Data on both ASQ and BSID-II assessments of 1034 infants were analysed. Four of five ASQ domains had specificities greater than 90% for severe neurodevelopmental delay at 18 months of age. Sensitivities ranged from 23% to 62%. The correlations between ASQ communications subscale and BSID-II Mental Development Index (MDI) (r=0.38) and between ASQ gross motor subscale and BSID-II Psychomotor Development Index (PDI) (r=0.33) were the strongest correlations found.
Conclusion At 18 months, ASQ had high specificity but moderate-to-low sensitivity for BSID-II MDI and/or PDI <70. ASQ, when administered by trained healthcare workers, may be a useful screening tool to detect severe disability in infants from rural low-income to middle-income settings.
Trial registration number NCT01084109.
- community child health
- paediatric neurology
- developmental neurology & neurodisability
Data availability statement
Data are available upon reasonable request. Data will be available upon request to the corresponding author.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THE STUDY
This is the largest study with data on both Ages and Stages Questionnaire (ASQ) and Bayley Scales of Infant Development second edition (BSID-II) at 18 months of age from low-income and middle-income countries, providing concurrent validity on both tests.
Because ASQ and BSID-II assessments were done by two different groups of certified experts, neither the results of the screening test (ie, ASQ) nor the reference standard (ie, BSID-II), were biased.
ASQ was not completed by the parent or guardian as is its original intent. However, we are providing evidence of the validity of ASQ in respect to BSID-II when done by certified providers, allowing us to see the maximum potential of this test. Validation of ASQ in the local cultural context in the manner used here could increase its value for international use.
Introduction
Nearly 200 million children under 5 years, most from south Asia and sub-Saharan Africa, live in socioeconomic conditions that undermine their developmental potential.1 Medical, nutritional and socioeconomic conditions increase the risk of poor cognitive, motor and social development in children.2 More than 80% of children with neurodevelopmental delays live in low-income and middle-income countries (LMICs).3 Recent findings suggest that almost half the children under 5 years of age living in LMICs do not reach their cognitive potential.4 Furthermore, 2005–2015 Early Childhood Development Index data collected from 35 LMICs showed that 26.2% of the children performed poorly in the socioemotional domain and 14.6% in the cognitive domain.5 Low-cost screening tools completed by parents or caregivers could increase the detection of developmental delay, facilitate the initiation of corrective therapies and improve neurodevelopmental outcomes in early childhood.6–8 This in turn could reduce the intergenerational transmission of poverty.9
The Bayley Scales of Infant Development (BSID) test remains one the most used tool for the identification and diagnosis of neurodevelopmental delays among infants.10 However, while the acceptability and reliability of BSID have been studied in many LMIC settings, it takes over 1 hour to administer, requires a trained professional to complete and is not practical for routine screenings in a low-resource setting.6 11 Additionally, while the cost of administering this test varies between countries based on salaries, one study estimated that it costs around US$400 per infant to undergo assessment using BSID.12 A more user-friendly tool that reduces cost, takes less time to complete and allows parental assessment would be extremely beneficial.12
Studies have found that parents provide reliable information about the development of their child when compared with a healthcare professional.13 Furthermore, responses from parent-administered standardised screening tools closely matched those obtained by trained professionals.14 The Ages and Stages Questionnaire (ASQ), a low-cost, easy to use parent-administered screening tool that measures the development of young children between ages 1 month and 5½ and takes around 15 min to complete, is such a candidate.7 ASQ has been translated into 16 languages in LMICs and at least 23 LMICs have used the questionnaire.15 However, there is a need for in-country validation of the ASQ tool prior to its universal use.15
Whereas mixed findings exist comparing the ASQ and BSID in high resource settings, there have been no studies that have compared ASQ with BSID second edition (BSID-II) in LMIC.16 17 The objective of this analysis was to determine the performance of ASQ as a screening tool for neurodevelopmental impairment when compared with BSID-II for the diagnosis of moderate-to-severe neurodevelopmental impairment among infants at 12 and 18 months of age in LMICs. With ASQ as index test, this diagnostic accuracy study assumed that BSID-II was the reference standard for neurodevelopmental impairment.
Methods
This is a diagnostic study in which data on ASQ performed at 12 and 18 months and BSID-II performed at 18 months of age were collected.
Study participants
Study participants were recruited as part of the First Bites Complementary Feeding trial (CF), a cluster randomised controlled trial implemented in rural communities in the Democratic Republic of Congo, Zambia, Guatemala and Pakistan between October 2008 and January 2011, with a 1-year daily intervention starting at age 6 months and follow-up until 18 months of age. Details of the CF study protocol, including study participant consent, recruitment and its findings have been published previously.18 19
During the study period, a study nurse identified infants born within rural settings of the four study sites from the Global Network Maternal and Newborn Health Registry and approached the mother for consenting purposes when the infant had reached 3–4 months.20 At 6 months of age, all consented study participants were assigned to clusters and randomised to receive either lyophilised beef (intervention arm) or an equicaloric micronutrient fortified rice-soy based micronutrient fortified cereal (control arm) and then followed-up for 12 months, until 18 months of age. During the 12-month follow-up, all study participants were visited on a weekly basis and provided the study food according to their study arm adjusted to the age of the study participant. Anthropometric measurements were obtained at baseline and every 3 months during the study period. Lastly, the socioeconomic status (SES) score was calculated as a composite variable by using a series of economic indicators (eg, dwelling, roof, wall, floor, toilet type, water source) to score the family’s relative wealth.19 The composite was standardised such that the average SES score was 0, below average <0 and above average >0.19 The analysis was restricted to participants who had both the ASQ at 12 and 18 months and BSID-II assessments at 18 months.
Neurodevelopment assessment procedures
All study participants underwent neurodevelopmental assessment by ASQ certified study nurses who were trained in the administration of the screening tool through a standardised training module by trained professionals.18 The study nurses completed the ASQ screening tool through an interview with parents or caregivers at 12 and 18 months of age.18 Additionally, a neurodevelopmental assessment was conducted at 18 months of age using the BSID-II by a different group of trained professionals (BSID-II assessors) who underwent standardised training by experienced professionals (eg, psychologists) to minimise variability within and among clusters.18 Furthermore, to ensure uniformity among sites, video recordings of the BSID-II assessors administering the tool were made and reviewed by the child development consultant.18
Both assessors were blinded from the findings of the other test. Additionally, the BSID-II assessors were only informed of the study participants’ age and were blinded to the participant’s clinical history and study arm. Video recordings of the administration of the BSID-II were made randomly for each examiner, and the recordings were reviewed for quality assurance by an external expert in child development (FJB).18
After completion of the ASQ assessment at both 12 and 18 months of age, the mother was escorted to the health facility for her infant to undergo the BSID-II assessment. The BSID-II assessors in each study site administered the BSID-II directly to the child, in the presence of the mother. If the mother was not available for BSID-II on the scheduled day, she was given another appointment. The BSID-II was administered within ± week of the ASQ administration. Following the assessment, the ASQ and BSID-II test results from the study participants were entered into the database and transferred to the RTI International, North Carolina, USA, for data analysis and interpretation by investigators who were masked to the randomisation arm of the study participants.
Measures
BSID-II: The BSID-II is comprised of a Mental Development Index (MDI), Psychomotor Development Index (PDI) and a Behaviour Rating Scale. It can be used for assessing infants between ages 1 and 42 months.21 This tool underwent minor adaptations when translating it into the local dialects to ensure culturally appropriate objects were used but with the same intention as the original examination; modifications were reviewed and approved by the consultant (FJB).21 If an object in the BSID-II was not common in the environment, we substituted a comparable item that was common for the child.22 Two cut-off points based on an age-standardised normal distribution of scores defined neurodevelopmental delay.21 Moderate neurodevelopmental delay was defined as an MDI or PDI score of <1 SD below the mean (ie, <85 (between 84 and 70)). Severe neurodevelopmental delay was defined as an MDI or PDI score of <2 SDs below the mean (ie, <70).21
ASQ: The ASQ is a tool to assess a child’s development and behaviour by parent report for infants, toddlers and preschool children between the ages of 4–60 months of age. Questions are written at a fourth to sixth grade level allowing most parents to complete the test independently. For this study, the questions were administered by the study nurse to consistently address the varying reading abilities among parents. It is available in multiple languages and was translated into the local dialects for the four study sites as necessary. Each study site made minor adjustments to the questionnaire to adapt to their respective settings. All changes were reviewed and approved by a child development expert (FJB). Throughout the study, periodic video recordings were obtained and reviewed by the external expert consultant (FJB) to assure consistency in test administration by study nurses. The ASQ addresses five domains with six questions per domain with answer options being ‘Yes’, ‘Sometimes’ or ‘Not Yet’.23 The ASQs were scored according to the ASQ User’s Guide using age-specific norms of <2 SDs below the mean of any of the five domains to define a failure.24 Thus, in addition to the ASQ composite score, the following standard cut-offs were set to define positive screening in the ASQ domains: ≤38.7 for communication, ≤35.7 for gross motor, ≤30.7 for fine motor, ≤38.6 for problem solving and ≤38.7 for personal-social.25
Statistical analyses
To evaluate the relationship between ASQ, done at 12 and 18 months, and BSID-II, done at 18 months, results, a cross-tabulation table was constructed for both moderate and severe neurodevelopmental delay to include the ‘positive’ or ‘negative’ results for ASQ and BSID-II tests. ASQ communication, problem-solving, gross motor, fine motor and personal and social domains were compared separately with an overall pass/fail composite outcome of BSID-II MDI or PDI, respectively, defining in turn moderate (<85) and severe (<70) delay. Cohen’s κ was computed to determine the chance of independent agreement between screening and diagnostic classifications. Concurrent validity of the specified ASQ domain score cut-offs in identifying children with BSID-II PDI and/or MDI scores <70 and separately <85 was assessed by calculating sensitivity, specificity and accuracy. In addition, using raw scores without categorisation, the concurrent diagnostic agreement of the ASQ results with the BSID-II assessment was analysed via Spearman rank-order correlation coefficient for ASQ domains scores compared with BSID-II PDI and MDI scores. P values were based on separate χ2 tests. Lastly, since the CF trial found no difference between treatment arms and study participants among both groups were predominantly breast fed with no difference between the two groups, we combined the groups for this analysis and presented the findings based on the BSID-II scores in table 1.19 Statistical analyses were performed using SAS V.9.3 (SAS Institute, Cary, North Carolina, USA).
Baseline demographics by moderate or severe neurodevelopmental delay
Patient and public involvement
Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our research.
Results
Descriptive information
Of the 1062 study participants in the CF trial, 1034 had both ASQ and BSID-II assessment data at 18 months (97.4%). Data were similarly distributed from the four LMIC settings where the CF trial was conducted (online supplemental figure 1). Baseline demographic characteristics of mothers (ie, SES, maternal education, number of living children) and child study participants (ie, sex, birth weight, stunting rates) according to neurodevelopmental outcome are summarised in table 1.
Supplemental material
There were 127 (12.3%) and 89 (8.7%) infants with moderate delay and 6 (0.6%) and 7 (0.7%) with severe delay by the BSID-II MDI and PDI, respectively. Ninety-four (49.5%) infants had at least moderate neurodevelopmental delays while five (38.5%) infants met the stricter criterion for severe neurodevelopmental delays were men (table 1). Fifty-one (28.8%) mothers to infants who had moderate or severe neurodevelopmental delays and seven (53.8%) mothers to infants who had severe neurodevelopmental delays had no formal education. Lastly, families of infants with moderate or severe neurodevelopmental delays had an SES composite score of 0.03 and the subset of infants with severe neurodevelopmental delays had a composite score of −0.2 (table 1).
Screening performance for ASQ at 12 months of age
The sensitivity, specificity and accuracy of ASQ at 12 months to diagnose moderate and severe neurodevelopmental outcomes at 18 months are shown in table 2 and table 3, respectively. At 12 months, all five ASQ domains had moderate specificity (70.5%–85.8%), with one (gross motor) of the five domains having medium sensitivity (51.9%) and the rest of the four being low (20.1%–27.0%). All five domains had 67.1%–73.9% correctly characterised cases for BSID-II MDI and/or PDI <85 (table 2). Additionally, diagnostic accuracy was similar for the same five domains for severe delay diagnosis defined by MDI and/or PDI (<70), with specificity ranging between 66.9% and 84.9%, sensitivity for two (ie, communications and gross motor) of the five domains ranging between 53.9%–61.5%, and 66.8%–84.3% for correctly characterised cases for all five domains (table 3). At 12 months, the ASQ composite scores for both BSID-II MDI and/or PDI <85 and <70 were low for all test characteristics (table 3). Furthermore, at 12 months, the correlations between ASQ gross motor domain and BSID-II MDI (0.14; p<0.0001) and BSID-II PDI (0.24; p<0.0001) were the strongest correlations found (table 4). Lastly, at 18 months, the correlations between the ASQ communication domain and BSID-II MDI (0.38; p<0.0001) and between ASQ gross motor domain and BSID-II PDI (0.33; p<0.0001) were the strongest correlations found (table 4).
Test characteristics of the 12-month ASQ for detection of a 18-month BSID-II MDI and/or PDI scores <85 (moderate or severe neurodevelopmental delay)
Test characteristics of the 12-month ASQ for detection of a 18-month BSID-II MDI and/or PDI Scores <70 (severe neurodevelopmental delay)
Correlation coefficient between 18-month BSID-II scores and ASQ domain scores at 12 and 18 months
Screening performance for ASQ at 18 months of age
The sensitivity, specificity and accuracy of ASQ at 18 months to diagnose moderate and severe neurodevelopmental outcomes are reported in table 5 and table 6. At 18 months, four of the five ASQ domains had high specificity (93.60%–98.34%), low sensitivity (9.47%–30.00%) and 71.7%–85.8% correctly characterised cases for BSID-II MDI and/or PDI <85 (table 5). Diagnostic accuracy was high for the same four domains for severe delay diagnosis defined by MDI and/or PDI (<70), ranging between 92.81% and 96.46% for specificity and 92.0% and 95.5% for correctly negative cases except for communications (71.1%), but sensitivity was 23.08%–46.15% (table 6). Communication was the only ASQ domain that had a low specificity with both at least moderate (75.83%) and severe (71.26%) neurodevelopmental delays. Similarly, at 18 months, the ASQ composite scores for both BSID-II MDI and/or PDI <85 and <70 were low for all screening evaluation characteristics (tables 5 and 6).
Test characteristics of the ASQ at 18 months for detection of a BSID-II MDI and/or PDI scores <85 (moderate or severe neurodevelopmental delay) at 18 months of age
Test characteristics of the ASQ at 18 months for detection of a BSID-II MDI and/or PDI scores <70 (severe neurodevelopmental delay) at 18 months of age
Discussion
This large study cohort in four LMIC showed that four of the five domains of the ASQ had high specificity for identification of infants with severe neurodevelopmental delays at age 18 months. However, for the 12-month ASQ, specificity was moderate for all five domains. The communication domain of the ASQ at both 12 and 18 months of age had moderate specificity at best for the diagnosis of moderate neurodevelopmental delay. However, the fair agreement and strength of correlation between 18-month ASQ communication domain and MDI could be considered useful to stratify the risk of neurodevelopmental delay with the ASQ communication domain. While ASQ has been validated against diagnostic evaluation instruments, including BSID, it has not been validated against the BSID-II among populations in LMIC settings.7 26 Some previous studies that compared results of parent-administered screening tools such as the ASQ with the results of professionally administered neurodevelopmental evaluations such as the BSID-II suggest that ASQ is inaccurate to classify and predict poor neurodevelopmental outcomes.13 Other studies have shown ASQ as having high sensitivity when compared with BSID-III at 24 months.27 The variability between the findings could be due to the diversity among the study sites and the possible inadequate adaptation of these tools to the local language and settings.
While the BSID remains the most used tool for diagnosis of neurodevelopmental delays among infants, evaluations performed in the early years have not always been found to be predictive of future developmental outcomes.28–31 While the BSID diagnostic test has been culturally adapted for use in some countries, it is not widely used in LMIC settings due to the skills required to perform the test and the duration of the test.6 32 Alternatives are needed. To this end, the ASQ allows both family members and healthcare providers to screen the infant either at home or the health facility using a much simpler tool.6 Thus, the ASQ could be used in many settings where it is not feasible to identify infants with suspected neurodevelopmental delays using BSID. Broad screening at an early age is imperative for timely referrals to be possible for early interventions that have the possibility to mitigate neurodevelopmental delay.33 Studies have shown that screening results based on report by mothers with a low level of education match well those of professionals.34 Additionally, a parent-administered assessment tool would be beneficial and cost-efficient as it will provide the parents with the opportunity to closely monitor their infant’s development and increase their involvement in the child’s upbringing.6 Therefore, based on the evidence generated out of this study, it may be reasonable that parents may need specific training to match those results obtained by certified providers. It is important to note that since this study ended, ASQ-3 and BSID-III have been published, where ASQ has new cut-off scores and BSID-III is more aligned with ASQ domains.35 36 The cut-offs used in our analysis were based on published findings, therefore, for this analysis, our aim was to determine if the ASQ screening tool with the established cut-offs was a reasonable assessment tool. We did not aim to explore new cut-off values as that was outside the scope of the current study, one which may require a much larger sample size. The Chinese adaptation of ASQ-3 was validated against the BSID-III where the overall sensitivity and specificity of ASQ-3 was 76.52% and 40.97%, respectively.37 However, while future research to test this would be valuable, the new cut-off scores are unlikely to invalidate our findings regarding the utility of the ASQ as a useful and feasible screening tool in low-resource settings.
Use of the ASQ by the healthcare provider in the infant’s neurodevelopment assessment provides the healthcare system with the opportunity to screen for delays at an early stage and implement prevention strategies. Unlike other studies, which found agreement only between ASQ and MDI BSID-II,17 our study showed moderate to strong agreement between ASQ and the composite score of both BSID-II MDI and PDI <70 and <85 at 18 months. Although several studies have been published discussing the discrepancy between the ASQ and BSID-II assessment results, these are the first data comparing the diagnostic validity of the ASQ in a large generalisable population from four diverse countries in LMIC settings.26 38 39 The recommended minimum standards for good screening tests suggest that specificity should be greater than 70%–80%.12 When this standard is applied to the ASQ as used here, then all five domains at both 12 (except gross motor at 12 months among BSID-II MDI and/or PDI <70) and 18 months of age were within the acceptable range, but they were not within the expected minimum sensitivity rate of 80%.12 40
Among the strengths of this study are, first, that this is the largest study with data on both ASQ and BSID-II at 18 months of age from LMICs. Second, because ASQ and BSID-II assessments were done by two different groups of certified experts, neither the results of the screening test (ie, ASQ) nor the reference standard (ie, BSID-II), were biased. Last, since both assessments were done by certified personnel, we do not believe that reliability was an issue in this study. Therefore, we did not examine the intrarater reliability of the examiners. This study has limitations including that, while both the ASQ and BSID-II assessment tools were used by certified providers who were fluent in the local languages, the tools have not been validated in the local languages. Second, ASQ was not completed by the parent or guardian as is its original intent. However, we are providing evidence of the validity of ASQ in respect to BSID-II when done by certified providers, allowing us to see the maximum potential of this test. Validation of ASQ in the local cultural context in the manner used here could increase its value for international use.
In summary, our study concludes that the ASQ at 18 months of age performs with acceptable accuracy for identification of infants with moderate-to-severe neurodevelopmental outcomes, indicating it may be a useful screening tool among infants from LMIC settings. The 18-month ASQ assessment provides a fast and cost-effective alternative to existing standardised assessments.
Data availability statement
Data are available upon reasonable request. Data will be available upon request to the corresponding author.
Ethics statements
Patient consent for publication
Ethics approval
This research study was reviewed for approval by the Institutional Review Board (IRB) of each participating institution (USA: UAB IRB—F0806190150; UNC IRB—08-0594; Drexel University IRB—245353; University of Colorado IRB—07-1045; Zambia: UNZA BREC—002-01-05B; Pakistan: Aga Khan University—939-CHS-ERC-08; DRC: KSPH—ESP/CE/019/2008; Guatemala: Comite de Etica Independiente—07-1045). The IRBs are registered at OHRP with FWAs.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @AlbertManasyan
Deceased FJB died on September 23, 2020
Contributors AM drafted the initial manuscript, supervised the data collection, interpreted the analysis results, and reviewed and revised the manuscript. AAS reviewed and revised the manuscript. TN conducted the data analysis and reviewed and revised the manuscript. EC, MM, ATK, SS, FN, RLG and FJB supervised the data collection, and critically reviewed the manuscript for important intellectual content. KMH and NFK conceptualised and designed the study, designed the data collection instruments, interpreted the study findings, and reviewed and revised the manuscript. NG and EM supervised the data collection and analysis, designed the data collection instruments, and reviewed and revised the manuscript. JLW and FJB reviewed and revised the manuscript and critically reviewed the manuscript for important intellectual content. MK-T reviewed the manuscript for important intellectual content. WAC reviewed and revised the manuscript, interpreted the findings, and provided important intellectual content. All authors approved the final manuscript as submitted and agreed to be accountable for all aspects of the work. AM is responsible for the overall content as the guarantor.
Funding Research reported in this publication was supported by the Eunice Kennedy Shriver National Institute of Child Health & Human Development of the National Institutes of Health under grants (U01 HD043464) and the Office of Dietary Supplements. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.