Economic evaluation research in the context of Child Welfare policy: A structured literature review and recommendations☆
Introduction
In 2008, an estimated 3.3 million referrals were made to Child Protective Service agencies regarding child abuse and neglect in the United States. Almost two thirds of these allegations were investigated by Child Welfare agencies and 23.7% were substantiated (772,000 children) (“Child Maltreatment 2008,” 2010). The total cost of the US Child Welfare system reaches nearly $20 billion, and the type and quality of services provided to children in the system have profound implications for their physical and mental well being, their chance for stable and permanent home environments, government budgets and broader societal costs. Yet, despite the continuing development of effective, evidence-based interventions that could improve the quality of the services offered by the system, such interventions have not been widely adopted.
One reason for this disappointing adoption rate is that, in a context of ongoing program operations, evidence-based interventions’ superior effectiveness must be considered in relation to how well they serve specific policy objectives, and in relation to any additional resources required to implement and maintain the interventions at levels ensuring continued effectiveness. As Child Welfare decision makers contemplate adopting evidence-based interventions, they must overcome uncertainty as to whether the long-term benefits and averted costs of evidence-based interventions justify the investment.
The tools of decision science, cost-effectiveness analysis, and cost-benefit analysis—here generally termed economic evaluation—are well suited to supporting Child Welfare policymakers in decision making processes that they confront in the complex environment of real-world operations (Gold, 1996). Decision analyses are intended to aid decision makers in best achieving a set of objectives (e.g., child safety, permanence, and well-being). Such analyses identify policies and interventions that are most likely to be successful given current uncertainties (e.g., how effective an intervention in a given system's population). In support of the overall decision analysis, cost-effectiveness and cost-benefit analyses can be used to explicitly acknowledge the reality of budget constraints and other limited system resources, identifying those interventions that can feasibly achieve the decision makers’ objectives. Such analyses typically involve assessing the costs and benefits of multiple interventions, often using computer-based models that simulate the long-term, systemic impact of each specified decision alternative, and comparing them to provide recommendations about the best action to take. Such analyses provide their greatest value when they are employed in an iterative fashion, allowing policymakers to consider a variety of what-if scenarios and to evaluate multiple decisions (e.g., how best to prevent foster care placement disruptions and how best to promote permanency) holistically. Decision makers can then realistically judge the likelihood that their decisions will achieve their objectives, given the additional challenges of multiple funding streams, including Title IV-E and Medicaid, as well as the need to develop working alliances with sister agencies such as Child Mental Health that are likely to be delivering the evidenced based interventions.
Such analyses have been used successfully to consider complex decisions involving prevalent chronic health conditions with long time courses and multiple outcomes and interventions. Capitalizing on continued methodological and data advances, studies have considered the prevention and management of HIV/AIDS, cardiovascular disease, diabetes, HPV and cervical cancer, as well as other cancers (Frazier et al., 2000, Gaspoz et al., 2002, Goldhaber-Fiebert et al., 2008, Sanders et al., 2005, Tosteson et al., 2008). For example, cost-effectiveness analyses have informed subsequent policy recommendations in the context of HIV/AIDS prevention programs. Relying in part on model-based cost-effectiveness analyses, the Institute of Medicine (IOM) recommended a shift from funding programs based on high current AIDS prevalence to targeting cost-effective prevention efforts to subgroups at high risk of infection such as needle exchange programs for injection drug users (Ruiz & Institute of Medicine (US). Committee on HIV Prevention Strategies in the United States, 2001).
Recently, the IOM issued a document describing the methodological approaches and challenges of applying economic evaluation in the context of childhood programs generally (National Research Council (US). Committee on Strengthening Benefit-Cost Methodology for the Evaluation of Early Childhood Interventions, Beatty, & National Research Council (US). Board on Children Youth and Families, 2009). It reviewed numerous approaches to generalizing and extending outcomes measured by empirical studies in ways that are accessible and useful for policy makers. These approaches, including modeling, have a number of important strengths relevant to Child Welfare decision makers seeking to move beyond “adopt” versus “do not adopt” decisions made as if in a vacuum and without considering the wider systemic and policy context. For example, the advantages of modeling approaches include: (1) mapping the efficacies measured in randomized trials to the effectiveness likely to be observed in real-world populations needed for comparative effectiveness; (2) projecting intermediate outcomes to long-term, policy-relevant outcomes; and (3) considering complex interactions between multiple interventions; (4) facilitating their easy repetition, in scenarios varied to take account of evolving policy maker concerns. It is important to note that the successful application of economic evaluation methods and modeling to support them requires expertise and guidance such as that provided by the IOM.
Provided with key sources of input, economic evaluations can use computer simulations to address alternative scenarios envisioned by policy makers. There is now an important opportunity to support current decision making by addressing current research gaps and evaluating the cost-effectiveness of interventions targeting the Child Welfare system, using nationally representative longitudinal data on children in Child Welfare, data on evidence-based interventions, and advances in decision-analytic simulation modeling. The aims of this study were to: (1) identify prior guidance and reviews calling for the use of decision-analytic methods in the Child Welfare context; (2) characterize the current literature on the costs and benefits of interventions for children at risk of and involved with the Child Welfare system; and (3) identify areas within this domain where additional economic evaluation research could provide the most value, incorporating findings on costs and return on investment into the promotion of evidence-based practice. Fulfilling these aims will help us to build effectively on existing knowledge, particularly with regard to prior analyses of evidence-based interventions in the context of Child Welfare. This study is a stepping stone to answering questions of ultimate concern: how best to maximize the return on investment and promote positive child outcomes involved with the system.
Section snippets
Methods
The structured literature review focused on research concerning the costs and effectiveness of interventions for children at risk of and already involved with the Child Welfare system. Searches targeted databases of peer-reviewed articles as well as repositories of non-peer-reviewed “gray” literature such as government reports and foundation publications. Both reviews of previous research and original research articles were included. Reviews were included because in addition to providing a view
Searches and articles identified
The structured literature review of economic evaluation research in the context of the Child Welfare system yielded 49 articles that met all inclusion criteria (Fig. 1). Searches of PubMed MEDLINE and JSTOR resulted in 930 articles whose titles and abstracts were reviewed. Of these, 109 articles were selected for full-text review. Other databases, repositories, and bibliographies yielded 1,710 items, resulting in 50 articles for full-text review. From this combination of searches, the full-text
Conclusions
To improve the well-being of children and families it serves and that of the wider society, the Child Welfare system must be concerned with improving its services. One path to upgrading services is for Child Welfare administrators is to implement effective, evidence-based interventions that could be delivered within the system to improve Child Welfare-related outcomes. However, competing demands of an already overburdened system, multiple funding streams, the need to work with sister agencies
Acknowledgement
The authors gratefully acknowledge the research assistance of Rachel Rubinfeld.
References (83)
Foster home care is more cost-effective than shelter care: Serious questions continue to be raised about the utility of group care in child welfare services
Child Abuse & Neglect
(2005)- et al.
The need for economic analysis in research on child maltreatment
Child Abuse & Neglect
(2006) - et al.
Relative cost-effectiveness of treatments for adolescent depression: 36-week results from the TADS randomized trial
Journal of the American Academy of Child and Adolescent Psychiatry
(2009) Costs and effectiveness of interventions in child maltreatment
Child Abuse & Neglect
(1990)- et al.
Is stacking intervention components cost-effective? An analysis of the Incredible Years program
Journal of the American Academy of Child and Adolescent Psychiatry
(2007) - et al.
Overcoming challenges for the economic evaluation of investments in children's health
Academic Pediatrics
(2009) - et al.
Family support and parenting education in the home: An effective extension of clinic-based preventive health care services for poor children
The Journal of Pediatrics
(1989) - et al.
A cost/effectiveness evaluation of lay therapy treatment for child abusing and high risk parents
Child Abuse & Neglect
(1986) Current challenges and future research in measuring preferences for pediatric health outcomes
The Journal of Pediatrics
(2009)- et al.
Benefits and costs of prevention and early intervention programs for youth
(2004)
Intensive family preservation programs: Program fidelity influences effectiveness – revised
The comparative costs and benefits of programs to reduce crime
Economic analysis of a child abuse and neglect treatment program
Child Welfare
Role of home visiting in improving parenting and health in families at risk of abuse and neglect: Results of a multicentre randomised controlled trial and economic evaluation
Archives of Disease in Childhood
Economic evaluation of home visiting programs
The Future of Children
Cost-effectiveness of selective serotonin reuptake inhibitors and routine specialist care with and without cognitive behavioural therapy in adolescents with major depression
British Journal of Psychiatry
Improving decision analyses: Parent preferences (utility values) for pediatric health outcomes
The Journal of Pediatrics
Multidimensional treatment foster care for girls in the juvenile justice system: 2-year follow-up of a randomized clinical trial
Journal of Consulting and Clinical Psychology
Prevention of behavior problems for children in foster care: Outcomes and mediation effects
Prevention Science
Using a specialized foster-care community treatment model for children and adolescents leaving the state mental-hospital
Journal of Community Psychology
Differences in risk factors and adjustment for male and female deliquents in treatment foster care
Journal of Child and Family Studies
Measuring costs of child abuse and neglect: A mathematic model of specific cost estimations
Journal of Health and Human Services Administration
Large group community-based parenting programs for families of preschoolers at risk for disruptive behavior disorders – utilization, cost-effectiveness, and outcome
Journal of Child Psychology and Psychiatry and Allied Disciplines
Family-centered, home-based intervention project for protective services client (Final Report)
Methods for the economic evaluation of health care programmes
Economic evaluation in health care: Merging theory with practice
What works? A study of effective early childhood mental health consultation programs
Family management and deviant peer association as mediators of the impact of treatment condition on youth antisocial behavior
Journal of Consulting and Clinical Psychology
Iterations of the SafeCare model—An evidence-based child maltreatment prevention program
Behavior Modification
Refining the costs analyses of the Fort Bragg evaluation: The impact of cost offset and cost shifting
Mental Health Services Research
Public costs of better mental health services for children and adolescents
Psychiatric Services
Family-centered, home-based services in child protection—A review of the research
Social Service Review
Cost-effectiveness of screening for colorectal cancer in the general population
JAMA
Cost effectiveness of aspirin, clopidogrel, or both for secondary prevention of coronary heart disease
The New England Journal of Medicine
Cited by (27)
Agent-based simulation and child protection systems: Rationale, implementation, and verification
2024, Child Abuse and NeglectExpanding the repertoire of evaluation tools so that evaluation recommendations can assist nonprofits to enhance strategic planning and design of program operations
2021, Evaluation and Program PlanningCitation Excerpt :More recently, a review of 30 economic evaluations in mostly high-income countries highlighted that most of the studies were considered to be of poor or average quality (Madsen, Eddleston, Hansen, & Konradsen, 2018). This could be quite detrimental as inaccurate cost data can seriously mislead decision-making (Franklin, Lomas, Walker, & Young, 2019). The limited use of cost studies has been linked to a number of issues (see Table 1).
The costs of implementing and sustaining a trauma and mental health screening tool in a state child welfare system
2020, Children and Youth Services ReviewLess money, more problems: How changes in disposable income affect child maltreatment
2017, Child Abuse and NeglectCitation Excerpt :The analysis provides additional evidence regarding the relationship between income and child maltreatment and identifies an additional economic indicator that may be related to child abuse and neglect. This is important for understanding the predictors of child maltreatment and for responding to the call for more economic analysis in child welfare research (Corso & Lutzker, 2006; Goldhaber-Fiebert, Snowden, Wulczyn, Landsverk, & Horwitz, 2011). Child maltreatment has been written about since the times of the Romans (Holt, 1994).
Development and feasibility of a sibling intervention for youth in foster care
2014, Evaluation and Program PlanningEconomic analysis of social services for the elderly in Serbia: Two sides of the same coin
2014, Evaluation and Program PlanningCitation Excerpt :The method is also used for the analysis and implementation of social protection services. Goldhaber-Fiebert, Snowden, Wulczyn, Landsverk, and Horwitz (2011) underline the need to apply economic analysis in child welfare. According to them, even economically developed countries, such as the USA, have to use economic analyses, such as a cost–benefit analysis and a cost-effectiveness analysis, due to the number of children (beneficiaries) and the expenses of the services provided.
- ☆
This work was supported by funding from the NIMH (P30-MH074678; PI: J. Landsverk). Dr. Goldhaber-Fiebert was supported in part by an NIH NIA Career Development Award (K01 AG037593-01A1: PI: Goldhaber-Fiebert).