Elsevier

Child Abuse & Neglect

Volume 35, Issue 9, September 2011, Pages 722-740
Child Abuse & Neglect

Economic evaluation research in the context of Child Welfare policy: A structured literature review and recommendations

https://doi.org/10.1016/j.chiabu.2011.05.012Get rights and content

Abstract

Objectives

With over 1 million children served by the US Child Welfare system at a cost of $20 billion annually, this study examines the economic evaluation literature on interventions to improve outcomes for children at risk for and currently involved with the system, identifies areas where additional research is needed, and discusses the use of decision-analytic modeling to advance Child Welfare policy and practice.

Methods

The review included 19 repositories of peer-reviewed and non-peer-reviewed “gray” literatures, including items in English published before November, 2009. Original research articles were included if they evaluated interventions based on costs and outcomes. Review articles were included to assess the relevance of these techniques over time and to highlight the increasing discussion of methods needed to undertake such research. Items were categorized by their focus on: interventions for the US Child Welfare system; primary prevention of entry into the system; and use of models to make long-term projections of costs and outcomes.

Results

Searches identified 2,640 articles, with 49 ultimately included (19 reviews and 30 original research articles). Between 1988 and 2009, reviews consistently advocated economic evaluation and increasingly provided methodological guidance. 21 of the original research articles focused on Child Welfare, while 9 focused on child mental health. Of the 21 Child Welfare articles, 81% (17) focused on the US system. 47% (8/17) focused exclusively on primary prevention, though 83% of the US system, peer-reviewed articles focused exclusively on prevention (5/6). 9 of the 17 articles included empirical follow-up (mean sample size: 264 individuals; mean follow-up: 3.8 years). 10 of the 17 articles used modeling to project longer-term outcomes, but 80% of the articles using modeling were not peer-reviewed. Although 60% of modeling studies included interventions for children in the system, all peer-reviewed modeling articles focused on prevention.

Conclusions

Methodological guidance for economic evaluations in Child Welfare is increasingly available. Such analyses are feasible given the availability of nationally representative data on children involved with Child Welfare and evidence-based interventions.

Practice implications

Policy analyses considering the long-term costs and effects of interventions to improve Child Welfare outcomes are scarce, feasible, and urgently needed.

Introduction

In 2008, an estimated 3.3 million referrals were made to Child Protective Service agencies regarding child abuse and neglect in the United States. Almost two thirds of these allegations were investigated by Child Welfare agencies and 23.7% were substantiated (772,000 children) (“Child Maltreatment 2008,” 2010). The total cost of the US Child Welfare system reaches nearly $20 billion, and the type and quality of services provided to children in the system have profound implications for their physical and mental well being, their chance for stable and permanent home environments, government budgets and broader societal costs. Yet, despite the continuing development of effective, evidence-based interventions that could improve the quality of the services offered by the system, such interventions have not been widely adopted.

One reason for this disappointing adoption rate is that, in a context of ongoing program operations, evidence-based interventions’ superior effectiveness must be considered in relation to how well they serve specific policy objectives, and in relation to any additional resources required to implement and maintain the interventions at levels ensuring continued effectiveness. As Child Welfare decision makers contemplate adopting evidence-based interventions, they must overcome uncertainty as to whether the long-term benefits and averted costs of evidence-based interventions justify the investment.

The tools of decision science, cost-effectiveness analysis, and cost-benefit analysis—here generally termed economic evaluation—are well suited to supporting Child Welfare policymakers in decision making processes that they confront in the complex environment of real-world operations (Gold, 1996). Decision analyses are intended to aid decision makers in best achieving a set of objectives (e.g., child safety, permanence, and well-being). Such analyses identify policies and interventions that are most likely to be successful given current uncertainties (e.g., how effective an intervention in a given system's population). In support of the overall decision analysis, cost-effectiveness and cost-benefit analyses can be used to explicitly acknowledge the reality of budget constraints and other limited system resources, identifying those interventions that can feasibly achieve the decision makers’ objectives. Such analyses typically involve assessing the costs and benefits of multiple interventions, often using computer-based models that simulate the long-term, systemic impact of each specified decision alternative, and comparing them to provide recommendations about the best action to take. Such analyses provide their greatest value when they are employed in an iterative fashion, allowing policymakers to consider a variety of what-if scenarios and to evaluate multiple decisions (e.g., how best to prevent foster care placement disruptions and how best to promote permanency) holistically. Decision makers can then realistically judge the likelihood that their decisions will achieve their objectives, given the additional challenges of multiple funding streams, including Title IV-E and Medicaid, as well as the need to develop working alliances with sister agencies such as Child Mental Health that are likely to be delivering the evidenced based interventions.

Such analyses have been used successfully to consider complex decisions involving prevalent chronic health conditions with long time courses and multiple outcomes and interventions. Capitalizing on continued methodological and data advances, studies have considered the prevention and management of HIV/AIDS, cardiovascular disease, diabetes, HPV and cervical cancer, as well as other cancers (Frazier et al., 2000, Gaspoz et al., 2002, Goldhaber-Fiebert et al., 2008, Sanders et al., 2005, Tosteson et al., 2008). For example, cost-effectiveness analyses have informed subsequent policy recommendations in the context of HIV/AIDS prevention programs. Relying in part on model-based cost-effectiveness analyses, the Institute of Medicine (IOM) recommended a shift from funding programs based on high current AIDS prevalence to targeting cost-effective prevention efforts to subgroups at high risk of infection such as needle exchange programs for injection drug users (Ruiz & Institute of Medicine (US). Committee on HIV Prevention Strategies in the United States, 2001).

Recently, the IOM issued a document describing the methodological approaches and challenges of applying economic evaluation in the context of childhood programs generally (National Research Council (US). Committee on Strengthening Benefit-Cost Methodology for the Evaluation of Early Childhood Interventions, Beatty, & National Research Council (US). Board on Children Youth and Families, 2009). It reviewed numerous approaches to generalizing and extending outcomes measured by empirical studies in ways that are accessible and useful for policy makers. These approaches, including modeling, have a number of important strengths relevant to Child Welfare decision makers seeking to move beyond “adopt” versus “do not adopt” decisions made as if in a vacuum and without considering the wider systemic and policy context. For example, the advantages of modeling approaches include: (1) mapping the efficacies measured in randomized trials to the effectiveness likely to be observed in real-world populations needed for comparative effectiveness; (2) projecting intermediate outcomes to long-term, policy-relevant outcomes; and (3) considering complex interactions between multiple interventions; (4) facilitating their easy repetition, in scenarios varied to take account of evolving policy maker concerns. It is important to note that the successful application of economic evaluation methods and modeling to support them requires expertise and guidance such as that provided by the IOM.

Provided with key sources of input, economic evaluations can use computer simulations to address alternative scenarios envisioned by policy makers. There is now an important opportunity to support current decision making by addressing current research gaps and evaluating the cost-effectiveness of interventions targeting the Child Welfare system, using nationally representative longitudinal data on children in Child Welfare, data on evidence-based interventions, and advances in decision-analytic simulation modeling. The aims of this study were to: (1) identify prior guidance and reviews calling for the use of decision-analytic methods in the Child Welfare context; (2) characterize the current literature on the costs and benefits of interventions for children at risk of and involved with the Child Welfare system; and (3) identify areas within this domain where additional economic evaluation research could provide the most value, incorporating findings on costs and return on investment into the promotion of evidence-based practice. Fulfilling these aims will help us to build effectively on existing knowledge, particularly with regard to prior analyses of evidence-based interventions in the context of Child Welfare. This study is a stepping stone to answering questions of ultimate concern: how best to maximize the return on investment and promote positive child outcomes involved with the system.

Section snippets

Methods

The structured literature review focused on research concerning the costs and effectiveness of interventions for children at risk of and already involved with the Child Welfare system. Searches targeted databases of peer-reviewed articles as well as repositories of non-peer-reviewed “gray” literature such as government reports and foundation publications. Both reviews of previous research and original research articles were included. Reviews were included because in addition to providing a view

Searches and articles identified

The structured literature review of economic evaluation research in the context of the Child Welfare system yielded 49 articles that met all inclusion criteria (Fig. 1). Searches of PubMed MEDLINE and JSTOR resulted in 930 articles whose titles and abstracts were reviewed. Of these, 109 articles were selected for full-text review. Other databases, repositories, and bibliographies yielded 1,710 items, resulting in 50 articles for full-text review. From this combination of searches, the full-text

Conclusions

To improve the well-being of children and families it serves and that of the wider society, the Child Welfare system must be concerned with improving its services. One path to upgrading services is for Child Welfare administrators is to implement effective, evidence-based interventions that could be delivered within the system to improve Child Welfare-related outcomes. However, competing demands of an already overburdened system, multiple funding streams, the need to work with sister agencies

Acknowledgement

The authors gratefully acknowledge the research assistance of Rachel Rubinfeld.

References (83)

  • S. Aos et al.

    Intensive family preservation programs: Program fidelity influences effectiveness – revised

    (2006)
  • S. Aos et al.

    The comparative costs and benefits of programs to reduce crime

    (2001)
  • K.A. Armstrong

    Economic analysis of a child abuse and neglect treatment program

    Child Welfare

    (1983)
  • J. Barlow et al.

    Role of home visiting in improving parenting and health in families at risk of abuse and neglect: Results of a multicentre randomised controlled trial and economic evaluation

    Archives of Disease in Childhood

    (2007)
  • W.S. Barnett

    Economic evaluation of home visiting programs

    The Future of Children

    (1993)
  • S. Byford et al.

    Cost-effectiveness of selective serotonin reuptake inhibitors and routine specialist care with and without cognitive behavioural therapy in adolescents with major depression

    British Journal of Psychiatry

    (2007)
  • A.E. Carroll et al.

    Improving decision analyses: Parent preferences (utility values) for pediatric health outcomes

    The Journal of Pediatrics

    (2009)
  • P. Chamberlain et al.

    Multidimensional treatment foster care for girls in the juvenile justice system: 2-year follow-up of a randomized clinical trial

    Journal of Consulting and Clinical Psychology

    (2007)
  • P. Chamberlain et al.

    Prevention of behavior problems for children in foster care: Outcomes and mediation effects

    Prevention Science

    (2008)
  • P. Chamberlain et al.

    Using a specialized foster-care community treatment model for children and adolescents leaving the state mental-hospital

    Journal of Community Psychology

    (1991)
  • P. Chamberlain et al.

    Differences in risk factors and adjustment for male and female deliquents in treatment foster care

    Journal of Child and Family Studies

    (1994)
  • Child Maltreatment 2008. (2010). Washington, D.C.: US Department of Health and Human Services, Administration for...
  • Child Welfare Monitoring. (2010). Retrieved October 18, 2010, from...
  • C. Conrad

    Measuring costs of child abuse and neglect: A mathematic model of specific cost estimations

    Journal of Health and Human Services Administration

    (2006)
  • C.E. Cunningham et al.

    Large group community-based parenting programs for families of preschoolers at risk for disruptive behavior disorders – utilization, cost-effectiveness, and outcome

    Journal of Child Psychology and Psychiatry and Allied Disciplines

    (1995)
  • Dalziel, K., & Segal, L. (2010). Economic evaluation of harm prevention: What are the special challenges? In Economic...
  • Daro, D. (1988). Confronting child abuse: Research for effective program design. New York; London: Free Press; Collier...
  • L. Dennis-Small et al.

    Family-centered, home-based intervention project for protective services client (Final Report)

    (1986)
  • M.F. Drummond et al.

    Methods for the economic evaluation of health care programmes

    (2005)
  • M.F. Drummond et al.

    Economic evaluation in health care: Merging theory with practice

    (2001)
  • F. Duran et al.

    What works? A study of effective early childhood mental health consultation programs

    (2009)
  • J.M. Eddy et al.

    Family management and deviant peer association as mediators of the impact of treatment condition on youth antisocial behavior

    Journal of Consulting and Clinical Psychology

    (2000)
  • A. Edwards et al.

    Iterations of the SafeCare model—An evidence-based child maltreatment prevention program

    Behavior Modification

    (2008)
  • Estimating Child Welfare Service Costs: Methods Developed for the Evaluation of Family Preservation and Reunification...
  • Foster Care: Incomplete Implementation of the Reforms and Unknown Effectiveness. Report to Congressional Requesters....
  • E.M. Foster et al.

    Refining the costs analyses of the Fort Bragg evaluation: The impact of cost offset and cost shifting

    Mental Health Services Research

    (2000)
  • E.M. Foster et al.

    Public costs of better mental health services for children and adolescents

    Psychiatric Services

    (2005)
  • Foster, E. M., Jones, D., & G, C. P. P. R. (2006). Can a costly intervention be cost-effective? An analysis of violence...
  • H. Frankel

    Family-centered, home-based services in child protection—A review of the research

    Social Service Review

    (1988)
  • A.L. Frazier et al.

    Cost-effectiveness of screening for colorectal cancer in the general population

    JAMA

    (2000)
  • J.M. Gaspoz et al.

    Cost effectiveness of aspirin, clopidogrel, or both for secondary prevention of coronary heart disease

    The New England Journal of Medicine

    (2002)
  • Cited by (27)

    • Expanding the repertoire of evaluation tools so that evaluation recommendations can assist nonprofits to enhance strategic planning and design of program operations

      2021, Evaluation and Program Planning
      Citation Excerpt :

      More recently, a review of 30 economic evaluations in mostly high-income countries highlighted that most of the studies were considered to be of poor or average quality (Madsen, Eddleston, Hansen, & Konradsen, 2018). This could be quite detrimental as inaccurate cost data can seriously mislead decision-making (Franklin, Lomas, Walker, & Young, 2019). The limited use of cost studies has been linked to a number of issues (see Table 1).

    • Less money, more problems: How changes in disposable income affect child maltreatment

      2017, Child Abuse and Neglect
      Citation Excerpt :

      The analysis provides additional evidence regarding the relationship between income and child maltreatment and identifies an additional economic indicator that may be related to child abuse and neglect. This is important for understanding the predictors of child maltreatment and for responding to the call for more economic analysis in child welfare research (Corso & Lutzker, 2006; Goldhaber-Fiebert, Snowden, Wulczyn, Landsverk, & Horwitz, 2011). Child maltreatment has been written about since the times of the Romans (Holt, 1994).

    • Economic analysis of social services for the elderly in Serbia: Two sides of the same coin

      2014, Evaluation and Program Planning
      Citation Excerpt :

      The method is also used for the analysis and implementation of social protection services. Goldhaber-Fiebert, Snowden, Wulczyn, Landsverk, and Horwitz (2011) underline the need to apply economic analysis in child welfare. According to them, even economically developed countries, such as the USA, have to use economic analyses, such as a cost–benefit analysis and a cost-effectiveness analysis, due to the number of children (beneficiaries) and the expenses of the services provided.

    View all citing articles on Scopus

    This work was supported by funding from the NIMH (P30-MH074678; PI: J. Landsverk). Dr. Goldhaber-Fiebert was supported in part by an NIH NIA Career Development Award (K01 AG037593-01A1: PI: Goldhaber-Fiebert).

    View full text