The theory-driven approach to validity

doi:10.1016/0149-7189(87)90025-5

Evaluation and Program Planning

Volume 10, Issue 1, 1987, Pages 95-103

https://doi.org/10.1016/0149-7189(87)90025-5 Get rights and content

Abstract

Current approaches for dealing with validity tend to focus on one type of validity and play down the importance of others. For example, internal validity is treated by Campbell and Stanley as “sine qua non,” whereas, Cronbach judges external validity to be prime importance in research. However, these single validity oriented approaches may achieve one type of validity at the expense of other types of validity.

As an alternative, we propose a theory-driven approach to validity. Our central argument is that a model or theory should be formulated in a program evaluation and the modeling process should include the identification of potential threats to validity in research. In this sense, the purpose of applying a technique such as randomization is to strengthen rather than to replace the model or theory. That is, program evaluation should be model or theory-oriented instead of research design oriented. Randomization can be used in any research if it is applicable, but we need rely less on randomization as a safeguard to internal validity if the threats in a research can be adequately identified and controlled.

Since the structural relationships among the relevant variables and intervening processes are carefully modeled, the theory-driven approach avoids the pitfalls of black-box evaluation and provides better understanding of the causal mechanisms underlying the relationships between treatment and effects. Unlike randomized experiments which aim only at internal validity, a theory-driven approach provides a general framework to deal with internal, external, construct and statistical conclusion validity.

References (30)

D.F. Alwin et al.
Causal Models, Unobserved Variables and Experimental Data
American Journal of Sociology
(1974)
B.S. Barnow et al.
Issues in the Analysis of Selection Bias
M.P. Bentler et al.
A Head Start Reevaluation: Positive Effects are Not Yet Demonstrable
Evaluation Quarterly
(1978)
R.A. Berk et al.
Selection Biases in Sociological Data
Social Science Research
(1982)
G.H. Bracht et al.
The External Validity of Experiments
American Educational Research Journal
(1968)
D.T. Campbell et al.
Experimental and Quasi-Experimental Designs For Research
(1963)
H.T. Chen
Sensible Evaluation: The Theory Driven Approach
(1987)
H.T. Chen et al.
Evaluating With Sense: The Theory-Driven Approach
Evaluation Review
(1983)
H.T. Chen et al.
Validity in Evaluation Research: A Critical Assessment of Current Issues
(1985)
P.W. Chommie et al.
Evaluation of Outcome and Process
Social Work
(1974)

J. Cohen

Statistical Power Analysis for the Behavioral Sciences

(1969)

J.S. Coleman et al.

High School Achievement: Public, Catholic and Private Schools Compared

(1982)

T.D. Cook et al.

Quasi-Exprerimentation: Design and Analysis Issues for Field Settings

(1979)

H.L. Costner

Theory Deduction, and Rules of Correspondence

American Journal of Sociology

(1969)

L.J. Cronbach

Designing Evaluations of Educational and Social Programs

(1982)

Cited by (134)

What does it mean for an evaluation to be ‘valid’? A critical synthesis of evaluation literature
2022, Evaluation and Program Planning
Citation Excerpt :
One concept related to quality that has faced such adaption/invention is that of ‘validity’ and this is the focus of our paper. Thirty years ago, Chen and Rossi (1987:95) concluded that “since the early development of program evaluation, issues of validity have been of major concern”. Almost 20 years later, Norris (2005:439) echoes this, stating “one of the defining problems of the field of evaluation remains the validation of evaluative judgments”.
Validity is both a critical foundation for, and widely contested concept in the field of program evaluation. Given the wide and divergent treatment of the term, this article sought to enhance the field’s understanding and application of validity by systematically identifying, collating and integrating existing conceptions of validity. A critical interpretive synthesis approach was taken to combine the best elements of systematic literature searching with integrative review. We found that conceptions of validity could be roughly collated into three categories pertaining to the evaluation research, evaluative judgement and reporting, and beyond the evaluation proper. The result was a wide-ranging map of validity concepts against the various stages of the evaluation process. This made clear several gaps in the validity thinking, resulting in the development of comprehensive but not yet complete framework that weaves together related aspects identified into a cohesive whole. It is our hope that this initial iteration of a cohesive validity framework will be the starting point for a cooperative effort to improve professional evaluation practice.
Realist protocol for implementing integrated care pathways in France (waiver 51 LFSS 2018)
2022, Annales Pharmaceutiques Francaises
Le paradoxe des études qui sont prétendument conduites, en vie réelle est qu’elles n’ont pas réussi à se libérer des techniques des essais randomisés dont elles prétendaient s’affranchir. Dans la perspective successioniste qui est celle des méthodes expérimentales ou quasi expérimentales classiques, la cause précède toujours l’effet de manière linéaire, et toute interférence qui est susceptible de menacer la stabilité de cette relation doit être neutralisée en mobilisant les techniques statistiques appropriée. Dans des systèmes complexes où tout bouge en même temps du fait d’interrelations multiples qui rendent impossible la construction d’un contrefactuel, ces éléments ne sont plus appelés à être considérés comme autant de facteurs de confusion qu’il convient de contrôler, mais comme des facteurs décisifs du bon déroulement de l’expérimentation. Le protocole présenté dans cet article propose une technique évaluative alternative mobilisant les enseignements du réalisme critique qui semblent les plus à même d’appréhender ce qui se passe « dans la boîte noire » enregistrant tous les événements qui sont susceptibles de survenir entre la mise en œuvre des expérimentations article 51 et le relevé de leurs résultats. Le rôle de l’évaluateur est de mettre les acteurs au cœur du changement, puisque celui-ci se réalise (ou non) en fonction de leurs réactions et des éléments de contexte. Cette théorie explicative crédible permet de comprendre : comment cela fonctionne ? Pour qui cela fonctionne ? pourquoi les perdants perdent, les gagnants gagnent ? et dans quelles circonstances ?
The paradox of real world researches supposedly conducted in real life is that they did not succeeded in freeing themselves from the techniques of the randomised trials which they pretended to escape. In the successionist perspective of classical experimental or quasi-experimental methods, the cause always precedes the effect in a linear manner, and any interference that is likely to threaten the stability of this relationship must be neutralised by mobilising the appropriate statistical techniques. In complex systems where everything moves at the same time due to multiple interrelationships that make it impossible to construct a counterfactual, these elements are no longer considered as confounding factors that need to be controlled, but as decisive factors in the smooth running of the experiment. The protocol presented in this article proposes an alternative evaluative technique mobilising the teachings of critical realism, which seems to us to be the most appropriate for understanding what happens “in the black box” recording the events that occur between the implementation of the Article 51 experiments and the observed results. The role of the evaluator is to put the actors back at the heart of the change, since it is achieved (or not) according to their reactions and the contextual elements. This credible explanatory theory allows us to understand: how does it work? For whom does it work? Why do losers lose, winners win? and under what circumstances?
Towards program theory validation: Crowdsourcing the qualitative analysis of participant experiences
2018, Evaluation and Program Planning
Citation Excerpt :
Theory-driven evaluation attempts to model the connections between a program’s activities, outcomes, external factors, inputs and outputs to guide the program’s evaluation (Chen & Rossi, 1983; Donaldson, 2007). The concept of applying a theory-driven perspective to evaluation first appeared in Edward Suchman’s 1967 book and was subsequently refined and popularized through a series of publications by Chen and Rossi in the 1980s (Chen & Rossi, 1983, 1987, 1989; Weiss, 1997b). Theory-driven evaluation has since become a widely touted approach with many top evaluation theorists contributing to its development (e.g., Bickman, 1989; Chen & Rossi, 1983, 1987, 1989; Chen, 1990, 1994, 2012; Lipsey, 1993; Weiss, 1997a, 1997b).
This exploratory study examines a novel tool for validating program theory through crowdsourced qualitative analysis. It combines a quantitative pattern matching framework traditionally used in theory-driven evaluation with crowdsourcing to analyze qualitative interview data. A sample of crowdsourced participants are asked to read an interview transcript and identify whether program theory components (Activities and Outcomes) are discussed and to highlight the most relevant passage about that component. The findings indicate that using crowdsourcing to analyze qualitative data can differentiate between program theory components that are supported by a participant’s experience and those that are not. This approach expands the range of tools available to validate program theory using qualitative data, thus strengthening the theory-driven approach.
Theory of change: a tool for participatory planning and evaluation in community health
2020, Gaceta Sanitaria
Existe un creciente interés por encontrar respuestas innovadoras para evaluar intervenciones complejas en salud pública. En esta nota metodológica, se presenta la teoría del cambio como una herramienta útil para facilitar tanto la planificación como la evaluación de intervenciones en promoción de la salud comunitaria. Una teoría del cambio es una representación visual de cómo funciona un programa o una intervención, en la que se identifican sus diferentes componentes y cómo cada elemento se vincula con otro: cuáles son sus objetivos, los resultados esperados y a través de qué acciones se pretenden alcanzar. Es una herramienta de investigación aplicada a la práctica que se desarrolla mediante métodos participativos con las diferentes personas implicadas en la intervención, para identificar y decidir conjuntamente qué es lo que hay que evaluar y cómo evaluarlo.
There is a growing interest in finding innovative approaches to the evaluation of complex interventions in public health. This methodological note presents the theory of change as a useful tool to facilitate both the planning and the evaluation of community health promotion interventions. A theory of change is a visual representation of how a programme or intervention works, in which its different components can be identified, together with how they relate to each other: what its objectives are, its expected results and through which actions they are expected to be achieved. It can be an applied research tool, developed using participatory approaches together with all relevant stakeholders, who can identify and jointly decide what to evaluate and how to.
Cryptocurrency in the Aftermath: Unveiling the Impact of the SVB Collapse
2023, TechRxiv
Cryptocurrency in the Aftermath: Unveiling the Impact of the SVB Collapse
2023, arXiv

View all citing articles on Scopus

^☆: This article is a revised version of a paper presented at the 1985 American Association Annual Meeting held in Washington, DC.

View full text

The theory-driven approach to validity☆

Abstract

Causal Models, Unobserved Variables and Experimental Data

American Journal of Sociology

Issues in the Analysis of Selection Bias

A Head Start Reevaluation: Positive Effects are Not Yet Demonstrable

Evaluation Quarterly

Selection Biases in Sociological Data

Social Science Research

The External Validity of Experiments

American Educational Research Journal

Experimental and Quasi-Experimental Designs For Research

Sensible Evaluation: The Theory Driven Approach

Evaluating With Sense: The Theory-Driven Approach

Evaluation Review

Validity in Evaluation Research: A Critical Assessment of Current Issues

Evaluation of Outcome and Process

Social Work

Statistical Power Analysis for the Behavioral Sciences

High School Achievement: Public, Catholic and Private Schools Compared

Quasi-Exprerimentation: Design and Analysis Issues for Field Settings

Theory Deduction, and Rules of Correspondence

American Journal of Sociology

Designing Evaluations of Educational and Social Programs