University of FloridaSolutions for Your Life

Download PDF
Publication #PEOD3

Elaborating Program Impacts Through Data Analysis1

Glenn D. Israel2

The analysis of program impacts involves several phases (Israel, 1992). Initially, a preliminary data analysis should be conducted to identify errors in the data from coding. The preliminary analysis also should examine the distribution of variables in the data set in order to assess whether or not the data is suitable for the planned statistical methods. In the next phase of the analysis, change in the impact indicators is assessed. This phase of the analysis addresses the questions, "Have the target variables changed? Have these variables changed in the expected direction?" Although a negative response to these questions might be interpreted as evidence of the program's failure, other contextual factors may have suppressed or hidden the impact of the program.

The third phase consists of the analyzing changes. Bivariate statistics, such as correlation coefficients and the Chi-square statistic, can be used to answer the question, "Are program variables associated with changes in the target variables?" In other words, is participation in a workshop or demonstration related to knowledge gained or use of recommended practices? Then multivariate statistical methods can be used to assess how those associations are affected by contextual factors. For example, are changes in people's behavior influenced by participation in an Extension program after accounting for the effects of another organization's program. The utility of multivariate methods lies in further clarifying the relationship between program variables and target variables. This paper illustrates how relationships can change when other variables are included in the analysis. A discussion of how evaluation findings can be affected is also included.

ELABORATING PROGRAM IMPACTS

As Patton (1982) notes, one object of the analysis is to clarify the relationship between program activities (variables) and target variables. Rosenberg (1968) and others refer to the process of clarifying relationships as elaboration. When a relationship between a program variable and a target variable is observed, the evaluator often is interested in determining how that relationship is changed when a contextual variable is introduced. Examination of contextual variables may result in rejection of the initial interpretation of the impact of the program; it may reinforce the initial interpretation, or lead to revision (Rosenberg, 1968). Quantitative multivariate analysis can increase our confidence in the finding of program impact and specify under what conditions that the program works best. These methods also can help to identify ways to improve the program.

The basic method to elaborate on relationship between the program variable and target variable is to stratify by a relevant contextual variable to examine the contingent associations (Rosenberg, 1968). This means that the data set is divided into groups (e.g. males and females; low education and high; participants and non-participants in a competing program). Then the relationship between the program variable and the target variable is examined for each group.

In addition, specification of the time order of the program and contextual variables is used to further clarify the effect of the program on the target variable. Does the contextual variable occur prior to the program (i.e., is antecedent) or does the contextual variable "intervene" between the program and target variables? Four types of relationships are shown below to illustrate how initial interpretations can be affected by the process of elaboration.

Spurious Relationship

In this situation, the impact of the program disappears when a contextual variable is added to the analysis. As shown in Table 1a, the initial relationship shows program participants to be more likely to adopt practice Y than nonparticipants (70 percent vs. 50 percent, respectively). When an antecedent contextual variable, education, is used to stratify the association, the relationship between participation and adoption disappears for each level of education (Table 1b). The initial relationship between program participation and adoption of practice Y is due solely to the "marginal" relationship of education with both of the variables. That is, the initial conclusion that participation causes adoption of practice Y must be revised to account for the greater rate of participation of highly educated persons in the program as compared with participation of persons with low education (75 and 25 percent, respectively).

Spurious relationships are perhaps the greatest threat to the validity of evaluation studies. If the effects of relevant contextual variables are not accounted for, then the evaluation risks making conclusions about program impacts which are incorrect. On the other hand, if the relationship between the program variable and target variable remains unchanged when a contextual variable is included in the analysis, our confidence that the program's impact is real is increased.

Antecedent Conditional Relationship

In this situation, the effect of the program is revealed to be greater for one group than another when a contextual variable is included. As shown in Table 2a, the initial relationship shows program participants to be more likely to adopt practice Y than nonparticipants (60 percent vs. 20 percent, respectively). When education is used to stratify the association, the effect of the program disappears for clientele with low education and increases for clientele with high education (Table 2b). There is no difference in the percent who adopt practice Y for program participants and nonparticipants in the low education group. At the same time, the difference increases from 40 percent between participants and nonparticipants in the overall table (Table 2a) to 55.2 percent for the high education group in the contingent table (Table 2b). Thus, program impact is limited to conditions of high education in this example. In a conditional relationship, at least one of the partial relationships (subtables) shows that the impact of the program on the target variable is maintained or strengthened.

Distorted Relationship

In a distorter relationship, the effect of the program reverses (from negative to positive or from positive to negative) when a contextual variable is included in the analysis. The initial relationship in Table 3a shows program participants to be less likely to adopt practice Y than nonparticipants (60 percent vs. 70 percent, respectively. When education is added to the analysis, the relationship reverses for clientele with both high education and low education (Table 3b). For each education group in the expanded table, 5 percent more of program participants adopt practice Y than do nonparticipants. The data in Table 3b shows that there was a difference in the number participating in the program for persons with high and low education. This difference lead to the distortion of program impact seen in Table 3a.

Suppressor Relationship

When the impact of the program on the target variable initially appears to be absent, the inclusion of a contextual variable can reveal a positive impact. As shown in Table 4a, there is no difference between program participants and nonparticipants regarding the adoption of practice Y in the initial analysis. When the contextual variable, education, is included, a positive impact of program participation on the adoption of practice Y is found for both levels of education in this example (Table 4b). For each level of education, program participants adopt practice Y by 10 percentage points more than nonparticipants. Again, a difference between the number or persons with high education who participate and those with low education hides the true impact of the program.

SUMMARY

The relationships which are illustrated in the tables in this paper show how the use of contextual variables in the data analysis can guard against erroneous interpretations of program impact. A key step in elaborating relationships is determining which contextual variables need to be included in the analysis. Selection the right ones will increase the credibility of the conclusions about the presence or absence of program impacts.

The primary contribution to the elaboration of evaluation data using quantitative, multivariate techniques is to avoid making incorrect interpretations and to increase confidence in the validity of the findings. Israel (1987) provides an introduction to the multivariate techniques which can be used in evaluations.

REFERENCES

Israel, Glenn D., 1987. Models and Techniques For Data Analysis in Evaluation Studies. Program Evaluation and Organizational Development, IFAS, University of Florida.

Israel, Glenn D., 1992. Phases of Data Analysis. Program Evaluation and Organizational Development, IFAS, University of Florida. PEOD-1. September.

Patton, Michael Quinn, 1982. Practical Evaluation. Beverly Hills, CA: Sage Publications

Rosenberg, Morris J., 1968. The Logic of Survey Analysis. New York: Basic Books.

Tables

Table 1. 

Table 1a. Bivariate Relationship Showing Program Impact.

 

Adopt Practice Y

Program Participant

No

Yes

Total

No

50

50

100

Yes

30

70

100

Table 2. 

Table 1b. Spurious relationship between program participation and adoption, controlling for education.

 

Education

 
 

------- High ------

------ Low -------

 
 

Adopt Practice Y

Adopt Practice Y

 

Program Participant

No (%)

Yes (%)

No (%)

Yes (%)

Total

No

5 (20)

20 (80)

45(60)

30 (40)

100

Yes

15 (20)

60 (80)

15(60)

10 (40)

100

Table 3. 

Table 2a. Bivariate Relationship Indicates A Program Impact

 

Adopt Practice Y

Program Participant

No

Yes

Total

No

80

20

100

Yes

40

60

100

Table 4. 

Table 2b. Conditional Relationship.

 

Education

 
 

------- High ------

------ Low -------

 

Program

Adopt Practice Y

Adopt Practice Y

 

Participant

No (%)

Yes (%)

No (%)

Yes (%)

Total

No

70 (87.5)

10 (12.5)

10 (50)

10 (50)

100

Yes

20 (33.3)

40 (67.7)

20 (50)

20 (50)

100

Table 5. 

Table 3a. Bivariate Relationship Indicates No Program Impact.

 

Adopt Practice Y

Program Participant

No

Yes

Total

No

30

70

100

Yes

40

60

100

Table 6. 

Table 3b. Distorter Relationship.

 

Education

 
 

------- High ------

------ Low -------

 

Program

Adopt Practice Y

Adopt Practice Y

 

Participant

No (%)

Yes (%)

No (%)

Yes (%)

Total

No

20 (25)

60 (75)

10 (50)

10 (50)

100

Yes

4 (20)

16 (80)

36 (45)

44 (55)

100

Table 7. 

Table 4a. Bivariate Relationship Indicates No Program Impact.

 

Adopt Practice Y

Program Participant

No

Yes

Total

No

40

60

100

Yes

40

60

100

Table 8. 

Table 4b. Suppressor Relationship.

 

Education

 
 

------- High ------

------ Low -------

 

Program

Adopt Practice Y

Adopt Practice Y

 

Participant

No (%)

Yes (%)

No (%)

Yes (%)

Total

No

18 (30)

42 (70)

22 (55)

18 (45)

100

Yes

4 (20)

16 (80)

36 (45)

44 (55)

100

Footnotes

1.

This document is PEOD3, one of a series of the Agricultural Education and Communication Department, Florida Cooperative Extension Service, Institute of Food and Agricultural Sciences, University of Florida. Original publication date October 1998. Revised March 2006. Reviewed January 2012. Visit the EDIS website at http://edis.ifas.ufl.edu.

2.

Glenn D. Israel, associate professor, Department of Agricultural Education and Communication, and Extension Specialist, Program Evaluation and Organization Development, Institute of Food and Agricultural Sciences (IFAS), University of Florida, Gainesville FL 32611.


The Institute of Food and Agricultural Sciences (IFAS) is an Equal Opportunity Institution authorized to provide research, educational information and other services only to individuals and institutions that function with non-discrimination with respect to race, creed, color, religion, age, disability, sex, sexual orientation, marital status, national origin, political opinions or affiliations. For more information on obtaining other UF/IFAS Extension publications, contact your county's UF/IFAS Extension office.

U.S. Department of Agriculture, UF/IFAS Extension Service, University of Florida, IFAS, Florida A & M University Cooperative Extension Program, and Boards of County Commissioners Cooperating. Nick T. Place, dean for UF/IFAS Extension.