A good evaluation design does not require a control group, but the results will be much more interesting and powerful if you have one. The purpose of program evaluation is to understand the extent a program is achieving desired program outcomes and impacts. For example, if a program is designed to increase participant self-esteem a program evaluation will let you know the extent self esteem has been increased for participants. When designing a program evaluation, a decision whether or not to include a control group needs to be made.
A control group or comparison group is a group of people who are similar to participants that have not gone through the program or services. The benefit of using a control group is that it allows you to report with confidence the unique impact of your program. It allows you to say how your program is making a difference in the lives of participants that is fundamentally different from what is happening to those who are not going through your program. It provides credible evidence that participants would not have naturally experienced the desired outcomes without the program or services you provide.
Good evaluation can still occur without a control group. It will at least let you know the outcomes your participants experienced. However, an evaluation with a control group is so much better. To highlight this, let’s examine the outcome data from the evaluation we conducted of Paper Circle’s Circle Round the Square. This is an intensive five-week summer-arts enrichment program that provides students with hands-on experiences with a wide range of arts and wellness activities. These activities are lacking in the K-8 school curriculum. One of the program’s outcomes was to increase self-esteem. Participant data reveals that students experienced on average a 4% increase in self-esteem based on pre and post self-esteem scores (see data below). These results reveal that self-esteem increased as a result of this program; yet, it is difficult to determine if participant self-esteem was likely to increase regardless of the participation in the program.
IMAGE PENDING
^ Cohen’s d is a measure of effect size. It provides an indication of how big or small a significant difference is. Typically less than .2 is consider a small effect, .5 is a medium effect, and .8 is a large effect
The story becomes much more interesting when the control group is added. Students of the same age and who attended the same school as the participants served as a control group in this evaluation design. The control students completed the same self-esteem questions before and after this summer program. The results revealed that control students experienced a 3% decline in self-esteem scores. Interestingly, we learned that students who did not participate had slightly higher levels of self-esteem at the start of the program and after the program both groups had similar levels.
As shown in Figure 1., adding the control group confirms that not only does the program increase self-esteem of participants, but that other similar students saw a decrease in their levels of self-esteem over the summer when they did not participate in this program. In addition, this program learned that their participants saw significant increases in creativity and wellness when compared to the control group. This story is much more compelling and helps make the case for the unique impact of the program.
IMAGE PENDING
As a result, this information can be used to show funders how their dollars benefited students. It addition, it can be used to ask for increase funding to expand the number of students this program is able to serve.
For your next evaluation, I challenge you to consider adding a control group. Think about what group of individual is similar to your participants. In many cases, obtaining a control group requires only a little more time, effort, and resources. The information gained from the comparison is well worth it in terms of program improvement and ability to clearly communicate your programs unique impact and value.
Looking for assistance with your evaluation design or need an external evaluator? Measurement Resources is here to help! Contact us today for your free 20-minute strategy session.
Want more information on how to increase funding, morale, positive press, and organizational impact? Join the Achieving Excellence Community and receive our free eBook, Ten Tips to Open the Door to More Grants (and Other Funding): Overcoming Common Mistakes of Outcomes Measurement.
Sheri Chaney Jones
November 2013
