Sidestepping four common mistakes when implementing measurement efforts can help social sector organizations develop successful and action-oriented performance and outcome measures, save time and money, and increase impact.
Nearly a quarter of a century has passed since the Government Performance and Results Act (GPRA) was passed into law. According to President Clinton, the intention of the GPRA was to “chart a course for every endeavor that we take the people’s money for, see how well we are progressing, tell the public how we are doing, stop the things that don’t work, and never stop improving the things that we think are worth investing in”.
Yet, here we are in 2017 and the majority of government and nonprofits organizations are still struggling with measuring their impact and using performance and outcome measures to achieve significant social change. What we’ve learned from the research in Impact & Excellence is that the leaders who have figured it out are the ones increasing their revenues, generating efficiencies, and changing their communities in significant ways.
Based on interactions with hundreds of nonprofit and government leaders about the struggle to implement measurement cultures, observing both successful initiatives and ones that never got off the ground, I’ve identified four of the most common mistakes and the strategies to overcome them.
- Overlooking uniqueness in short-term outcomes
- Underestimating fear
- Devaluing follow-up and action
- Unrealistic expectations of change
Over a four-part blog series, I will be addressing each of these four themes. Together, they suggest ways for social sector leaders to get more from their performance and outcome measurement efforts as these organizations face challenges ranging from increased client demand and uncertainty in the availability of essential funding streams.
Mistake 1. Overlooking uniqueness in short-term outcomes
A well-defined theory of change for an organization is a critical component of successful social sector leadership. Therefore, a powerful measurement framework in one organization does not necessarily translate to meaningful insights in another. Sometimes organizations or funders rely on one specific outcome measure to evaluate success. Popular measures frequently used as standalone measures of success include graduation rates, recidivism rates, increased test scores, and cost. Each of these is an important and useful measure, and organizations which have missions aligned with these outcomes should certainly be tracking them. The problem does not lie with the effectiveness of the measures themselves but with an imbalanced reliance on them. When one of these measures becomes the only data point shared, the picture of overall effectiveness is incomplete.
Using a single indicator of success makes it more difficult to take appropriate, data-driven actions. If the outcome is less than desirable, that one measure is unlikely to provide decision makers with insights about why things are not going as planned. Is it because participants are not satisfied with services and are therefore less engaged? Or is it because there is inadequate funding to provide the proper services to achieve these outcomes?
Alternatively, if the outcome is desirable, a single measure is unlikely to indicate if different groups are successful at different rates. One measure is also unlikely to answer the question of whether the program could be delivered in a more cost-effective manner. Using just one measure limits the organization from the fullest understanding of what, specifically, is contributing to a success or failure.
A Common Story
For example, one housing organization, on the insistence of a funder, measured success against the number of their clients who were stably housed after six months in placement. Although this was a long-term outcome of this programs efforts, the results were often demoralizing because this program was not responsible for the clients’ direct placement into the housing programs. The primary mission of this organization was to stabilize a family in crisis and refer them to an organization who could help them find permanent housing. Once their client was linked to the housing program, that client’s long-term success was no longer within their direct control.
When they received the less than ideal updated housing retention numbers, they felt powerless. These numbers did not reflect the changes that were occurring in the clients lives as a result of their programs. The numbers also didn’t provide any insight on what this particular organization should be doing differently to increase the likelihood of success for their clients once they were referred.
It wasn’t until the leaders decided to take measurement into their own hands that data became a powerful tool for their overall success. The organization defined their theory of change and clearly identified the short-term outcomes they had most control of, such as stabilizing a family in crisis and identifying and developing a plan to remove barriers to housing. This clarity allowed them to attach measurable goals to these efforts which helped them understand what extent people were better off as a result of their services. They tracked this data on all their clients and used it as part of their continuous improvement efforts. This measurement plan gave them the insight to change things within their control that would likely impact a client’s ability to remain in stable housing.
10 Short-Term Outcomes Assignment
Measuring and managing to short-term outcomes is often the best first step an organization can take in implementing a measurement system. Start by clarifying the organization’s mission and outline what ideally should happen immediately after a person engages in their program that will likely predict their success in the future. This will ensure a program is achieving the necessary changes in knowledge, skills, behavior or support needed for a client to succeed at the desired medium-term or long-term impacts.
Short-term outcomes often look like one of the ten types of outcomes listed below. Use these as a guide to select the short-term outcomes which are unique to your programs and services. Once you define what these look like for you, attach measures to them and review them regularly. You will be amazed at how focusing and managing to these short-term outcomes will naturally start to produce greater medium and longer-term impacts. Not all ten of these measures will be right for every organization, but you should find at least three that are appropriate.
- Increased knowledge and learning: What percent of participants increased their skills and knowledge as a result of your program or service? This is measured by a simple pre-post test.
- Changed attitudes: What percent of participants improved their attitudes as a result of your program or services? This can be reported by ratings from the participant, parent, teacher, co-worker, employer, counselor, etc.
- Increased readiness: What percent of participants are meeting the minimum qualifications for the next level? This can be measured by giving a skills and knowledge test that includes these next level concepts or skills. Also, you can assess this using an attitude survey.
- Reduction of undesirable behavior: What is the incident rate of undesirable behavior after the program? This is measured by relapse, recidivism, or self-reported behavior frequency. The smaller the better for this measure.
- Increased desirable behavior: What percentage of participants achieved the desired goal? This is measured by predetermining the goal and recording success after a set period of time.
- Maintenance of new behavior: What percentage of participants moved to the next level or do not reenter the program or system? This is simply measured by tracking those individuals that move or reenter.
- Increased social status: What percentage of participants improved or developed new positive relationships? This is measured by observation or self-report items.
- Increased economic conditions: What percentage of participants retained employment, moved to permanent housing, increased income, etc.? This is measured by a self-report or observation measures.
- Improved health conditions: What percentage of participants reduced incident of health problems? This is measured by a self-report or review of medical records, etc.
- Increased economic development: How many jobs did your organization create? This is measured by calculating the jobs created and the total dollars injected into the economy.
Start the process today!
Are you looking for a painless and efficient process to develop your logic model and to establish your core measures? Our Quick Start Performance Measurement System is the best solution for you! In this cost-effective program we do all the hard work while you have all the fun on the road to greater accountability and impact. Contact us today to schedule a call to explore if this is the best program for you and your organization.
Don’t forget to visit next week! Learn how fear is the biggest barrier to your measurement efforts and what you can do to overcome it.
