High-performing nonprofits and government agencies invest significant resources in conducting surveys, among other forms of research, to improve the quality and efficacy of their programs and services. But developing the survey questions can be challenging for professionals entrenched in these organizations, given that they often bring assumptions and biases, and concerns about how the responses might reflect on them personally, or on their teams. Measurement Resources Company has designed effective survey instruments for social sector clients over the past 12 years. Here are a few tips that we share with our clients at the beginning of this process.
1. Include a “what’s-in-it-for-them” proposition.
Because of the amount of effort required for organizations to develop, distribute, and analyze a survey, it’s important to ensure that they see the return on this investment in terms of adequate response rates that provide meaningful data on which to act. On the other side of this equation, given that organizations are asking community members to invest their time and thoughts into providing this valuable feedback, it’s just as critical that you include a “what’s-in-it-for-them” proposition. Benefits can include even the most basic and obvious points: a promise to use the data to improve the delivery of programs and services that will directly impact the respondents. And we often encourage our clients to sweeten the deal by providing incentives such as a drawing for gift cards or other prizes.
2. Let respondents know how long it will take to complete the survey.
To reduce the rate of survey abandonment, let respondents know up front how much time they will need to complete the survey. Even some of the more comprehensive surveys should take no more than 10 minutes to complete.
3. Check for implicit bias and leading questions.
No matter how amazing your services might actually be, your questions should not in any way be something along the lines of: “On a scale from 1 to 10, with 10 being ‘out-of-this-world amazing,’ exactly how amazing are our services?” This example is exaggerated of course, but even experienced professionals can inadvertently assert certain qualities in the framing of their questions. For this reason, having a third party to review your survey questions for implicit bias and leading questions can help ensure that your questions are stated neutrally, and that the responses you receive accurately depict the opinions and experiences of the respondents.
4. Speak to the masses.
Surveys are not the time for extravagant writing. It is a best practice to use clear, concise language that is free of idioms, slang, colloquialisms, technical or industry-specific jargon. Especially for broader population surveys, making sure the language is easy for those speaking English as a second language (ESL) is also critical for collecting data from diverse populations and ensuring that the solutions that come out of the study are equitable. Likewise, the same linguistic rules apply if you intend to translate your survey into multiple languages.
5. Avoid the pitfalls of pre- and post-survey questions.
There are some instances where it is appropriate to ask questions like this in a pre-survey (a survey taken before individuals participate in a program or service): “How would you rate your knowledge of topic A?” Then followed by a post-survey that includes a question such as: “After taking this course, how would you rate your knowledge of topic A?” But one caveat to keep in mind is that these sorts of questions could potentially skew your results. For example, if an individual thought they knew a great deal about topic A prior to the course, they would rate their knowledge as high. But after taking the course, they come to the realization that their prior knowledge is just the tip of the iceberg, then their score would go down significantly after the course, resulting in what might appear to be a decrease in knowledge, when in fact they learned quite a bit as a result of taking the course.
6. Balance quantitative with qualitative questions.
Quantitative questions involve the collection of numerical data to determine statistical results. However, including qualitative questions is equally valuable because the responses provide the context needed to better understand the numerical data. Combined, the information collected will form the issues your organization needs to prioritize and address, and can reveal potential solutions or improvements that can be implemented to increase overall satisfaction and produce better outcomes.
7. Include a range or ratings that includes neutral territory.
For close-ended questions, using 5- to 7-point Likert scales, with the middle rating as being neutral, is a best practice. These answer choices often look like this:
- Very satisfied
- Somewhat satisfied
- Neither satisfied nor dissatisfied
- Somewhat dissatisfied
- Very dissatisfied
8. Measure for cultural relevance.
Hands down, our nonprofit clients who focus on delivering culturally relevant programs and services significantly outperform their competitors in terms of achieving more and better outcomes—at a fraction of the cost of what it would take for the public sector to address. Measuring this attribute is an absolute must for all social sector surveys.
9. Limit the number of questions.
Survey length is the top reason respondents abandon surveys. In addition to limiting the overall number of questions in the survey, adding features such as skip logic, or not requiring comments can help speed up this process.
10. Measure change over time.
The majority of survey questions should be developed with the intent of measuring change over time. In other words, the survey contains a core set of questions meant to be conducted again over regular intervals. Whether conducted annually, biennially, or some other schedule, the goal is to assess how well the organization implements continuous improvements that come out of the results of the previous survey’s findings, while adapting to a changing environment. The remainder of the survey should be used to answer those burning questions that will help your organization understand new trends and better anticipate the emerging needs of constituents.
How Measurement Resources Helps Social Sector Organizations Maximize their Return on Research Investments
Having worked with more than 200 social sector organizations, Measurement Resources helps you gather the right data to change actions, outcomes, and lives. From bringing a fresh perspective to the survey development process, plus decades of combined experience and expertise, we simplify the process and extract the valuable insights you need to master high performance.
Contact us today to learn how we can help you use data to drive impact.
