Not all survey results are created equal. With the proper survey sample size, surveys can yield powerful data to gauge program outcomes. They are also great resources for understanding the needs and desires of your ideal clients. Knowing this information can help you improve your operations and services and communicate your impact and value to funders and stakeholders Without a good survey response rate, your survey results may leave you with more questions than answers.
Surveys are most useful when they reflect the collective thoughts and opinions of your program participants or ideal clients. This means you need enough people to take your survey to feel confident that the responses from the survey would be replicated if you surveyed everyone in your target population. The important, but often overlooked, question in survey implementation is how many responses are needed to be able to make decisions with confidence regarding the survey data?
The ideal number of survey responses will depend on the number in your total population. Therefore, the first step is to estimate the total number of people in the population for which you want to survey. If you are only trying to make inferences about your program participants and you served 100 people, then your population is 100. On the other hand, if you goal is to make inferences about your ideal clients—single mothers in Ohio—your total population would be approximately 339,000.
The ideal survey response rate is not based on a percentage of the total population. Instead it is a bit more complicated and is based on the following factors: confidence level, confidence interval, and variance. In this article, I provide a brief description of each of these to help you understand how a good sample size is determined.
¨ Confidence level is the degree of certainty that one can have when they draw inferences about a population based on data from the sample. It is the level of probability that the researchers have so they can accurately generalize a characteristic they find in a sample to the population. A confidence level of 90% means researchers are 90% confident that the sample accurately represents the populations. Without going into a detailed explanation of statistics, a confidence level is based on standard deviations of the data.
¨ Confidence interval is a range or margin of error that one permits when making inferences from a sample of a population. Unless a survey is completed by every person in the desired population, the results found from a survey will not provide complete accurate information regarding a population’s true values. The true population values falls somewhere within the range of the confidence interval. The confidence interval is usually stated as a positive-to negative range, such as +/- 3% error. If 57% of a sample of program participants reported an increase in exercise after a program in a survey with a +/- 3% confidence interval, for example, the true population value may be as high as 60% (+3%) or as low as 54% (-3%).
¨ Variance is the distribution of a variable in a population expressed as a percentage in decimal form. For sample size calculation purposes, variance is always 0.5. This is used because it is the largest measure of variance available and allows a researcher to look at multiple variables within one survey without needing different sample sizes. This assumes that 50 percent of the population has a characteristic and 50 percent of the population does not.
It is great if you have a vague understanding of these concepts to discuss why a particular sample size was achieved and what it means regarding the generalizability of your results. As you can see in the Table 1, you need a larger percentage of the total population to complete a survey the smaller the population size. Similar to how you use a similar size spoon to taste test soup when cooking whether you’re cooking for your family or a party of 50, a similar size sample is required for large population sizes. There is an industry standard that regardless of population size over 100,000 a good sample size is around 400. Table 1 highlights ideal sample sizes for a variety of population sizes to obtain survey results with a +/-5% margin of error at a 95% confidence level. To calculate specific sample sizes with your desired parameters you can consult the several sample size calculators on the internet.
Table 1. Ideal Sample Sizes Based on Population Sizes
| Population Size | Sample Size | Population Size | Sample Size |
| 50 | 44 | 2,500 | 333 |
| 100 | 108 | 5,000 | 357 |
| 150 | 132 | 7,000 | 364 |
| 200 | 152 | 10,000 | 370 |
| 500 | 217 | 50,000 | 381 |
| 750 | 254 | 100,000 | 383 |
| 1,000 | 278 | 275,000 | 384 |
Before you launch your next survey, determine your ideal response rate related to your desired confidence level and confidence interval. This will allow you to create a plan and appropriate budget to obtain your ideal sample size. If you have a large population, a good rule of thumb is to always strive for at least 400 surveys. You will need even more respondents if you want to compare different demographic groups. When writing up your survey results, instead of just reporting the percentage of the population that completed your survey, use a survey sample calculator to report the confidence level and intervals. Let your stakeholders know how likely the same results would be found if the survey was conducted with a different group of people in the same population. A good sample size will motivate stakeholders to give more credibility to the data you are presenting.
If you are looking to gauge your outcomes or gain insights from ideal clients through a survey, Measurement Resources is here to help! We will help you design great measures, plan for the ideal response rate, and analyze your survey results. Our favorite part is to celebrate our clients’ success on their increased impact on the world! We’d love to help you make data-driven decisions with confidence. Contact us today for your free 20 minute strategy session.
Want more information on how to increase funding, morale, positive press, and organizational impact? Join the Achieving Excellence Community and receive our free eBook, Ten Tips to Open the Door to More Grants (and Other Funding): Overcoming Common Mistakes in Outcomes Measurement.
Sheri Chaney Jones
Measurement Resources Company
May 2014
