Insights.

One of the easiest things to mess up when preparing a survey is to ignore the science of sampling the population you want to evaluate. Survey samples often begin with a certain bias, and most of the time that is a good thing. For example, a magazine for knitters sends a survey to readers to better understand their habits. That survey is designed to sample a population of people with the expressed interest of knitting, and the questions that follow proceed from that original assumption. When explaining collected data, the survey’s sample needs to be stated up front. For example:

Good: "60% of our readers buy yarn more than once a month."

Misleading: "60% of people buy yarn more than once a month."

Now, consider a slightly more subtle case. A recent survey invitation posted in a paper industry newsletter will be working with a sample (and responses) that are skewed to some degree by the delivery of the invite. Mostly, the survey is perfect because it's at the end of the very newsletter for which they are collecting the data and the questions asked pertain to the industry. However, if a third party paper mill wanted to use this industry newsletter to invite people to take a general survey about personal paper usage, they would have started with a very narrow sampling of the overall population of paper users. The responses could be so skewed as to be unusable. The sample selection matters.

This Survey Monkey invitation is what triggered this entire post. At the end of the survey, Survey Monkey asks the reader to take more surveys for a chance to win $100 and to help a charity of their choice. The sample for these surveys becomes very questionable because people may stretch the truth to qualify for more surveys.

Professional survey takers could easily be ignorant of the industry they claim to know and/or not really reading the questions, just "burning" through the survey to get to their entry for $100.

And speaking of incentives ...

It's hard to get people to take your survey. Adding an incentive is a good way to increase response. Also, many times people will spend more time with a survey if they feel like they are being compensated for their time. It's like a give and take. The incentive is probably better if it's on the small side, just enough to say you value the survey taker’s time but not so much to attract people outside your target population. The best incentive is something in line with your product or survey to further qualify your responders. For example, if Pampers offered a chance to win a trip to Maui, a bachelor might take their survey even though he doesn't have kids. However, if Starbucks offered a free coffee as an incentive to take a survey about coffee, only coffee drinkers would be incentivized.

All of this should seem like common sense but the low-cost online survey options available today provide anyone the ability to easily create a survey that, in the end, could deliver conclusions that are useless or just plain wrong. So, when designing a survey, carefully study the whole process from start to finish. What do you want to know? Who do you want to target? Who is likely to respond? How do you pose the questions in the most unbiased way? And when you have finished the survey, make public your sampling methods to make the most out of the data collected.

Surveys are just one of the many tools that the data experts at iDfour use to evaluate a business’ effectiveness and project their opportunities for growth. Contact us today to see how we can help you make your next project a success!