The Trouble with Market Research

Wayne Rash
I recently received a research report from an analyst firm that contained some findings that I found surprising, and that didn't match what I'd seen in the real world. The report, among other things, divided all mobile phone users into two demographic groups: all people under 55 and all people over 55. Then it made some sweeping generalizations about how these people selected their mobile devices.

What I found when I dug deeper into this report gives a good illustration as to why you should be wary before you spend thousands of dollars on these reports. I won't burden you with too many details except to say that the analyst firm made a big deal of the 900 respondents who were surveyed in the report. What the firm didn't say was that the respondents weren't randomly selected-they were invited to opt in to the survey, which is another way of saying they were self-selected.

Now, you see a lot of surveys on the Internet, and the vast majority of what I've seen are just that kind. You are asked if you'd like to participate in a survey when you're visiting a website of some sort. While these surveys can produce useful information, especially if they're asking visitors to the site whether they found what they wanted, or whether using the site was easy enough, that's not the same thing as being a statistically valid survey. To be valid, participants must be randomly chosen. While there can be some limits on the randomness-for example, you can limit your questions only to people of a specific gender or people who smoke or drive cars-you can't base a scientific survey on people who volunteer to take part.

Unfortunately, actually finding out this information can be difficult. You might see some kind of attention-grabbing headline that highlights a fact in the alleged survey, and then offers to sell you the complete report for several thousand dollars. It's only when you buy the report that you find out that you've been misled. The study doesn't necessarily present accurate survey data, but rather something else that fits the needs of the company presenting the study.

Fortunately, these companies, if they're even slightly reliable, will let you reach out to the researchers and ask them. Usually, they'll tell you. In the study I reference above, the principal researcher admitted that the age of 55 just sort of came out in a meeting. He also admitted that the survey population was self-selected. In other words, the survey wasn't performed scientifically, and the study population's demographics were chosen on a whim.

The problem with this sort of study is that analyst firms target companies that need good marketing data, and need it quickly so they can react to a rapidly changing marketplace. If a study is offered that seems to reflect on something you're trying to find out, there's a strong temptation to spend the money and use the information in the study to guide your business decisions. But do you want to base your decisions on flawed data? Probably not.

In reality, it never pays to move too quickly on these offers. When any research firm presents an unsolicited offer, it pays to spend a couple of days verifying the information, and the methodology, before you buy. While it's possible that weak methodology may still produce useful results, at least you should know before you spend the money. Then, when you base your decisions on the data, you'll know where the holes are.

Add Comment      Leave a comment on this blog post
Feb 17, 2011 5:02 PM Alex Khaikin Alex Khaikin  says:
I would like to comment and clarify a couple of points. 1. Website and Web Page based surveys are, primarily, used for website evaluation, visitors' profile or e-shopping analysis. Website visitors are invited to participate in a survey using �banner� type invitation, "pop-up" window or weblink. The surveys are an effective and inexpensive method to obtain the attitudes and opinions of current customers or website visitors. While the respondents can be randomly selected, they are invited to opt in and, as a result, are self-selected. The same, by the way, is true for any employee or customer surveys that are done based on already established contact lists. 2. I absolutely agree, based on significant experience in the field, some and especially larger companies commission research to �support� their marketing strategies. They want to see what they �want to see�. This creates significant pressure on an independent research company because it wants to keep the business. 3. A more accurate and cost effective way to conduct an unbiased awareness, perception and usage studies is via email online surveys. Respondents for the studies are recruited for participation through email invitation from web panels. Web panels are very large, demographically and geographically representative, Internet-based panels for customer, business to business and, sometimes, employee surveys. They included millions of consumers, business owners and professionals. The panels are consistently supported and refreshed to reflect demographic changes and to ensure statistically representative sample. As most surveys and research project require relatively small sample size (up to 1,000 completed response), the main reason to support and consistently refresh such large panels is to minimize the impact of, so called, �professional� or �self-selected� respondents. Each panel member may expect to participate in the surveys not more than 1-3 times a year. For this reason, participation reward system is also based on random drawings of various prizes depending on length, complexity and topic of each survey. Panels recruitment sources: - Web advertising - Permission-based databases - Public relations (local papers, web portals, etc.) - Partner-recruited specialty panels - Alliances with heavily trafficked portals Major benefits of web panels: - Worldwide coverage - Cost efficient (significantly cheaper vs. equivalent phone survey) - Short reply time (2-5 days) and high response rate (over 50%) - High accuracy - statistically representative of the general population - No need to collect demographic information during the survey (the data is collected during the panel design process) - Supports consistent follow-up analysis of virtually the same sample (change in awareness level before and after advertising, etc.) - Allows incorporation of visual effects and objects (pictures, movies, etc.) Reply
Apr 28, 2011 5:04 PM Scott Smith Scott Smith  says: Reply
Oct 7, 2011 4:10 AM Eye Care Eye Care  says:
Wow that was unusual. I just wrote an really long comment but after I clicked submit my comment didn�t show up. Grrrr� well I�m not writing all that over again. Anyways, just wanted to say great blog! Eye Care Reply
Dec 1, 2011 5:12 AM Automobiles Zone Automobiles Zone  says:
This is the good step u have taken. Thanks for writing this, its clear you have spent a good amount of time on your sites development. Reply

Post a comment





(Maximum characters: 1200). You have 1200 characters left.



Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.