NBC News Monkeying Around With Us

You can survey your neighborhood or the block where you live, but there are not enough people there to enable you to do a poll, a valid poll that is. And that is the problem with too many polls. Sampling is the key to a reliable poll. Bad sample, bad results.

Here is a good overview for the challenges of polling in today’s environment, especially when 16 percent of voters don’t have internet and most voters have mobile phones. Getting the right sample for a national poll, around 1200 people, requires contacting at least three or four times that number. It’s a complicated business, and amazingly reliable when done right.

For a poll to be reliable—and communicated properly—it needs to establish its margin of error, usually plus or minus 3 to 4 percent, and determine its confidence level. Stuff above my pay grade, but this may help.

Before the 2016 election, MSNBC’s Chuck Todd regularly released surveys they did with a contract or collaborator known as Survey Monkey and call them Polls. It was shortly after Chuck Todd was the Chief Political Reporter of NBC where he did a great job getting down into the weeds of polling and finding the interesting stuff above the low-hanging fruit where most other networks stopped. Then the Monkeys showed up.

They are who they are: an online survey company for all sorts of applications from customer satisfaction surveys to marketing studies. I used them years ago when I did a survey with The Wharton School among their students. That is how they have a data base of over 3 million people—former customers and users. Nothing scientific here, just a collection of folks who have used Survey Monkey over the years.

When you use Survey Monkey for general purposes, as NBC does, a sample is taken of this narrowly drawn customer base: it should not be confused with a national random survey that most polling firms rely on. In effect you are asking a bunch of cyclists what they think of seat belts—they are both transportation questions, but the sampling base is not very useful to your seatbelt inquiry. Interesting, but not relevant. They are not a polling firm like the established Gallop Poll or reliable Pew Research Center.

But MSNBC and its parent NBC don’t mind confusing the two and leave you guessing.

Well over a year ago, MSNBC started inviting viewers to critique the kind of job they were doing. My complaint, I am sure I was not alone, was they were running surveys but calling them polls–letting you figure it out. I never expected to hear from Chuck, but one night when he was giving feedback on what viewers were saying. Chuck said, “Some of you are upset at us for not making a distinction between surveys and polls. Get over it.” I don’t exactly remember the first part, but I’ll never forget the last part: “Get over it.”

Then all was quiet until recently. Zoo unattended, budgets tight, the Survey Monkeys are back, and hopping around. And NBC is back to calling surveys polls.

If you think this is nothing more than distinction without a difference, consider this. Polls are usually closed ended questions conducted by polling firms, companies that specialize in taking polls. Surveys have open-ended questions and are usually conducted by universities, academic organizations and research firms. If that is not a big enough difference for you and you end up saying this post is meaningless, then we have arrived at a point–sooner than I thought–where words have no meaning.

If you are still hanging in there on this one, check out the difference between Survey Monkey and NBC on the question of who is to blame for the recent shutdown of government, and the results of a more reputable, albeit generally considered “conservative favoring” polling firm associated with Quinnipiac University.

We doubt the Greek Chorus for this post, by A Tribe Called Quest, will be in any Monkey Survey or Gallop Poll. That is precisely why the editorial board chose them.

Bottom line: be vigilant, check the sampling error range and stay within 3 percent plus or minus, keep the monkeys in the zoo.