How not to do a public survey

Traverse City Light & Power, the public utility that supplies power to that area, sent out a press release this morning about a public survey they apparently commissioned from the Northwest Michigan College Research Services department. You can read the full survey here (PDF).

The survey was about what customers of the company valued most — low cost, environmental sensitivity, reliability, etc. — and it also asked their customers whether they supported a proposed biomass plant that the company wants to build to produce electricity. The press release says:

By over a two to 1 margin, a recent survey of Traverse City Light & Power customers shows they are “very supportive” or “somewhat supportive” of L&P’s initiative to build a local biomass plant, compared to those saying they are “somewhat unsupportive” or “very unsupportive.”

Which is true. But like any poll, it’s always a good idea to take a look at the methodology used to reach those numbers because there are any number of ways those results can be skewed in either direction, whether through sampling errors or biases, subtle or misleading wording and so forth.

And this poll has one of the most blatant types of methodology problems. Immediately before asking the question about support for the proposed biomass plant is asked, the questioner delivered a narrative clearly supporting one side or the other — in this case, clearly supporting the pro-biomass side. That narrative said:

The next question addresses renewable energy. Traverse City Light & Power is currently investing in wind, solar, and landfill gas electric power generation. To further diversify, Light & Power is considering building a local renewable power plant which uses wood chips, commonly known as biomass, for fuel. This plant would be relatively small in size, fully enclosed, and located in an industrial area. The plant’s emissions would be less than coal, would meet Federal and State standards, and would likely increase truck traffic in the immediate vicinity. An independent evaluation of the impact on Traverse City Light & Power customers has determined that biomass is lower cost than other energy sources, including wind, solar, coal, and natural gas.

I called TCL&P to make sure that part of the text was actually read to the respondents as the poll was being taken, and Jim Cooper, their marketing manager, confirmed that it was.

This is an obvious way to skew the results of a poll and get the results you want. A proper and more objective way of asking the question would have been to merely present the basic facts — “Traverse City Light & Power wants to build a biomass plant that would burn wood from Michigan’s forests to produce electricity” — and then asked if they supported that proposal.

When you start the question with a narrative supporting the idea, you’re biasing the results right from the start. Cooper’s defense of this was that the claims in the narrative were “true” and therefore they stand behind the results of the poll. But an organization that was opposed to the idea could poll the very same people and begin that question with a narrative containing claims they believe to be true and the poll results would likely shift considerably.

If your goal is to get an accurate reading of public opinion, you never introduce that kind of narrative into the poll taking process. That’s something you only do if you’re trying to reach a particular result with the poll, not if you’re merely seeking an objective measurement.

I have no idea what TCL&P’s customers actually do think about the proposal without any prompting from a survey taker pushing them in one direction or another. It’s certainly possible that they support it strongly. But this poll does not support that conclusion; it only supports the conclusion the company wanted to reach.