Their popularity makes accurate polling more difficult

As more and more Americans, particularly younger people, do away with landline telephones and begin to use cell phones exclusively, experts on public opinion research both inside and outside of the industry have been struggling to come to terms with how this might affect the accuracy of telephone polls that attempt to measure the public’s political leanings and how to adjust to that new reality to make their polling as accurate as possible.

Pew Research just completed a study that compared the political opinions of those who use cell phones exclusively or predominately to those who continue to use primarily landline telephones. The study notes that 14.5 percent of Americans currently use cell phones exclusively, while another 22.3 percent have both landlines and cell phones but receive all or almost all of their calls on their cell phones. That number is expected to continue to grow over the next few years, and as it grows the differences between those two groups should lessen.

It will surprise no one that the demographics of those who use cell phones exclusively or predominately differs significantly from the demographics of those who use landlines predominately. But the study also found that there were significant differences between the cell-only group and the cell-mostly group:

Continued – 

The cell-only and cell-mostly respondents in the Pew poll are different demographically from others. Compared with all respondents reached on a landline, both groups are significantly younger, more likely to be male, and less likely to be white. But the cell-only and cell-mostly groups also are different from one another on many characteristics. Compared with the cell-only, the cell-mostly group is more affluent, better educated, and more likely to be married, to have children, and to own a home.

The Pew study also found interesting differences in political views and voting tendencies among these groups. Across the board, the first three groups (landline-only, landline-mostly and cell-mostly) were fairly close together in terms of political opinion, but the cell-only group were significantly different from those three. For example, when asked which candidate they were leaning toward voting for, the Obama supporters were fairly close in the first three groups (46 percent, 48 percent and 49 percent respectively) while the cell-only group supported Obama 61 percent of the time. When asked whether they were likely to support the Republican or the Democrat in their local congressional races, the first three groups were all at 52 percent for the Democrat, while the cell-only group was at 63 percent for the Democrat. Those are significant differences.

The problem for pollsters is that the cell-only group is underrepresented in polls. James Hanley, a political scientist from Adrian University who teaches a course in research methods, told the Michigan Messenger that cell phone users do get called by pollsters in proportion to their numbers but they are less likely to respond:

The large polling firms use random digit-dialing software, so they do call cell phones as well as landlines. The real problem is that few landline owners have caller ID, whereas every cell phone automatically displays the number that is calling. So cell phone users are less likely to answer, because they can readily see that they don’t recognize the number.

Because of the much lower response rate of the cell-only group, pollsters have to call many more cell-only users in order to get the numbers of cell phone respondents to be in proportion to their percentage of the population. That means that getting a truly representative sample for their polls is considerably more expensive. For those pollsters who can’t afford the additional expense, their samples may simply not be representative of the population as a whole but only representative of that portion of the country that uses landlines primarily. And given the significant differences in political opinion between the two groups, that can have a dramatic effect on the accuracy of the result.

So how have the pollsters responded to this and attempted to keep up with these demographic changes? One of the most prominent publications in the business, Public Opinion Quarterly, published an entire issue last year devoted to this question. The experts in that issue discussed a range of factors that pollsters should take into account, from the time of day cell phone users are most likely to respond to the fact that cell phone users may be more likely to be doing other things while talking on the phone, thus possibly reducing the quality of the survey.

There are basically two possible categories of responses by pollsters: change the way they gather data in order to insure that they have the proper proportion of cell phone users to landline users in their data sample (which is much more expensive to do for the reasons stated above) or apply demographic weighting techniques, mathematical adjustments to the results that are intended to mimic the results if they had a truly representative sample (which is less likely to be accurate). John and Patrick Ehlen discuss the problems with using demographic weighting in one of the articles in the Public Opinion Quarterly:

Regardless of how weighting may be approached, it seems reasonable to conclude that many differences between the landline and cell-only populations arise from differences in demographic factors, such as age. However, establishing appropriate values for weights is difficult, since the size and the demographic characteristics of the cell-only population are moving targets. Those 18 and over with cell phones and no landlines increased from 2.8 to 11.8 percent in the United States from the first half of 2003 to the last half of 2006 (Blumberg and Luke 2007), showing an annual rate of increase of about 51 percent. Such a rapid increase in the size of the cell-only group raises the specter of equally dramatic shifts in demographic characteristics over time, making it difficult to maintain appropriate demographic weights.

The authors note that while greater adoption of cell phones among other demographic groups should reduce the differences between cell users and landline users over the long term, in the short run, those differences are likely to increase, which only magnifies the difficulty of developing accurate demographic weighting to mimic real world results. The only really workable answer appears to be better sampling to ensure that the sample is truly representative, which is prohibitively expensive for some pollsters.

So what does this mean for the rest of us? It means that in looking at a given poll result, it is important to take a close look at the methodology of the study to see how they accounted for this issue — if they accounted for it at all. If the poll has a sample group that includes cell phone users in proportion to their percentage of the population, it is more likely to be accurate. If the poll uses demographic weighting to account for it, it may or may not be accurate depending on the exact techniques used. If it does not account for cell phone users at all, it is likely to be significantly inaccurate.