Margin for Error in Alberta: Not all the Polls Were Wrong

Posted by on Apr 27, 2012 in Blog | 0 comments

Bruce Cameron, President of Return On Insight, April 25, 2012

The day after the most surprising election campaign in Alberta’s history, I fielded dozens of media calls. The first question wasn’t, “How did the PCs engineer one of the most amazing comebacks in Canadian election history?”

No, the first question was always, “Why were all the polls wrong?”

This is an extremely important question the polling industry must answer. But not all the polls were wrong. Return On Insight was the only polling firm that predicted a PC government.

On air, during the CBC election coverage, I said, “We weren’t as wrong as everyone else.” And although the studio audience laughed, and host David Gray quipped, “not a slogan you’ll want to use on your business cards,” the important fact was this: ROI saw the vote shifting toward the PCs and we predicted the 41-year-reign of the Tories would continue.

How did ROI draw that conclusion? And why did so many other pollsters and pundits miss it? (One online pollster was so convinced of a Wildrose majority — something he bet against before the writ dropped — that he paid off his bet to a Wildrose strategist five days before the vote.)

One week before the election we saw the PC momentum begin to gather strength, similar to a big swell forming offshore.  I offered the wave analogy on election night, adding that our mistake was underestimating just how big that PC wave would become as it started breaking on shore. A friend of mine who heard the comment sent me a tweet:

Parker Hogan (@TPHogan) 4/23/12 9:50 PM  @roitweets the best surfers see the big wave far out in the ocean before the others, and ride it to victory when the wave swells.

Strategic voting played an essential role in the Alberta outcome, and ROI identified that potential voting shift, noting that “if Liberal support continues to decline in Calgary, then the PCs will win many close races in that key battleground.” We also observed that one week before the election, strategic voting had not yet emerged in Edmonton, but we predicted that if and when it did, the PC tide would overwhelm earlier Wildrose momentum.

In fact, telephone polling ROI conducted just days before the vote in a cross section of different Calgary ridings indicated an electorate on the edge: equal proportions of the electorate were undecided, leaning Wildrose, or leaning PC .  Delving beneath the surface, we followed up on our instinct that the election was becoming a referendum on Alberta’s future. We asked which side of the divide Calgarians would choose—PC or Wildrose. The result? Three days before the election, we saw that if Calgarians had to choose one of the two major parties, more would choose the PCs than the Wildrose.

So why did so many other polls predict a Wildrose majority?  Why were many of the polls wrong?

  1. Mixing Methodologies: Not only was the Alberta election a volatile one, but the polling industry and media outlets who reported on those polls used three different methodologies: telephone surveying, online surveying from panels of voters, and Interactive Voice Recognition or IVR push button polling. The use of all three methods illustrates the extent to which the polling industry has arrived at an important cross-road. ROI maintains that while no single methodology is 100% perfect, the method we used for the Alberta election (telephone surveying) yielded the most accurate prediction. A well-crafted survey, conducted by real people, via telephone, based upon a scientifically drawn random probability sample, yielded more accurate results than either online surveys or IVR polling.
  2. Herd mentality: The common theme for most of the campaign was that Wildrose support was very strong across southern Alberta and throughout Calgary. Many polls simply confirmed that trend. Sun Media was the most blatant in its pro-Wildrose coverage, at times doing more cheerleading than reporting.  ROI polls conducted for CBC showed the PCs ahead in Calgary one week before the vote, but many others dismissed that finding because it didn’t line up with their own poll results.
  3. “Calgary”: With such a volatile and shifting electoral battle, regional definitions took on more importance than usual. For instance, many pollsters reported that the Wildrose had an eight- to 10-point lead over the PCs in “Calgary,” but failed to mention that their definition of “Calgary” was the Census Metropolitan Area, which includes outlying areas like Okotoks, Airdrie and Chestermere. In those exurban fringe areas, Wildrose had a lot of support. ROI defined “Calgary” as the City itself, not the outlying areas, which revealed the PCs in the lead. We published that poll via CBC on April 16th.

But the most important barometer of accuracy in polling is the sample frame from which the respondents are chosen. A survey sample framework has to be as broad, inclusive and representative as possible, and this is where online pollsters with relatively small panels went wrong.

So, what did we learn?

First, the polling industry needs to get its house in order. The public rolls its eyes when pollsters debate the methodological nuances of IVR versus online versus telephone approaches. But ROI maintains there is no single solution without shortcomings.  And those shortcomings need to be objectively analyzed and accounted for:

  • good telephone surveying needs to account for the increasing incidence of people without landlines,
  • online survey companies need to pay more attention to the size, diversity and freshness of their panels,
  • and IVR firms need to stick to what they do best: identify voters for political parties, not predict election outcomes for media outlets.

Equally culpable in mis-reading the Alberta election outcome were some members of the media.  The media has a responsibility to increase their understanding of the differences in accuracy between various polling methodologies. And that means avoiding mistakes like commissioning cheap overnight IVR surveys and then passing off the results as the gospel truth.

But most of all, seasoned political watchers learned yet again that there is no substitute for insightful analysis and judgment, derived from political experience. When the full story of the Alberta election is written, it will be noted that the two pollsters who got it right (out of 20), were myself and Greg Lyle, whose firm Innovative Research were the internal pollsters for the PC party.  What do we have in common?  We both spent part of our long public opinion careers learning from one of the masters in the profession: Dr. Angus Reid.

Return On Insight is a strategic research consulting company with offices in Alberta and British Columbia Canada. ROI analyzes public opinion and explores consumer insights for public and private sector clients seeking to develop communications initiatives, improve brand positioning, and assess audience and stakeholder engagement efforts.

 

 

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *