Reid Research runs a good poll. Pity TV3’s reporting of it doesn't match.

Duncan Garner’s report on TV3’s latest poll was poorly done. I usually think our TV political journos do a reasonable job despite their lack of familiarity with the statistical concepts they are reporting on. Not today.

(I should note here that some of what I am about to say is not the best of news for my favoured parties of the left. But I am saying it anyway. I am not a fan of the cognitive dissonance involved in believing any analysis that happens to show your team in a positive light. I would rather deal with the truth, even if it not pleasant to hear.)

The horse race
Garner reports that Labour has made up over eight points on National since their last poll, the largest one poll change this electoral term. While trivially true, this statement is silly and misleading. Why? Because the previous poll was probably wrong.

TV3’s September poll has National on 57.4 and Labour on 26.6, a gap of almost 31 points. That was a decidedly high-side estimate. Our Poll of Polls, which uses multiple polls to come up with a overall analysis of the political scene, estimated the National – Labour gap at around 24 points at that time. That TV3 poll represented the second highest estimate of National’s support all year, coupled with the second lowest estimate for Labour. It is little surprise that the gap narrowed from that level, because the earlier high level was likely due to sampling error. Of course, there was no mention of this possibility in Garner’s report. Last month’s poll was, like usual, taken as gospel. Which is pretty dumb.

Why the change?
Duncan does not explicitly speculate about what caused the shift in support. But in the opening paragraphs of his report on the poll, he does discuss National’s campaign launch in some depth, and in the broadcast version of the same report he discusses the first week of the campaign, with its various TV ads and policy announcements and campaign launches.

Which is all very interesting, except that the TV3 poll’s last day in the field was October 26th. Most of the interviews were actually done before the RWC final, and all were done before the big opening TV productions, before Labour’s superannuation / savings announcement, and before National’s campaign launch.

Linking any of those things with the poll result is lazy journalism. Viewers did not have access to the poll’s field dates, and it is the journalist’s job to present plausible explanations of the patterns the poll reveals. TV3 did not do that, instead implicitly inviting viewers to draw spurious links between unrelated events.

What if?
The frame many journalists are putting around recent polls is that if Labour surges, the Greens keep surging, and New Zealand First surges, then a coalition of the three parties has a shot at taking out a wilting “National + hangers-on” grouping. Here is why that kind of analysis is simplistic: The continued surge of the Greens hurts Labour’s ability to surge; and vice versa; and there is no sign whatever of the needed New Zealand First surge.

I posted recently about the relationship between changes in Labour’s vote and changes in the Green vote. When Labour picks up one percent, about half of it appears to come from the Greens, and half of it comes from elsewhere, most likely National. That pattern makes is really hard for both Labour and the Greens to surge at the same time unless the relationship between Labour and the Greens changes dramatically in the next four weeks. There is no sign of that. And one look at our Poll of Polls estimates for New Zealand First tells us they have been trending down steadily and are most unlikely to regain any seats this year.

So, where do we stand?
The left-right gap is around 16% with four weeks to go. The potentially-up-for-grabs parties look like winning around six seats. New Zealand First is a goner (I know, I know, but I’m calling it anyway.) That means for the left to have a chance, National has to drop below around 47% (give or take 1% depending on Epsom), with all that lost support going to the left, most likely Labour. The last time National polled that low was in early 2009, we estimate their support has been permanently above 50% since that time, and their trend is up, not down. The last time Labour and the Greens together polled 42% was in late 2008, and their combined trend is mildly down. But the campaign, short though it is, is just now underway. That is where we stand.

Comments (9)

by Tim Watkin on November 01, 2011
Tim Watkin

I'm interested in the margins in that poll... NZF up slightly. Writing them off too soon Rob?

And the 'all others' is up to 2%. Very margin of error stuff, but the largest since the last election - other small parties about the register?

It'll also be interesting to see whether UF can get off zero, with a decent ad, Dunne doing well in the Q+A debate etc.

by Chris de Lisle on November 01, 2011
Chris de Lisle

It all sounded a bit like TV3 has realised that a foregone National victory makes for a boring election campaign and are trying to make it seem like 'anyone could win it!'

I wonder how the way polls are reported afftects the attitudes of the voting public. Now that we've been given a (phony) narrative, will it become reality?

by Rob Salmond on November 01, 2011
Rob Salmond

@Tim: I checked on NZF. TV3's previous poll had them down at 1.9, compared to a Poll of Polls average at the time of around 2.37. So seeing them move up to 2.4 in the latest TV3 poll is a vaguely intereesting move in relation to the earlier TV3 poll, but not in relation to the rolling average. I wouldn't read much into it, especially considering everything we're talking about is well within the margin for error. I'm happy to stand pat with "they are a goner." If anyone wants to have a charitable wager on this, I'd be happy to oblige. I already have one with an ACT candidate who bet me ipredict had it right when it said they would get 5% and 6 MPs. Easy money. My charity is looking up!

by Ian MacKay on November 01, 2011
Ian MacKay

Chris. Some countries ban the publication of polls for some weeks prior to polling day. So should we.

by Tim Watkin on November 01, 2011
Tim Watkin

It's an interesting question Chris... If media report their polls simply by saying "here are the numbers", then putting the tables on the screen or the page, you can give neutral coverage. As soon as you put any words around that, pointing out trends or changes, you're choosing to point out some things rather than others and there's implied bias.

If TV3 had said "Nats still out in front, Labour's chances are pretty small", they would have been lambasted. I take Rob's point about interpretation and not being afraid to say 'hey, our last poll was probably at the high end, so looking at other polls this is probably a correction'. But I'd rather than media talked about the election as a live race and not a foregone conclusion.

by Tim Watkin on November 01, 2011
Tim Watkin

I guess Ian's approach is the other one... no polls at all.

Of course that means more power to individual journalists and media businesses, who get to choose topics of discussion, just without reference to a measure of public opinion...

by Rob Salmond on November 01, 2011
Rob Salmond

I think there is a middle ground that doesn't force media to publish no polls at all or publish in a "just the facts" frame (Tim's fear), and also doesn't involve the same kind of silliness I saw in TV3's report (and some others from earlier in the year). By all means media should place some context around a set of poll numbers, and provide trends etc. Their job is to be analysts, so they should engage in analysis. I think their mistake is to take the analysis part too lightly.

Simply listing a set of events that occured at the same time as the poll and nudge nudge suggesting they might be causing the results isn't good enough. If certain events are causing the polling shift, then the shift should be concentrated in particular demographics. Are they? If you don't know, ask the polling firm. They know. Be a sleuth about it - that it what turns the reporter into more than a person in a suit reading at a camera.

Treating last month's poll with the same caution you afford this month's poll is also a pretty good, basic start.

Finally, looking at larger trends helps put really insightful context around a poll. You don't even need a poll of polls for that, although they help! For example, the knowledge that Green surges tend to come at the expense of Labour doesn't take a rocket scientist to suppose, is reasonably easy for an analyst either at the media firm or at the polling firm or a talking head to check, and puts *helpful* analysis around the numbers.

My issue is not the presence of analysis, it is that the analysis is poorly done.

by Matthew Percival on November 01, 2011
Matthew Percival

The best bit had to be Garner's interpretation of the Rena poll. Roughly one third thought the government had performed poorly, one third thought they did OK and one third thought the government had performed well.

Garner comes out the the brilliant line of 2/3rds of respondents thought the government had only performed ok or poorly!

Yes Duncan, but 2/3rds also thought the government had performed OK or well!

by David Hall on November 02, 2011
David Hall

"Do you ever get the feeling that the only reason we have elections is to find out if the polls were right?” Robert Orben.

Post new comment

You must be logged in to post a comment.