On ABC Radio National’s new Drive programme on Friday the authenticity of research came under discussion between host Julian Morrow and guest Amber Jamieson @ambiej in the Twitterati segment . 

Media reports indicated research had found it is now more expensive to live in Sydney and Melbourne than in New York New York. But an insightful blog article from a Matt Cowgill (here) debunked the media reports, observing that the research failed to take exchange rates and inflation into account.

Morrow placed fault with the media reporting on the issue, since the research had the valid purpose of assessing the expense of various cities for people working internationally and being paid in US dollars.  Authentic research had been mistreated by the media, a common enough phenomenon, anecdotally, and one that does occasionally receive media attention itself.

The research in this case was wrongly used to fabricate a story for media outlets.  But unless the story results in a flood of optimistic Australian emigrants falling on hard times in the Big Apple, it’s a relatively harmless example.  There are worse forms of abuse, when research credible and otherwise generate ‘facts’ of various degrees of inaccuracy  that populate the media and, whether we like it or not, contribute to our picture of what the world is like.

On the RN Drive programme on Friday at 730ish a worse form of abuse was imminent, since the political panel discussion that closed the show carried a particularly crass example of poorly represented public opinion supposedly grounded in (unreferenced) research.

Morrow described his Friday political panel as a ‘high council of political wisdom.’  This week the press were represented by Mischa Schubert, the tiny pseudo-intellectual class of tankthinkers by Chris Berg of the IPA, and the people were represented authoritatively by Ipsos Research Australia director Rebecca Huntley. Not the interests of the people. The opinions. Huntley, apparently, has her finger on the pulse. She can read the ‘public mind’:

Morrow:  Has the public mind moved to the opposition and the prospect of an Abbott government?’ 

Huntley:  The leader of the Opposition hasn’t come up much since the election. The general perception of him during the election is that he ran a good election…a disciplined election.

This is a casual comment on an end-of-week panel discussion on RN’s relatively light Friday Drive show. But in research terms – in terms of facts and data – Huntley’s made a big claim. It was only the first:

‘Despite the fact that the polls continuously say that Kevin Rudd is popular with people, what is clear is that the idea of him coming back isn’t one that makes people happy. People think the entire thing is a disaster.’

‘When people talk about the Labor time in Government they can’t believe it’s only been four years. For them it feels like eight years.  They actually have a feeling – the kind of fatigue you see from voters – that normally comes after three or four terms.’

This was how Huntley described the public mood following the election of the Labor Govt in 2007: 

Of course it started with an incredible sense – and we got it in our groups, which was extraordinary, cause most people don’t give two hoot about politics – but this sense of time for a change and excitement. And then a little bit of bewilderment – what’s this guy on about – and then a bit of concern around the GFC, and then some real frustration, and then just this sudden kind of this sudden act of getting rid of this man, that people were a bit worried about but weren’t actually ready to toss.

At least, Huntley gave a superficial impression of being an authority: on public opinion, on the Public Mind.  But she did none of the work that goes along with being an authority on a subject. Her language was sloppy (‘There was this kind of sense that..’). She did not give statistical evidence.  She did not refer to the methodology employed by Ipsos Research Australia, or its limitations. Specifically, she made no mention of the sample sizes of quantitative research, did not report what questions were asked, gave no account of the conditions in which the research was undertaken or how the participants were selected. 

Research companies often employ commercial databases or build their own databases of respondents. These are clearly not representative of the populace in general. To what extent is the research referenced here generalisable to the population of Australia? To the extent that Huntley’s research is  focus group based, it would again be instructive  be informed how many people have been spoken to, the method of interview,  and again, how the citizens were selected. 

None of this was raised in the panel discussion.  There was no mention, even, of the difficulty of capturing and presenting public opinion in an accurate way. 

Surveys conducted by market research organizations are often not based on random sampling, which makes them ineffective.  Questions asked in surveys are not necessarily available. Surveys are frequently of the ‘omnibus’ variety, concerning a range of different topics.  When we hear that people answered a particular way to a particular question on a particular topic it should be remembered that the people who gave those answers may have been on question 46 of a 60 question survey that asks about the supermarket shopping experience, nuclear energy, wine and gay marriage.

Not only does the wording of survey questions have a significant effect on results, the order and number of questions can also affect respondents answers.

Despite this total lack of the sort of rigour usually associated with credible research – and no mention of the specific research supporting broad political statements –  Huntley carries the credentials and readily assumes the knowledge to speak directly for the people.  Jules Morrow introduced Huntley as a director of Ipsos and reinforced her position of authority by asking her directly for information on the tenor of ‘the public mind’.

Huntley’s rhetoric is couched in a small cache of phrases like  ‘There’s a view.. People think…   They feel that… What is clear is that… We get very strong messages…’  even ‘They’re, like…’

These are very broad and non-specific catch-all phrases that brush aside any obstacles with respect to the source of their veracity. But as rhetoric some of these propositions have the form of the fallacy Argumentum ad Populum, the assertion that because many people believe something to be true it must be true.  The presentation of findings from a research company is ideally analysis, not persuasion.  These statements, though, have the form of the latter.

Huntley’s credentials as a researcher with a market research company provide grounds for broad generalisations about public opinion, taking the form of assertions about what most people think or what some people think.  After making a few assertions containing phrases like ‘people think..’ and ‘The perception is..’ and ‘There’s a view..’ anything Huntley said seems as though it probably must quite likely also be what most or many people think:

Morrow: Isn’t it the case that at some mystical point..

Huntley: Yes.

Morrow: ..people just turn off.

Huntley:  It’s true. And there’s this sense of which it’s not unlike a relationship.

Huntley says at one point of Opposition Leader Tony Abbott  ‘… the reality is that he now looks like one of the most stable people in the Federal Parliament.’  .Now this is confusing. Who believes this?  Huntley?  The Australian populace?  Who said this? How many people? In what forum? How were they selected? What were they asked, or how was the discussion steered?  If this is the reality it’s all terribly vague.

 Huntley is reprising a role that others have taken (spokesmouths from Rehame and Media Monitors are past and local examples) as a chronicler of public opinion.  But a for-profit multi-national company such as Ipsos-McKay (locally Ipsos Research Australia) has its own interests, makes decisions in these interests about what research it should undertake, what questions to ask, how to present results and conclusions.  This introduces a bias that must, at least, be taken into account.

In Europe, for example, Ipsos has conducted a series of surveys that included questions about the nuclear energy industry.  The first bits of research in this series were actually commissioned by the UK’s Nuclear Industry Association.  Later research was done spontaneously by Ipsos.  The latest iteration of the research has found that public support for the nuclear energy industry is surprisingly high, despite the relatively recent disaster at the Fukiyama nuclear power plant in Japan.

Given that the British nuclear energy industry’s lobbying or ‘umbrella’ organization has been a client of Ipsos it is clear that continuing this line of research could be of advantage to Ipsos, directly or indirectly.

If being cynical about arrangements such as these it might be suggested that Ipsos not only has an interest in conducting research in certain areas of interest to clients, prospective clients, friends and other companies associated with Ipsos, there is also a real vested interest for the company in arriving at certain findings vis a vis public opinion.  With respect to nuclear energy the company has an interest  in finding that public support for the industry has not been adversely affected.  This bias may be just a perception, but it is something that Ipsos should identify.  This is particularly the case since the company in Europe has personal as well as professional links with the nuclear industry. Ipsos director Yves-Claude Abescat is also an executive head of French finance giant Societe Generale, which defends its funding of nuclear power more directly on its own website   Abescat’s role at Ipsos may have nothing to do with the company’s self-funded research into public support for the nuclear industry, or the finding that there is a relatively high level of public support for the nuclear industry.  But there’s the perception..

Bias or perceptions of bias should propel Ipsos to take further care in the preparation and preparation of research.  There is always the danger that the company could be seen as attempting to manipulate rather than simply record the tenor of the public mind.  The same perception of vested interest could also affect the credibility of political research.  If a company such as Ipsos, operating in economic environments , finds it has a preference for one political party or another, there is at least the problem of a perception of bias by way of vested interest.  It is natural that a large company might favour the party promising lower corporate taxes, for example, or a more laissez faire economic environment.  A party that is a better friend to a big business organization like Ipsos.

In this case,  it is clear that Rebecca Huntley’s comment on Radio National Drive is susceptible not only to claims of a lack of rigour, or at least the presentation of such, but also perceptions of bias based on what is good for the company.  This is not to make the assumption that all research is suspect because based on self-interest.  People who heart big corporations are often heard to remark that absolute focus on enrichment of the corporation is in the nature of that kind of organisation , and that it is well and good that this is so.  This ethos is honestly stated, but can only promote suspicion as to which are the presiding motives of the organization in all that it subsequently does.

Perhaps this blog-type tirade is all silly pedantry.  It’s the Friday evening drive time political panel, part of Radio National’s very slightly funky revamp, not intended as the most serious of political discussions. It is political discussion, though.

The presentation of research into public opinion is important because this sort of stuff – generalisations and statistics in the media – is the source material for what we as a community, a nation, a city, an age group, collectively think and believe.  Whether we like it or not, whether we’re cynical about it or not, having spent a lot of time thinking about how the media works or not,  it tends to affect, to some degree, our overall perception of ‘what society thinks.’   Public opinion, perversely, has a powerful influence on individual belief. 

Even where people reject the rest of society in forming their own views and convictions, work from first principles, a self-sustaining code, their actions in relation to their beliefs are affected by their idea of what society thinks.  What we think others believe powerfully influences aspects of our own identity in terms of belonging, isolation, and so on. Certain views are made acceptable or unacceptable, by virtue of public opinion and the way that we as a society measure it, talk about it, represent it.

Picking on Huntley and Radio National in particular may not be especially fair, but this case is representative of a more general thread that runs through the Australian media that that assumes public sentiment is easily measurable, known, knowable, suggesting it’s alright to casually summarise the public mood. 

Anyone who considers a handful of the people they know and the range of beliefs and opinions those people hold – on politicians, on social issues, in terms of what they think the universe is like, what they like and dislike – the idiosyncrasies and oddities and contradictions in the opinions of a few friends or colleagues or family members – must realise the fantastically complex task at hand in mapping and reporting the public opinion of a nation.  Impossible really, and the approximations that result should be labeled as such.

In this context the poll results printed on newspaper front pages and reported on in headlines on network news are gross approximations of the complexity and variation of sentiment in the Australian populace.  They fail utterly to record, for example, the citizens’ cynicism, complaints, disgust with the political system in general, forcing people into expressing a preference for one of the small range of potential political leaders they don’t choose.

An account of public opinion like the one given by RN’s Ipsos researcher is far worse. Research is not meaningless.  Nor is the idea of expertise. Rebecca Huntley may be full-time in the middle of the flow of political discourse, and this does bring insight.  Her research methodology may be rigorous, her focus groups well chosen, well directed, under good, readily repeatable conditions.  But the best research is presented together with information about its methodology, and its limitations. 

The best research findings are clear and specific.  Comment should be presented carefully, with proper qualification. Where there is not time or room to include qualifications, methodology, questionnaire questions and so on, there is the capacity to reference other media that does do that job in a publicly accessible way.  

It’s fair to speculate on the sentiment of the public in general, to make suggestions as to what the mood might be. It’s fair to present sound research, together with caveats, as an indication of public opinion. Taking care to maintain a high standard in this regard would befit Radio National and benefit Ipsos and its representatives.  Presenting public opinion as a fait accompli does both research and the ‘public mind’ a disservice.