"Government too generous with aboriginals, Canadians tell pollster."
Poll published June 30, 2012
Post Media/Global Television and IPSOS-Reid
On June 30, 2012, Post Media News and Global Television published a story reporting on the results of a recent public opinion survey they commissioned from IPSOS-Reid which touched on Canadian attitudes towards immigrants and Aboriginal Peoples.
Postmedia Network Inc., a wholly-owned subsidiary of Postmedia Network Canada Corp, is the largest publisher by circulation of paid English-language daily newspapers in Canada, representing some of the country’s oldest and best-known media brands. IPSOS-Reid is a highly-respected Canadian research company that is part of IPSOS, one of the world's leading survey-based marketing research firms.
This study consisted of four questions placed on an IPSOS-Reid online omnibus survey, conducted between Jun 11 and 18, 2012, and conducted on behalf of Post Media and Global Television for public release. The four questions are as follows (in what appears to be in this order):
1. To what extent do you agree or disagree with the following statement: “Canada should let in more immigrants than it currently does.” (response categories: strongly agree, somewhat agree, somewhat disagree, strongly
2. To what extent do you agree or disagree with the following statement: “Canada’s aboriginal peoples are treated well by the Canadian government.” (same categories)
3. To what extent do you agree or disagree: “Canada’s aboriginal peoples receive too much support from Canada’s taxpayers.” (same categories)
4. "Would you say that immigration has generally had a positive or negative impact on Canada?” (response categories are: very positive, fairly positive, neither positive nor negative, fairly negative, very negative, don’t know).
Documentation of this research includes the online story appearing in Post Media outlets and a news release available from IPSOS-Reid through their online subscription service. No video record of any stories appearing on Global TV could be found.
This commentary addresses the two questions on Aboriginal Peoples. The conclusion drawn from this research is that Canadians as a whole believe the country’s Aboriginal population are getting more than their fair share of the public’s hard-earned tax dollars (see headline above).
This piece of research is but one of many such media-sponsored polls that are published with great frequency in the Canadian media, so why does it merit an in-depth commentary? When a survey generates a headline in the national media that draws a provocative conclusion related to one of the most sensitive sources of social tensions in Canadian society today, it is important to know if the conclusion is sound enough to justify such a public statement of social fact. In such cases, there are three questions to be asked: 1) what does the research tell us?; 2) how well was it reported?; and 3) what purpose is served in publishing it?
1. What does the research tell us?
The central question to ask is whether the conclusions published are appropriately supported by the research conducted, based on established standards for public opinion research. There are several reasons why the answer may in fact be no.
a. Question structure. The “agree-disagree” structure used for both survey questions is a common form used on opinion surveys, and one that has strengths and limitations. It is an effective structure when the goal is to present or get a response to a particular idea or point of view. The primary limitation is what is referred to as “confirmation bias”, which means a tendency for respondents to agree rather than disagree to any statement (apart from their own particular opinion), simply because the statement is presented.
This bias is often subtle but has been demonstrated in studies, and the problem is that it cannot be quantified so there is no way to know in a given case how much of a bias is taking place, and no way to “correct” for it. On this survey, the fact that a majority of Canadians do in fact agree with both statements is evidence that such bias may be at play.
b. Question language. A more problematic aspect of these survey questions pertains to the language used to present the core ideas being tested, in particular the use of the term “Canadian taxpayers.” It is correct that a portion of federal government revenues come directly from taxpayers, but it is by no means the only source (in 2010-11 only 56% of federal government revenue came from individual taxpayers). But more important is the fact that the word “taxpayer” is anything but neutral or objective; it serves as a form of code that activates citizens’ sensitivities about the taxes they themselves pay.
This means that any survey question about public services, programs or spending that is phrased in terms of what it takes from taxpayers focuses on the costs rather than the benefits, and so can be expected to elicit a negative response. Ask the public about how much taxpayers are paying for other public programs and services (e.g. health care, defence, school teachers) and one might expect a similar reaction.
Alternatively, if the same question were phrased a bit differently, say, “Canada’s aboriginal peoples receive too much financial support from the Canadian government for education and employment training”, the response might have been different.
c. Response options. While most people have opinions on most topics covered in this type of survey, oftentimes they are not well-informed about what they are being asked about, and in some cases do not feel like they can or want to offer a response. For this reason, most survey questions provide an option for respondents to decline a response.
In this survey, neither of the two questions on Aboriginal issues offered respondents this option, which means they were required to provide an opinion whether or not they had one. This introduces yet another source of what researchers call “measurement error”, meaning that some individuals answering these questions provided an answer that didn’t conform to what they really think. Moreover, there is no way to know how many of the 1,101 Canadians completing this survey fit this category.
In summary, the two survey questions used in this poll do not meet commonly-accepted research standards that provide the basis for concluding that the results accurately portray current public opinion on current government support for Aboriginal Peoples.
A more effective way of framing this question would be to first provide respondents with information about the level of government funding for Aboriginal populations (e.g., as a percentage of overall spending, and/or in proportion to other areas of spending), and then ask if this amount seems too much, too little or about right, or if in fact they do not have an opinion. This type of question addresses the limitations outlined above, and would offer a more accurate read on Canadian opinions on this important public policy question. In addition, it would also help to include additional questions to get at the issue from different angles, for instance to see what sorts of government programs and services for Aboriginal Peoples are most and least likely to be seen as effective and a worthwhile investment.
The Post Media story portrays a black and white storyline, but the public’s view may in fact be more complex. For instance, research conducted by the Environics Institute in 2009 (as part of its Urban Aboriginal Peoples Study) revealed a conflicted public: Most non-Aboriginal Canadians in the country’s 10 largest cities believed that Aboriginal residents in their city have as good or better access to needed government services as do other residents; but at the same time they were divided on whether Aboriginal residents receive adequate services in such areas as education, employment training, social assistance and child welfare.
2. How well was the research reported?
Apart from how the research was conducted, another important consideration is the way in which it was reported in the media and the amount of detail provided by which readers can properly evaluate the validity of the research against the conclusions. Apart from a headline (see above), the Post Media story accurately reflects the results collected by the survey. At the same time, there are two issues with respect to how this study was reported:
a. Reporting margin of sampling error. The story inaccurately states an estimated margin of error for this survey, perhaps as a way to promote the accuracy of the results. It is scientific fact that margin of error statistics apply only to probability samples, and not to surveys conducted with the type of recruited online panels used for this particular survey. This is well known in the research industry, and Canada’s research industry association (the Marketing Research and Intelligence Association) adopted a Code of Conduct and Good Practice for its corporate members in 2007 which states clearly that:
“Members… should… refrain… from making statements about margins of sampling error on population estimates when probability samples are not used.”
b. Transparency in reporting. The credibility of published opinion research rests on the confidence with which media and the public can place in what is published. And the credibility of this kind of research is starting to suffer because of an expanding number of polls being published with little evidence of well-established standards in how they are evaluated or reported.
How can the findings of any given survey be certified as accurate? Since it is not feasible to directly validate published findings by replicating the work, the only way to evaluate published research is against established research standards (to answer such questions as whether the survey questions are appropriately framed and the data collection is performed by appropriate methods). To make this possible requires that such information be publicly available to anyone wishing to scrutinize the work. A few leading media sources in the US, notably the New York Times, are setting a new standard for reporting on the research they publish.
The Post Media story on this particular survey included the standard information (question wording, national results, survey sample size and field periods), but there are significant limits on how much detail about a survey any media story can include. A more detailed media release and key data tables (which show all response categories and results by region and demographic groups) are available from IPSOS-Reid, but only through a subscription service ($105 per year).
This kind of “pay wall” for the details of published survey findings presents a sizeable barrier to individual Canadians and even journalists, and inconsistent with the principle of transparency in reporting. Every other research company in Canada (and elsewhere) make the details of published surveys available at no cost, either by posting them online or responding to requests. This should be the norm, and one that IPSOS-Reid should be encouraged to embrace.
3. What purpose is served by publishing these results?
The story reported from this survey is that the Canadian people “writ large” believe that the country’s Aboriginal Peoples are getting more than they deserve, and the taxes paid by others to pay for these benefits are not well spent. This surely reflects the view of some Canadians, but what proportion is a question that unfortunately is not effectively answered by this survey.
The broader question then becomes what purpose is served by publishing this research, in terms of contributing to the challenges facing Canada’s Aboriginal community, and its relations with the non-Aboriginal community. In portraying a supposed national consensus not clearly supported by the research, this story:
• May be telling Aboriginal Canadians that they are resented by everyone else for receiving benefits (well beyond the level they actually receive), and reinforce their sense of living under a persistent cloud of discrimination and stereotype;
• May be telling non-Aboriginal Canadians that, collectively, they are resentful of what the Aboriginal community is receiving (whether or not they as individuals actually think this), and lead them to conclude mainstream society is less sympathetic to the plight of the land’s first peoples than may, in fact, be the case.
This indeed would be an unproductive outcome from this story, and reflect poorly on the contribution of media-sponsored polling in our society. Published surveys on significant social issues have an important role and sometimes there is a need to portray uncomfortable truths. At the same time, such research has consequences, and it is incumbent upon those of us entrusted do this work to bear in mind our responsibility to our profession and to our fellow citizens.
Keith Neuman, Ph.D.
The content of this commentary represents the perspective of the author and the Environics Institute, and in no way reflects the views of the Environics Group of Companies.