Research Digest

New international survey about national economies reveals diverging views between developed and emerging countries

The latest international public opinion survey from the Pew Research Center focuses on citizens’ views about their nation’s economies and standard of living.  This study was conducted in March-April 2013 in 39 countries that include more than 10 countries in each three important categories:  advanced economies, emerging markets and developing economies. This is perhaps the most significant new research to come out of Pew so far this year, with several main stories that stand out:

First,  there is general unhappiness globally about national economies, with concerns about current economic conditions, rising economic inequality, prospects for the coming year, and longer term economic mobility.  Opinions are more negative than in 2007, revealing the prolonged impact of the 2008-09 financial collapse and resultant recession, especially in wealthier countries.  At the same time, personal financial circumstances have not suffered the same decline, and in many countries people describe their personal finances as good.

Second, there is a striking divide in views about national economies between the citizens of emerging market countries, and those in countries with advanced and developing economies.  People in emerging economies (which include the BRIC countries, as well as such countries as Chile, Turkey, Indonesia and South Africa) are more than twice as positive about their national economies as are those living in most wealthy countries in North America and Europe, as well as in the developing world covering most of Africa, the Middle East and parts of Latin America and Asia. This disparity is also sharply reflected in views about economic mobility:  Majorities in most advanced countries believe their children will be worse off than their parents (most notably in France, Italy, Japan and Britain), while the opposite view is prevalent in emerging and developing economies (and strongest in China, Brazil, Chile and Malaysia).

Third, Canada stands out (along with Germany and Australia) as a wealthy country whose citizens are comparatively positive about their national economy and current standard of living, and have weathered the global economic downturn better than most. Canadians are among the most satisfied with the overall direction of their country (as they have been for much of the past decade), and the most likely to rate their own personal financial situation as good (at 82%, tied with Malaysia at the top of the list).  Also notable is the stark difference in the opinions of Canadians and Americans, which Pew describes as “one continent but two distinct public moods.”  On almost every measure, Canadians are happier with their current economic lives, although Americans express greater optimism about the future.

The final notable aspect of this study is its methodology.  Of the 39 countries covered, the survey was conducted as a face-to-face interview in 20 of them, including such advanced countries as Italy, Poland, Israel and Greece. Publics in the remaining 10 countries (all advanced economies) were surveyed by telephone.  Given the current global trend toward online communications in all facets of life and commerce, this study serves as a valuable reminder that what today may seem like “old fashioned” remains an essential tool for effectively capturing public opinion in many parts of the world.

Major new report on non-probability sampling for survey research

A central foundation of public opinion research over the past 50 years has been the use of probability sampling, the method by which it is possible to extrapolate the results from a survey of 1,000 respondents to a country’s entire population with remarkable accuracy. Over the past decade, the viability of probability sampling has been seriously challenged from a combination of increasing costs, declining response rates and the ability to reach individuals by telephone (the principal means of drawing probability sampling). 

As a result, researchers are now turning to methods that use “non-probability” sampling, mostly in the form of online surveys that use large-scale “opt-in” panels of individuals who are recruited to participate (rather than being randomly selected from the population).  Online surveys have now largely replaced telephone surveys for most market research applications, and are increasingly used for political and public affairs research.

Examples of non-probability sampling include the following:

Sample matching:  Matching selected characteristics of a survey sample to those of the population (e.g. demographic characteristics such as age and education level). This approach is widely used in political polls conducted online ;

Network sampling:  Starting with a small group of individuals in the target population, and then leveraging their own social connections to identify similar types of individuals. This approach is typically used to sample rare or difficult-to-find sub-populations .

Because the rise of non-probability sampling is the result of necessity rather than scientific advancement, the big question is how well they work in terms of generating representative samples and accurate results?  On this question hinges the very future of survey research, and to address it the American Association of Public Opinion Research (AAPOR) launched a task force in 2011 composed of leading research experts from industry, government and academia. The Final Report was released in mid-May, and is a must-read for everyone who conducts, commissions or consumes survey research.

The Task Force examined the strengths and weaknesses of various non-probability sampling methods in terms of providing a valid scientific basis for extrapolating survey findings to larger populations, and ended up with a few bottom line conclusions.  In contrast to probability sampling, non-probability sampling methods:

•  Do not fall under a single framework, but comprise a collection of methods that comprise a continuum in terms of accuracy and precision;

•  Rely heavily on significant assumptions about the population under study and survey methods being used;

•  Carry a greater burden of transparency in terms of describing the methods used and assumptions made when results are published, for purposes of evaluating the credibility of the research; and

•  Lack a coherent framework and empirical measures for evaluating the quality of the sampling and therefore the accuracy of survey results.

 The Task Force concludes that the acceptance of non-probability sampling among survey researchers, clients of research and the media will require the development of an agreed-upon basis for empirically evaluating the accuracy of non-probability sampling methods.

The report answers few of the questions it raises, and is presented as a first step in stimulating a broader debate about the survey research methods in the 21st century.  Such a debate is critical if  survey research is to continue making an important contribution in generating credible information and insight in our rapidly changing world.

MPO Post launches as new US non-profit polling source

The US media and digital town square is chock full of published opinion research – every day produces a handful of new polls revealing Americans’ opinions on the issues of the day, mostly around politics but also public policy and social issues (e.g., legalization of marijuana, immigration reform, consumer confidence).

Within this context comes a new D.C. based nonprofit research organization - Media and Public Opinion (MPO) Research Group - offering a brand new stream of ongoing surveys focusing on political, social and economic affairs in the USA. MPO Post was launched for about a year, and is now starting to publicize their research. The organization was founded by former market research entrepreneur Babak Bahador, who now teaches in New Zealand (and is currently a Visiting Scholar at George Washington University).

The MPO Post vision is to contribute valuable information to US policy debates (“from a novel perspective”), and also provide a research service for academic/policy researchers. In addition to the news website, they operate a research operation that runs their data collection engine – a monthly telephone survey based on a telephone panel of 10,000 US residents. Their website indicates they are non-profit, independent and non-partisan, although no information is provided about sponsors or funding.

On their website you’ll find brief reports on about a dozen of their polls, covering a range of topics and in some cases monthly tracking over the past 12 months. Each poll is based on modest national samples (N= 500 – 750), so are not sufficient to provide for regional, demographic or party breakdowns.  The choice of using telephone is interesting given current trends in research methods, and it is not clear how they are addressing cell phone coverage in managing their sample frames.

MPO Post is entering a crowded field of published US public opinion polling, but as a new independent organization with an ambitious vision they are definitely worth following.

Israelis and Palestinians both want peace, but can agree on little else

The conflict between Israelis and the Palestinians has been intractable, with no meaningful hope for resolution in sight.  But this is not because the people on both sides of the conflict do not want peace. Gallup International recently reported on a notable survey it conducted in the region, which provides a cogent example of how public opinion research can reveal important insights about conflicts that do not often conform to political and media discourse.

This study – consisting of in-person interviews with representative samples of 1,000 Israelis and 1,000 Palestinians conducted in August-September 2012 – reveals the conundrum facing leaders on both sides of this conflict. Clear majorities of both peoples want peace and support a peace process with the other side (this sentiment is especially strong among non-Jewish Israelis, who are mostly of Palestinian origin).  And when it comes to the preferred means of achieving “self-determination and security”, majorities on both sides favour non-violent forms of resistance and negotiation over armed struggle and military solutions.

This common ground on means and ends is by no means enough, and the two sides are divided in important ways:

First, there is widespread distrust of the other side’s leaders.  No more than a handful of Israelis (1 to 2%) have a favourable view of the current Palestinian leaders, while Palestinians are even more distrustful of Israeli Prime Minister Benjamin Netanyahu (2% favourable, versus 80 to 83% very unfavourable). 

There is also division on views toward US President Barack Obama as a broker for peace: he is trusted by close to half (45%) of Jewish Israelis, but only 8% of Palestinians, which significantly limits his influence with those in the West Bank and Gaza Strip.

Second, Palestinians appear to be more motivated to achieve peace with Israel than the other way around, in terms of how they see it affecting their own personal safety and security, economic prosperity and gaining internationals respect.

Finally, there are differences of opinion about a “two-state” solution to the conflict. This approach is heavily favoured among West Bank Palestinians (70%) and non-Jewish Israelis (85%), but by only half of Jewish Israelis (52%) and Gaza Strip (48%) Palestinians.  

Given these divides, it is no surprise that most Israelis (64%) and Palestinians (69%) do not believe a permanent peace between their peoples will ever be achieved.

This research does not reveal any clear path to resolving this longstanding conflict, but it does provide an essential window into the current opinions and priorities of the people, unfiltered by the heated rhetoric that dominates the public narrative.

Polling for the 21st century

The Internet and social media are rapidly transforming our world, and most certainly changing the rules of the game for public opinion and market research. The latest marker for how this trend is unfolding comes with the recent introduction of a new online product called Zipinion, described as a “crowdsourced polling application.”

What Zipinion offers is similar to what you can now get from Google Consumer Surveys, but perhaps aimed at a different segment of the market. For $20 they will provide 100 “objective” opinions on just about any topic you choose, dawn from a pool described as over 500,000 people, and will provide this within minutes.

This type of product is not intended as an alternative to traditional market research methods where accurate population parameters and in-depth analysis is required.  Rather, this is marketed as a “low cost, why not” way of testing concepts and ads in a way that is step up from asking your friends and colleagues. Their website offers such examples as logo comparisons, political mailers, real estate photos, and social media updates (e.g., testing a draft tweet before posting). 

Does this type of information mining still fit the definition of market research?  Clearly not if one uses the traditional criteria based on the scientific model which emphasizes validity, reliability and representativeness. But in the accelerating world of business and entrepreneurship that increasingly relies on quick online connectivity, getting instantaneous input from a random group of online enthusiasts is much easier and more fun than rigorous research methods.

Will this new style of research provide helpful guidance for design and business decisions, or might the random 100 people sometimes steer in the wrong direction?  There is really no way to know, so using this type of product represents an act of faith in the wisdom of small crowds.  Over time, the usefulness of this approach will become apparent.

US media standards for reporting on public opinion polls

These days the media is awash with stories featuring public opinion polls, covering topics serious and otherwise. Polling has proven to be a relatively inexpensive source for journalists to fill papers and websites.

The quality of this reporting on polls is wildly uneven. There are no common standards for publication, and most news organizations lack the internal expertise and journalistic discipline to apply established research standards in their assessment and publication of incoming polls.  Most stories that cover polls contain so little documentation of how the research was conducted that it is next to impossible for even the most informed reader or viewer to know whether it is an accurate reflection of public opinion.

There are, however, two major US news organizations (ABC News, the New York Times) that take this type of reporting seriously.  Both have published their own internal standards to guide their reporting on polls, and are worth knowing.  

ABC News publishes detailed standards that were developed by polling veteran Gary Langer (previously on staff at the network, and now acting as an external adviser). The standards include rigorous requirements for disclosure that guide decisions on whether or not to report on polls not conducted by the network. These requirements include identification of the research sponsor and field provider, a detailed statement of methodology, and the full questionnaire and marginal results for all questions. In most cases ABC News will only report on surveys based on probability samples, which rules out most Internet-based surveys (those that rely on recruited panels). There are also detailed standards for guiding the methodology of polls conducted by ABC News (e.g., sampling, interviewing, weighting the data, response rates, pre-election polling).

The New York Times takes a somewhat different approach to its polling standards, which were developed by a committee of editors and reporters. The Times starts with the premise that “reporting on polls is no different from reporting on any other information we give readers.” The focus is on overall quality of analysis and reporting, and reliance on the expertise of the News Surveys department to ensure the reporting meets accepted research standards with respect to disclosure, sampling, methodology, reliability and unbiased content. These standards also specify that disclosure includes release of the full questionnaire and marginal results, as well as sponsor and field provider, although it stops short of making these absolute requirements for publishing.  As well, there is no firm rule about probability samples, but it is stated that “In order to be worthy of publication in the Times, a survey must be representative, that is, based on a random sample of respondents.”

These news organizations stand out as leaders in responsible reporting on public opinion research, and provide useful models for how this can and should be done. Having the necessary expertise to properly evaluate incoming polls is essential to making this work, but as the ABC News model demonstrates this can be accomplished without having this expertise be located within the organization.

US to drop use of term “Negro” for census surveys

Race has always been an important but controversial identifier in the USA, and so it is worth noting  a recent Associated Press story reporting that the US Census Bureau will stop using the term “Negro” in future surveys. This term, used on census forms since the year 1900, will be replaced by two other terms, “black” and “African American.”

To some it may seem like this change is long overdue, since the word “Negro” now seems outdated and offensive to many (best reflected in the controversy over the title of Canadian Lawrence Hill’s bestselling “The Book of Negroes”, which inspired a public book burning in Amsterdam, and had to be published with a different title by its US publisher).

Why would the US government still be using this terminology in 2013?  In fact the Census Bureau has for a number of years been looking to replace the term “Negro” on its surveys, but found there continues to be some Black Americans who still identify with the term (in 2000 about 50,000 Americans specifically wrote in “Negro” to describe how they wished to be identified).  Keeping the term on census forms is good research practice (offering response choices used by the target population), but fewer relate to the term on each successive survey, and opposition to its use has been growing.  The change in census survey terminology will take effect in 2014.

In Canada, the census has long used the single term “black”, along with “white.”  All other terms are based on nationalities (e.g., Filipino, Chinese), except in the case of “Arab.”

The American Dream – 2012 Edition

Americans as a people seem to rarely agree on anything, and the current political climate is as polarized as anyone can remember.  But one thing on which there is a national consensus is the idea of the “American Dream.”  What exactly it means is difficult to pin down, and undoubtedly has evolved over time. This makes it a fitting topic to explore with Americans, which is what the non-profit group Public Agenda did last year

Public Agenda is a unique non-profit established in the 1970s by social researcher and author Daniel Yankelovich and former Secretary of State Cyrus Vance to promote problem-solving through non-partisan research and public  engagement.

In mid-2012, Public Agenda conducted an online survey with 2,041 Americans, supplemented by focus groups across the country. Results were released in October, and provide an interesting (if less than complete) portrait of how Americans define the American Dream in 2012. No one concept seems to dominate, but notable themes include the following:

• The words most commonly used to describe the Dream are “freedom” (24%) and “home ownership” (16%);

• Americans are more likely to define the American Dream as the opportunity for people from modest beginning to achieve a middle class life than a life of great wealth and fame; but many see the Dream as encompassing both ideas;

• One in four (24%) Americans claim to be now living the Dream.  Of the remainder, 40% say they are making progress toward this goal, while 15% have given up, and another fifth (21%) say it is not relevant to their lives;

• The single most important factor in achieving the American Dream is a strong work ethic;  Most believe a 15-year old with a strong work ethic but abusive parents is more likely to succeed than one with strong family support but weak ambition; and

• At the same time, there is recognition that external supports also matter. Americans are divided on whether the biggest threat to the American Dream is a decline in work ethic (43%) or people being shut out  because of economic trends (43%).

Unfortunately, Public Agenda has chosen to publish only top line findings (presented mostly through cute infographics), precluding any deeper analysis and conclusions.

How Britons consider their nation and where it's heading in 2013

So much of how Canadians define themselves is in relation to the USA that it is a rare treat to get a solid glimpse into the psyche of another country with which we have strong historic and ongoing ties. The think tank British Futures has provided just that in its just-released Second Annual State of the Nation report.

This report presents the results of a November 2012 national online survey with 2,515 residents of Great Britain (ages 16 to 75). The survey focuses on how Britons look forward to 2013, in the context of the 12 months just ended. The study is similar to the Environics Institute’s own ongoing Focus Canada research program, thereby providing a valuable basis for comparing the perspectives of these two Commonwealth countries.

And in fact there is much that Canadians and Brits have in common.  Like in Canada:

- Brits take great pride in their public health system:  It is at the top of the list of things that make them proud to be British, and the anniversary they most want to celebrate in 2013 (when their NHS turns 65);

- Britain has a large and growing immigrant population (11.3%, as of 2009-10), and like in Canada it is immigrants who are the most proud of being British.

- Native and foreign-born generally agree on what makes for a good citizen – speaking the language, obeying the law, gender equality and respecting faiths and ethnic backgrounds.

- There are regional tensions but generally peaceful ones.  In the event of Scottish independence, most Scots say they will still feel British because of geography, history and culture, and most others will never think of Scotland as being foreign.

- There is a growing economic and cultural divide between urban and rural;  the polychromatic, multicultural, global London, and regions like “ the white, stagnant and left-behind north-east.”

In both countries, the economy is the number one national issue of concern, but concerns are considerably greater in Britain – only 19 percent feel optimistic about the country’s economic prospects for the coming year, compared with 50 percent who are pessimistic (though this is a big improvement from the 9% versus 74% split from 12 months ago). The report finds that as the new year begins, “Britons are less gloomy about everything from the family to Britain itself.” 

Overall, only one-third (32%) of Brits believe their country is heading in the right direction, compared with 54 percent of Canadians.  And they are considerably less positive about the future of Europe.

British Futures is a one-year old independent, non-partisan thinktank focusing on addressing the the country’s challenges of identity and integration, migration and opportunity. They publish a series of reports online that are worth checking out.

What the Canadian government is spending on public opinion research

The Government of Canada has just released its annual report of what it contracts for public opinion research for the fiscal year 2011-12 (which ended almost 11 months ago).  Each year, the government publishes such a report on its overall spending, as well as the number and value of contracts by department and research supplier.  In some years (but not this one), the report highlights a few of the more noteworthy projects.

The government defines “public opinion research” as the collection of “opinions, attitudes, perceptions, judgements, feelings, ideas and reactions or views intended to be used for any government purpose, and might encompass any group of Canadians (e.g., employees, business representatives, users of government services, populations at risk), and utilize quantitative and/or qualitative methods. 

Given this broad scope and the many areas of government responsibility and services, one might expect the federal government to have a comprehensive research agenda to ensure it is listening to Canadians, informing them about important programs and issues, and ensuring its services are meeting customer needs. But the report reveals a very lean effort over the 2011-12 year, comprising just 100 projects across the entire government for a total value of $6,513,824 (or about 19 cents per Canadian).  But most important, this dollar figure reflects a steadily declining investment in public opinion research, down 18 percent from the previous year, and down by 79 percent since 2006-07 (the first full year of this government’s mandate) when the government report for that year identified 446 projects totalling  over $31 million.

Just how much should a national government in a country like Canada be spending on public opinion research with its citizens?  There may be no one right answer, but it is instructive to look at other comparable countries. The most recent figures for the US government range from $2 to $6 billion USD (or $12.88 per capita, taking the midpoint of the range), and for Australia the total is $31.4 million CDN ($1.54 per capita). These countries  (and others) are maintaining a substantial investment in canvassing their citizens despite tough budget constraints.  Why the Canadian government is pursuing a different path is a question worth asking.