Research Industry wrestles Margin of Error monkey
Almost any time you read about a public opinion poll you will see a sentence, usually at the end, stating a “margin of error” percentage, “plus or minus.” Those who know something about research will understand this to be a statistical measure of the representativeness of the sample of survey respondents recruited from the broader population under study. Those who know how survey research is done are likely aware that there is growing controversy about the use of this statistic as an indicator of survey quality.
The issues boils down to the following: The margin of error statistic applies to probability samples, such as those historically used for telephone surveys where every household is theoretically available to be sampled for a given survey. Such samples are increasingly difficult (and costly) to generate , and most surveys today are conducted through online methods that rely on non-probability samples. And yet the research industry and its clients continue to rely heavily on margin of error as the sine qua non indicator of survey accuracy.
The issue is well known within the research community but not discussed, until now. Annie Petit of Peanut Labs recently hosted a well attended webinar on this topic. The webinar featured four senior level researchers from the market research industry who discussed the relevance and applicability of margin of error in today’s world. The discussion is a bit dry, and may be difficult to follow for those who lack a basic understanding of margin of error and how it works since the webinar is aimed at industry practitioners. But the discussion is well worth listening to for those involved in survey research as a practitioner, client, journalist, or anyone who wants to understand better about how surveys are done.
There is no surprise in the fact that the four panellists all agreed that margin of error statistics are no longer as relevant in survey research today, and that they may do more harm than good in providing irrelevant and possibly misleading information about survey data quality. So why does the practice persist? In part this is because there are no other metrics of survey quality that offer the conciseness and face validity of margin of error. There are numerous sources of error that can affect the accuracy of survey results, and these are complicated if not impossible to measure.
The most revealing insight to come out of the webinar is how research companies are stuck with an irrelevant metric of survey quality because their clients demand it. Several panellists noted how they write survey reports that include a margin of error and then state it does not actually apply to the results (as a way to appease clients who insist the statistic is included). One panellist commented that to stop quoting margin of error even when it does not apply could well risk to the loss of valued clients. What this reveals is an underlying conflict between the science and business aspects of market research in today’s world. Commercial and media clients need data to drive or justify decisions, and they need to show their data is sound. Margin of error has been cast in the role of providing that seal of approval, and the inconvenient truth behind the science is easily ignored.
Not all survey research is conducted for business, and it would be illuminating to also hear the perspective on margin of error from practitioners in government, university and non-profit settings who are focused more on sound data than business confidence. Perhaps this will be the topic of a future webinar.
You can listen to the Peanut Labs webinar in its entirety here.
When survey research goes to war
Public opinion surveys are used for many purposes, and some have much less profile than others. A good example is how survey research is now being used by governments and their militaries as a counter insurgency tool in conflict areas. This research flies largely under the media radar, but is nicely discussed in a recent Monkey Cage blog post in the Washington Post by Andrew Shaver and Yang-Yang Zhou (both are Ph.D. political science candidates at Princeton University).
Shaver and Zhou discuss major research projects undertaken by US-led coalition forces in Afghanistan and Iraq to measure local population opinions and sentiment in support of military operations. These efforts are substantial in scope – in the case of Iraq entailing in-person interviews every month over a five year period, cumulatively totalling around 200,000 interviews. Topics included level of support for insurgent attacks against the coalition and Iraqi government forces, satisfaction with a range of public goods and services, and expectations about the capabilities of Iraqi security forces. Why this is relevant is because successful counterinsurgency initiatives are unlikely to succeed without local public support.
The scope of such ongoing investment would suggest that the research is proving valuable in helping to anticipate challenges facing military operations as well as measure progress in achieving public support. Shaver and Zhou report the data collected in Iraq revealed a clear positive relationship between public support for insurgent attacks against coalition forces and the actual number of such attacks. But they point out that such data do not tell us whether one leads to the other (does growing public sentiment lead to more attacks, or do such attacks result in more popular support?).
As well, the authors raise what they aptly describe as “the more fundamental and less exciting question of whether the survey responses accurately reflect the attitudes of the citizens they are designed to capture.” Do Iraqis and Afghanis tell the truth when being interviewed on surveys being conducted on behalf of an occupying army? There is no way to measure this precisely but it would have to be a concern to those sponsoring such research. Shaver and Zhou briefly outline some of the methodological approaches that have been developed to obtain accurate answers to sensitive survey questions. But these approaches were developed using western populations accustomed to survey participation, and their effectiveness with other cultures and contexts remain to be established.
Apropos this issue, the New England Chapter of the American Association of Public Opinion Research (AAPOR) is hosting a half day mini-conference entitled “New Frontiers in Preventing, Detecting, and Remediating Fabrication in Survey Research.” This free event will be held in Cambridge, MA on February 13, 2015 (and also broadcast over the web via WebEx). The event is likely to cover the issues facing surveys conducted by governments overseas, as the agenda will include speakers from the US State Department and the Arab Barometer.
What’s wrong with online survey research methods?
Perhaps the most significant trend in market and public opinion research in the past decade has been the emergence of online research methods as the dominant form of survey data collection. This trend has taken for three reasons: a) to leverage the expanding array of digital technologies and their rapid adoption across the population; b) to realize greater efficiencies and lower costs in collecting survey data; and c) to avoid having to deal with the challenges associated with telephone interviewing.
But what about the quality of online research? Is something critical being lost by this expanding reliance on online research methods? This question is addressed in the latest issue of MRA’s Alert Magazine, with a critique of online research methods by author Neil Chakraborty. He addresses a number of issues, but zeroes in the reliance on non-probability samples, which is widely considered to be the greatest limitation of online survey research.
Survey research blossomed in the latter half of the 20th century largely on the strength of the science of probability sampling that provided a statistically-credible basis for extrapolating results from small representative samples to the populations they are drawn from. This approach requires that every member of the population has a chance of being selected to be surveyed. This could be more or less accomplished when surveys were conducted by telephone, but cannot be done with Internet-based surveys because there is no online equivalent to telephone numbers (unlike telephone numbers, e-mail addresses cannot be randomly generated). This means that most online surveys rely on drawing samples from established panels of individuals who are recruited to participate through website promotions. However balanced such online samples might be, in terms of their demographic and regional characteristics, they do not possess the qualities of probability samples, and cannot be treated as such.
Chakraborty’s critique is not new, and his points have been covered by others (see for instance AAPOR’s 2010 report on online panels and 2013 report on non-probability samples). But it offers a useful overview of key issues, and includes an important admonishment to research practitioners to both focus on reducing survey errors, and be transparent about the limitations of their methods.
Practical advice about survey research goes video
When organizations, students and newcomers to the survey research business look for “how to” guidance, the choices are largely limited to textbooks, consultants and on-the-job experience. Now there is a new resource in the form of short practical advice videos that have been created by Elon University in Greensboro North Carolina (home of the Elon University Poll).
- Surveys in society
- What is sampling error?
- Methods of collecting survey data
- How to read a crosstab
- 7 tips for good survey questions
The material is basic, and the audience is primarily for beginners. But even seasoned professionals can benefit from refreshing their knowledge.
2013 Survey of American Jews -- Lessons for Canada
The survey addresses many themes, but chief among them is how Jewish identity and practice are changing among Americans, and the results may also be relevant to what is happening in the Canadian Jewish community. This was the focus of the Seventh Annual Elka Klein Memorial Lecture held at Congregation Darchei Noam in Toronto on June 23, 2014.
The event featured a presentation on the American survey results by Greg Smith, one of the senior Pew researchers responsible for the study. This was followed by a panel of Canadian experts who discussed how the American findings may or may not be relevant to the Canadian Jewish community (in the absence of any comparable survey of Canadian Jews).
The panel was moderated by Janice Stein (Director, Munk School of Global Affairs), and will include our member Frank Bialystok (University of Toronto), Bernie Farber (former CEO of the Canadian Jewish Congress) and Aaron Levy (Founder and Executive Director of Makom: Creative Downtown Judaism).
AAPOR launches new initiative on survey methods
The American Association of Public Opinion Research (AAPOR) righly bills itself as the leading association of public opinion and survey research professionals. In keeping with this role, AAPOR has just announced the launch of its latest special task force initiatives, in this case to encourage dialogue and deepen our understanding of today's survey methods.
The current research environment is characterized by an expanding array of methodologies, each with its own challenges — increased cost, under-coverage, low participation, mixing of modes, uncertainty in the links between theory and practical application. This trend is producing numerous reactions, from those who are developing and testing new techniques and innovative methods to address many of these issues, to those who have grown skeptical of contemporary survey methods, whether new approaches or traditional methods.
The Task Force will be composed of leading research methodologists, and will have three principal responsibilities:
- Organizing a "mini-conference" at the 2015 AAPOR Annual Conference (May 2015).
- Develop guidance to help consumer of survey data assess the quality of what they commission.
- Identify other ways to feature innovation and learning in this critical area of research methodology.
This type of independent, expert-driven initiative is critically needed at this point in time, and is well worth paying close attention to as it evolves over the next year or so. Watch for updates in this space.
Gallup Tracks Americans: Concerned Believers and Cool Skeptics on Climate Change
By Kristen Pue
Gallup recently concluded a series of public opinion reports on climate change, culminating in a final report that established three categories to articulate American public opinion about climate change. The report showed that age and political affiliation, but not level of education, are important in explaining public opinion on the subject. It also included trend data that can be used to compare the story of climate change skepticism in Canada and the US.
WHAT GALLUP FOUND
Gallup has released ten reports since March as part of its ongoing climate change series. Each of the reports addressed one aspect of American public opinion about climate change. The reports concluded that:
· Americans value the environment and a majority believes climate change is man-made. Fifty-seven percent of Americans believe that human activity is responsible for climate change, a proportion that has remained steady since 2001. Likewise, Americans prioritize environmental protection over economic growth and favor energy conservation over energy production to solve the nation’s energy problems.
· However, climate change is not a top of mind issue for most. Americans express a low level of concern about climate change, particularly when compared with other environmental concerns such as polluted drinking water, soil contamination, and air pollution. A majority of Americans believe the seriousness of global warming has been exaggerated.
· Terminology doesn’t matter. The term climate change has increasingly been used instead of global warming, in part because it is believed that climate change will be a more persuasive term. But Gallup found that there is no substantial difference in how the American public responds to the terms global warming and climate change.
Each of the first nine reports, summarized above, highlighted individual aspects of climate change public opinion. Gallup’s final climate change report attempted to pull these different snapshots into a single, coherent narrative. In particular, it determined that American public opinion about climate change is polarized, with two prominent clusters and a “mixed middle.”
CONCERNED BELIEVERS, MIXED MIDDLE, COOL SKEPTICS
Gallup organized Americans into three categories, according to their views on climate change.
· “Concerned Believers” (39%) believe that climate change is attributable to human activity, are concerned about it, and believe its seriousness has either been underestimated or given a correct level of attention in the news.
· “Cool Skeptics” (25%) do not believe that climate change is man-made, worry little about it, and believe news coverage has exaggerated the seriousness of the threat.
· The “Mixed Middle” (36%) constitutes those people that do not fit into the other two categories. They hold a range of views on the three questions.
Gallup then used this segmentation to offer some conclusions about demography and perceptions about climate change. They found that age and political affiliation are the key dividing factors on climate change opinion. 65% of Cool Skeptics identified as conservative, compared with only 18% of Concerned Believers. The majority of Concerned Believers are under fifty years old, whereas the majority of Cool Skeptics are older than fifty years old. It is worth noting that there is a relationship between age and political affiliation in the US (the average Democratic Party voter is younger than the average Republican Party voter). Interestingly, Gallup also found that climate change opinions in the US are not divided according to level of education.
TREND DATA: CONCERNED BELIEF REMAINS STEADY, COOL SKEPTICISM GROWS
Gallup also issued trend data for its three clusters, demonstrating the general trajectory of American opinions about climate change. The proportion of Concerned Believers has remained largely the same throughout this period. The Mixed Middle is shrinking, while the Cool Skeptics category has grown.
CLIMATE CHANGE OPINION TRENDS: COMPARING CANADA AND THE US
This section compares Gallup’s trend data on this issue to similar data for Canada. Although the Environics Institute’s Focus Canada 2013 Climate Change Report is based on answers to different questions, it is possible to qualitatively compare the two reports. Both attempt to measure and track climate change skepticism over time.
The Focus Canada data asks to what extent Canadians believe science has conclusively proven that climate change is caused by human activity. Notably, a clear majority of Canadians believe that climate change is man-made. Climate skepticism has risen and subsequently fallen, returning to just above 2007 levels.
The two surveys asked different questions, so direct comparison isn’t possible. But it is possible to draw some conclusions from analysing the trajectory of public opinion in each. In the US, climate change skepticism has been rising since since at least 2006, but in Canada views on climate change science have reverted, and are roughly the same today as they were in 2007.
Further, it is worth highlighting American and Canadian climate change public opinion in the context of major events and public discussions on the subject.
Climate Skepticism Began to Grow in 2006/2007
Uncertainty about whether climate change is attributable mostly to human activity began to grow in 2006 in the US and in 2007 (or earlier, as Focus Canada trend data on this issue only captures the period of 2007-2013) in Canada. There is no obvious reason for this to have occurred, but it may reflect the general intensification of public debate about climate change.
2006 would seem to have been an excellent year for climate change activists. An Inconvenient Truth was released and the Stern Report controversially argued the costs of adapting to climate change will be greater than the costs of prevention. That same year, the Environmental Protection Agency (EPA) in the US was sued for its failure to regulate greenhouse gas emissions. Massachusetts v. EPA was decided by the Supreme Court in 2007. The Court ruled in a 5-4 decision that the EPA has the power to regulate greenhouse gas emissions from automobiles and that it could not sidestep its authority unless there was a scientific basis for its refusal. It is surprising -- given these developments -- that American climate change skepticism grew during this period.
The 2007 Intergovernmental Panel on Climate Change Report is seen by some as a watershed moment for climate change science: it declared with 90% certainty that human activities cause climate change. However, climate change skeptics had some notable moments in 2007: questions were raised surrounding factual inaccuracies in An Inconvenient Truth and The Great Global Warming Swindle, a divisive climate denialist UK documentary, was released. Both Canadians and Americans became less certain during 2007 that climate change is caused by human activity, suggesting that rebuttals to An Inconvenient Truth and other climate change denialist publications were successful in calling into question the conclusive nature of climate change science.
“Climategate” Deepened Skepticism
In both data sets, climate change skepticism accelerated between 2009 and 2010, likely in response to the “climategate” scandal in which over 1000 emails between climate change scientists were stolen. Although this scandal did not actually unveil evidence that scientists had been manipulating climate change data, they did show that the scientists were very reluctant to share their data with people they saw as only wanting to make trouble. It is likely that this scandal impacted public opinion in both Canada and the US. In each case, the percentage of climate change believers -- in the US, “Concerned Believers” and in Canada those believing the science is conclusive that climate change is caused mostly by human activity -- reached its nadir in early 2010.
The Copenhagen Accord, which was signed in late 2009, likely contributed to the high rates of climate skepticism in 2010. The meeting in Copenhagen failed to deliver a global agreement on climate change, and was seen as a spectacular disappointment. This failure halted momentum on climate change action, and was followed with negative media coverage.
Belief in Climate Change is Slowly Recovering
In Canada and the US, the view that climate change is attributable to human activity is regaining popularity, but has not returned to 2007 levels.
The percentage of Concerned Believers in the US began to grow again in 2012. This may be because the sense of urgency expressed in the IEA report was made tangible for many in North America. Summer ice cover over the Arctic Ocean reached a record low. Hurricane Sandy catalysed discussion on climate change; most Americans now link climate change and extreme weather.
2013 proceeded with a leaked draft of the UN Intergovernmental Panel on Climate Change that made headlines, particularly for its findings on the implications of climate change for food security. This draft was timely, given the protracted heat wave across the Southern US in 2012, which prompted discussion about climate change and concerns about current and future wheat yields. The Intergovernmental Panel also declared 95-100% confidence that human activity is the primary influence on planetary warming.
Increasingly since 2010, Canadians and Americans believe that climate change is man-made. This may be due to a combination of increased scientific data about climate change – and efforts to articulate these findings in publicly salient ways – and extreme weather occurring in 2012.
For a comprehensive timeline on climate change, click here.
For more on climate change and American public opinion, check out the Yale School of Forestry and Environmental Studies’ Project on Climate Change Communication. This project, which has been ongoing for nearly a decade, conducts social and public opinion research on climate change; designs and tests strategies to engage the public on the issue of climate change; and empowers opinion leaders by giving them tools to more effectively engage audiences on climate change. As part of this mandate, they recently released the following report: Public Support for Climate and Energy Policies in November 2013.
The World Bank surveys the globe on open government
Sometimes important public opinion research does not receive the attention it deserves. The Global Opening Government Survey – a study undertaken through collaboration between the World Bank’s Open Government Practice and RIWI Corporation, a Toronto-based company – is one such survey.
Global in scope, the Global Opening Government Survey spanned 63 countries on every continent. It asked the fundamental question: Do citizens perceive their governments to be open, as well as why, and how they want government to be more open.
WHY DOES TIHS RESEARCH MATTER?
It’s an important topic. Government openness is an important democratic requirement; it enables citizens to make informed choices, to question policies and hold politicians to account. In this sense, the extent to which citizens perceive government as open may be a litmus test for a healthy, democratic environment.
This topic is also timely, given controversy in various countries surrounding how much citizens know about what their governments are doing – from news that the NSA’s PRISM surveillance system is being used on American citizens, to reports that CSEC (Communications Security Establishment Canada) has been using free airport wifi to keep watch on Canadians.
The survey method is innovative. The Global Opening Government Survey uses an innovative new technology for conducting online surveys with citizens on a global scale without relying on recruited opt-in panels. The method was developed from RIWI’s random domain intercept technology (RDIT), which connects with Internet users across devices and platforms who mistype a URL or type a URL for a website that no longer exists. On the error page, users are asked to answer a short set of questions.
It’s a hidden gem. The Global Opening Government Survey is worthy of attention because it hasn’t received any. If you haven’t heard about this study you are not alone; the data was presented at the World Bank’s Open Government Partnership Summit in London last year and then posted on a website, but has received very little media coverage.
One reason this study may have attracted little attention is that its publication was not accompanied by a comparative analysis of the findings that presented a global picture or cross-country comparisons. So the key insights and value of this research take some effort to identify.
WHAT DOES THE RESEARCH TELL US?
Perceptions of government openness do not correspond to measures of democratic quality. One might expect that people in well-established democracies would be the most likely to say their governments are open, since democratic systems rely on transparency and open debate. But this does not prove to be the case. The five countries whose citizens most believe their governments are open can all be found on the African continent: Tanzania, Ghana, Malawi, Liberia, and Kenya.
Developed democracies – countries that score well on democracy indexes like the Global Democracy Ranking – varied considerably in terms of public perception of government openness.Norway, Finland, Canada, and the Netherlands were near the top of the list in terms of citizens rating their government as the most open, while publics in the United States, Spain, and Italy were least apt to share this view. This discrepancy between actual level of democracy and perceived level of government openness suggests that populations across countries have different ideas and expectations about what it means to have an “open” government.
Citizens in developed democracies are less demanding for more government openness. More often than not citizens in developed democracies are less likely than others to express the desire for their government to become more open and to provide more information about what it does. They are also less likely to say they would be more likely to trust government or pay taxes if their government was more open. The results suggest that citizens in developed democracies are, comparatively, satisfied with current levels of government openness.
A few other notable findings:
Mongolia stands out. Of all countries surveyed, Mongolians have the least optimistic assessment of how open their government is, and the highest level of demand for more government. This is noteworthy given that Mongolia has a healthy democratic system with open information, a lively multi-party system with elections that are generally free and fair and an active civil society sector.
Only 25% of Latvians – less than any other country by far – think citizens should have a say in government spending. Denmark (42%) had the second lowest proportion of its public saying citizens should have a say in government spending. By comparison, this expectation is expressed by at least eight in ten citizens in such countries as Mongolia, Brazil, and Kenya.
HOW DO CANADIANS VIEW THE OPENNESS OF GOVERNMENT?
Canadians are generally positive about the openness of their governments. Of the 63 countries surveyed, Canada is 13th highest proportion of its citizens rating their government to be open; third among developed democracies.
Canadians also expressed a greater than average demand for more government openness, amongst developed democracies. Two-thirds (64%) of Canadians think citizens should have more say in government spending and contracting, which is a high proportion compared with publics in other developed democracies. Canadians are also more likely than citizens in most other developed democracies to say they would be more likely to pay their taxes if government were more open.
While these numbers are strong compared with other developed democracies, they are less than compelling in absolute terms. Just over half of Canadians would like government to be more open (53%), and to have more information about government (53%). A sizeable minority were unable to even offer opinions on these questions.
The Global Opening Government Survey provides valuable information about an important topic, and was conducted using an innovative new method. The raw data, publicly available online, is potentially useful for policy-makers, activists, and journalists. It is worthy of further attention and comprehensive analysis.
Majorities in 22 of 40 Countries Say Believing in God is Essential to Morality
To be a moral person and to have good values is it necessary to also believe in God? Many people from around the world answered “yes” in a recent Pew Research Center report; a clear majority in 22 out of 40 countries believe that morality emanates from one’s religious views.
In some countries, particularly in Africa and the Middle East, the view that it is necessary to believe in God to be moral and have good values was nearly universally held – for example, in Ghana there is 99% support for this perspective. In other countries, however, most people disagreed: in France only 15% believe morality is contingent upon belief in God.
The Asia-Pacific exemplifies this diversity of opinion: perspectives range from Indonesia and Pakistan on one end (where 99% of people believe it is necessary to believe in God to be moral) to China on the other (with only 14% support for this view).
Beyond the range of global perspectives on religion and morality, there are two additional findings worth highlighting.
Views about morality and God are closely linked to wealth.
People in richer countries are more likely to believe that morality does not require belief in God than people in poorer countries, a correlation shown below in this chart from the Pew Research Center’s report. For example, a majority of people in every European country surveyed hold that it is not necessary to believe in God to be moral.
Although people in wealthier countries are more likely to believe that morality exists independent of one’s belief in God, the USA and China are notable outliers. Just over one in ten people in China – a country of average wealth amongst those surveyed – hold that belief in God is necessary for morality. And, unlike all other western liberal democracies surveyed, a majority of Americans believe that theism is required for morality.
Viewpoints on this question remained remarkably stable.
Pew Research asked this same question in previous years (2002, 2007, and 2011), and opinions have changed very little in most cases. For example, Mexico’s responses have shifted by two percent between 2002 and 2013. Ghana is a notable exception: the percentage of people that believe it is necessary to believe in God to be a moral person was 73% in 2007 but skyrocketed to 99% in 2013.
This stability in opinions about whether belief in God is necessary to one’s moral character is notable given the substantial social changes that have taken place over the past decade. Austerity has done little to change the perspective of most Europeans on this question. The United States has adopted a more secular outlook on this question over the decade, but the change is slight – just 5%. And despite seismic societal shifts emanating from the Arab Awakening, the Middle Eastern countries included in this study retain the same aggregate views about morality and belief in God.
Where Canada Fits
Canada fits the trend that wealthier countries take a secular perspective on this question: just 31% of Canadians think it is necessary to believe in God to be moral, a view that has not changed since 2002. Of all countries surveyed, Canada is most similar to Germany. However, the percentage of people who believe that morality is inseparable from religion is smaller in most European countries than in Canada – among them France (15%) as well as countries like Italy (27%) that have large religious populations.
Younger and college educated Canadians are least likely to agree that morality is contingent upon belief in God.
How does this report relate to what we know about religion in Canada?
According to the Environics Institute’s Focus Canada data from 2011, eight in ten Canadians believe in God or a universal spirit. Likewise, most Canadians express a religious affiliation (69%), while just over a quarter identify as atheist, agnostic, or without a religion. So, most Canadians believe in God; at the same time, most believe that morality is not contingent upon believe in God. What explains this difference?
One possible explanation is that Canadians distinguish religious questions from certain other values. Lending some evidence towards this view, Canadians were asked in 2011: if they have a religious affiliation, do they share common values with those that don’t; if they don’t have a religious affiliation, do they share common values with those that do? A majority in both groups agreed that they share common values on most things.
Another possibility is that many Canadians may separate theism from morality because religion plays only a small role in their lives. For example, half of Canadians reporting a religious affiliation rarely if ever attend religious services.
The results of this Pew Research report reflect the complex relationship between religion and morality, a question that has become increasingly poignant as more people in rich, western countries identify as atheist or agnostic, while other parts of the globe are becoming more religious.
Asia Foundation Publishes Survey of Afghans as International Withdrawal Begins
In December 2013, the Asia Foundation released its ninth annual public opinion Survey of Afghans, which polled over 9000 Afghan men and women from all 34 provinces of the country. The Asia Foundation works throughout Asia to improve lives through projects supporting governance and law, economic development, women's empowerment, environmental stewardship, and regional cooperation.
Unlike a decade ago, Afghanistan is no longer front page news. But it might be soon, as the country is approaching a critical juncture at which it will either thrive or falter.
About Afghanistan: Why Survey Afghans?
Canada has been a part of the multilateral intervention in Afghanistan (the International Security Assistance Force) since 2001. Over 50 states have contributed to the mission, which is engaged in training the Afghan National Security Forces, providing civilian protection and conducting counter-terror operations. Afghan forces have now taken over leadership in securing the country, and the intervention is set to conclude at the end of 2014. Afghan government and military capacity has been steadily increasing, and sustained economic growth is encouraging. Still, considerable uncertainty lingers about what will happen to the country when international forces leave (although a scaled-back NATO train, advise, and assist mission will remain) -- particularly given difficulties with neighboring Pakistan, lack of progress in national reconciliation, and a recent suicide bombing incident. Noting this, and in light of upcoming elections in spring 2014, it is critically important to ask Afghans what they think about their country, the direction in which it is headed, and prospects for the upcoming year.
Surveying Afghans: It’s Possible
As you might expect, there are steep challenges attendant to survey research in Afghanistan. Because access to technology is spotty, interviewers must conduct face-to-face interviews. Getting a nationally representative sample is difficult given extreme insecurity in some portions of the country, and it is often difficult to survey women. Nonetheless, it is possible to do public opinion research in Afghanistan because over the past decade outside organizations have invested in the developing the requisite infrastructure nationwide.
The primary actor in this area is the Afghan Centre for Social and Opinion Research (ACSOR-Surveys). It conducts the field work for the Asia Foundation’s Survey of Afghans. ACSOR has a well-established survey research operation for in-person interviewing to a high standard across Afghanistan, which it cultivates through recruiting and training indigenous Afghan professionals.
Using the opportunity provided by this framework, in 2007 the Environics Institute teamed up with CBC, the Globe and Mail, La Presse, and the Munk School of Global Affairs to survey Afghans, asking what they think about their country and the presence of Canada and international forces.
Key Findings of the 2013 Survey of Afghans
The 2013 Survey of Afghans includes 89 questions on a variety of subjects, yielding a textured depiction of national and local public opinion. However, findings on four key issues stand out:
1. The Future of Afghanistan: A majority of Afghans believe the country is moving in the right direction, relatively unchanged since 2012. When asked why the country might be moving in the right direction, the top reasons were: reconstruction, schools for girls, good security, the active presence of the Afghan National Army/Afghan National Police, and an improved education system. When asked why the country might be moving in the wrong direction, Afghans are most likely to emphasize insecurity, corruption and unemployment. That security was listed as a reason for both optimism and pessimism suggests that Afghans may at some level feel safer but there remains a pervasive uneasiness about security. Interestingly, when asked about the biggest problems locally (rather than nationally), insecurity was well down the list of concerns (behind: unemployment, electricity supply, roads, and the availability of drinking water. There are, of course, substantial regional differences in how Afghans responded to these questions.
2. International Presence: The survey suggests Afghan trust in international civilian actors. Half of Afghans have some or a lot of confidence in international NGOs – for context, this is much lower than electronic media (68%), but higher than Parliament (47%) and equivalent to national NGOs. However, Afghans are afraid of international forces. 77% of respondents said they would experience some or a lot of fear when encountering international forces, the highest percentage for any activity listed.
3. The Upcoming National Elections: The survey reflects a mixed view of the upcoming elections. Although a majority of Afghans (61%) believe the elections will be free and fair, a similar proportion (59%) anticipated experiencing fear when voting in a national or provincial election. Moreover, Afghans described corruption as the main reason that elections might not be free and fair.
4. Women’s Rights: 90% of Afghans agree with the notion that, regardless of gender, everyone should have equal rights; although questions on particular rights yield lower levels of public support, the survey suggests consistent majority support for women’s rights.
· Political rights: most (83%) Afghans agree that every person should vote for himself regardless of what the community thinks, but only half (53%) say women should decide who to vote for on their own.
· Right to education: consistent with the priority placed on schools for girls as a reason that Afghanistan is moving in the right direction, Afghans see illiteracy as the most pressing problem facing women. The vast majority of Afghans (83%) believe that men and women should have equal opportunities in education.
· Economic rights: lack of job opportunities for women was seen as another pressing problem for Afghan women. Although women’s participation in the formal economy is extremely low (only 5% of women are employed), 63% of Afghans agree that women should be allowed to work outside the home; this has remained consistent since 2006.
The Asia Foundation's 2013 Survey of Afghans represents an important example of how it is possible to conduct high quality and relevant public opinion research in a relatively undeveloped war torn country like Afghanistan. It is critically important, for two reasons. First, it gives voice to Afghans; shielded by the confidentiality of a properly conducted survey, individuals are able to express themselves in a way that might not otherwise be safe. Second, with the question of “what will happen to Afghanistan?” looming, survey data – an aggregative instrument that can nonetheless capture nuance – provides us with the tools to contextualize new developments.
Download the full report here.