A recent poll suggests that 67% of Egyptians are “satisfied with the manner” in which security forces dispersed the sit-ins at Rabaa Al-Adaweya and Nahda Square. Meanwhile a different poll suggests that 79% of Egyptians believe the “massacres” on 14 August were “crimes against humanity”.
These two polls seem to be presenting contradictory facts about how Egyptians feel about the sit-in dispersals. Last week I made the case for considering more facts when forming opinions on the events in Egypt. These two polls indicate just how big a challenge that is currently.
The poll with findings in favour of the manner of the sit-in dispersal was conducted by The Egyptian Center for Public Opinion Research, Baseera. The Egyptian Centre for Media Studies and Public Opinion, ECM, fielded the poll indicating the dispersals were mostly seen as “crimes against humanity”. Neither of these polls describe their methodology clearly enough to fully judge the scientific soundness of either poll. Technical jargon assures us of their statistical validity, but neither really gives enough information. But advanced training in surveys or statistics is not necessary to know whether you can believe the results of these or any other opinion poll. Here are some common sense ways of knowing how to judge.
Lesson one: When a small percentage of a population is surveyed, you seek the opinions of the few to statistically represent the opinions of the entire population. The method is based on intuition. If you want to represent the opinion of the population by asking the few, then the few selected for polling need to look like the whole population. This means that a survey needs to poll people from all governorates. It needs to ask men and women, urban dweller and rural resident alike, of all ages that are representative of the entire population of Egypt. If a survey or poll does not indicate that its sample looks like the population, then it is not likely representing the opinions of the population. Neither of these two polls provides enough information to know how accurately they represent the entire population. Key details to look out for are a sampling of all governorates, divided by gender with some indication that the age distribution of the sample looks like that of the population.
Lesson two: When you ask the question influences the answers you are going to receive. The Baseera poll was conducted between 19-21 August, while the ECM poll was conducted on 15-16 August. ECM asking about the sit-in dispersals on the two days immediately after could have easily biased people’s opinions just as the Baseera poll could have by asking after the widespread violence seen on 16 August. Always take into consideration when questions were asked when thinking about the answers.
Lesson three: How you ask the question matters. In the Baseera poll, 23% felt security forces used excessive force in dispersing the sit-ins. ECM found that 79% of Egyptians believe the events on Wednesday 14 August described as a “massacre” were crimes against humanity. A no or yes response can sometimes really depend on how the question was worded. There is a big difference between asking whether a massacre like 14 August can be considered a crime against humanity OR whether 14 August was a massacre and therefore a crime against humanity. Similarly, what constitutes excessive force and how differently is that defined among the people polled? Changing the wording of the question can elicit a different answer from the same person. Be more trusting of polls that indicate word-for-word how the question was asked. You can make a better judgement on the answers by knowing the questions.
Lesson four: The size of the poll does not necessarily make it better or indicate greater accuracy. Baseera polled 1395 Egyptians while ECM asked 3678 people. It is not true that the latter is better or more statistically powerful at explaining popular opinion just because it surveyed more people. ECM never clearly stated how it asked its respondents questions. To phone 1395 people across the country in a manner proportional to the population would be a better option than asking 3678 people in Muslim Brotherhood geographic strongholds about their opinions. Since ECM did not provide more detail on its data collection, one cannot know whether or not this is a better poll just because it asked more people their opinion. The key here is to pay attention to how, where and when a poll contacted people. Online polls bias findings in favour of youth opinions, as they are bigger internet users than older generations. Landline polls taken during afternoon hours can easily over-represent the opinions of housewives.
These lessons should help in trying to understand these and other opinion polls that will surely follow. Advanced statistics is not necessary to judge the quality of information being received. Neither is an review of the poll’s methodology. Common sense and intuition will often give strong indication of the believability of a poll while you are reading and digesting its findings. Questions like who was asked, where, how and when will often reveal how seriously one can take the results of a poll. Both of these polls have been used to represent one fact or another. In reality, opinions are just that, opinions.
