How They Did It: Using a Mobile Phone Survey to Investigate South Sudan’s Conflict

Within just weeks, nearly everything South Sudanese millionaire Joseph Lugala Wani had built was gone. A hotel and 10 shops, destroyed. Looted and burned. Months later, as the urban refugee shared his story from his small rental home in neighboring Uganda, he told us it was government soldiers. They put tires inside the building and set it on fire, burning down the same hotel where officials used to stay.

Read more about the stories behind innovative investigations in GIJN’s How They Did It^[1] series.

Our team was reporting on forced displacement, property destruction, and occupation in South Sudan. It’s the newest country in the world, having gained independence from Sudan in 2011. We’d been hearing rumors of land grabs for months; some of our colleagues had even witnessed it first-hand. But we wanted to know who was responsible — and how widespread it was.

With that in mind, in 2017 I worked with my colleague, Lagu Joseph Jackson^[2], a South Sudanese journalist working in exile, to design a mobile phone survey to access people across the country. The goal was to reach people we couldn’t get to due to government restrictions, illiteracy, and poverty. From that came the project: Forced Out^[3].

Reporting on Places You Can’t Go

South Sudan had been cracking down on the media. Journalists were threatened, censored, jailed, and sometimes killed for reporting the truth about the country’s violent civil conflict. Many foreign reporters — especially those who had written stories that angered the government’s national security authorities — were denied accreditation and barred entry to the country, myself included. For journalists still inside, access to areas affected by conflict was difficult, expensive, and time-consuming. The country ranks 138 out of 180 on the Reporters without Borders 2020 World Press Freedom Index, between Palestine and Myanmar.

Many South Sudanese are displaced in camps or have sought refuge outside the country’s borders. Many people don’t own smartphones to document the atrocities being committed, as have been used in other conflicts in Syria or Sudan. Even when they do, many South Sudanese are afraid, and do not want to risk being caught with the evidence. But nearly everyone has a basic mobile phone.

That was why for us, and for other journalists working in similarly repressive environments, a phone poll may offer a solution. This same technique could also prove invaluable during the COVID-19 pandemic, as journalists find sources are self-isolating or that they’re working in regions with newly restricted access rules. Here are a few things we learned during the project that might be useful to other investigative journalists reporting on communities that are difficult to reach in person.

Many people don’t own smartphones to document the atrocities being committed, as are often used in other conflicts such as Syria or Sudan. But nearly everyone has a basic mobile phone.

A Phone Survey in Six Languages 

Our survey was pre-recorded in six languages — English, Arabic, Dinka, Nuer, Bari, and Madi. Respondents could answer by pressing numbers on their phone, and in some cases recording a message. The mobile phone survey we used was distributed through random dialing on a national network. For other reporting projects, different options could include sending a mass text message to the network for people to agree to participate, or calling numbers from a specific list — for example if you had a list of phone numbers of health workers or workers from a specific industry. We chose a pre-recorded voice survey because of literacy and internet penetration barriers, but you could also allow people to click through to an online survey or respond through voice messages or texts in regions with better phone and network access.

Collaboration Essential

Our team grew as the project did, and we added partners as needed. We began with a grant from the Pulitzer Center on Crisis Reporting^[4] that enabled Jackson and I to partner with Viamo^[5], a company that runs services and polls over cellular networks. We worked with statisticians and experts who provided invaluable advice on the methods we used.

We then were linked with African Defence Review^[6] (ADR), a media house focused on security. ADR sent journalists into the country, and helped us with satellite research to track the destruction nationwide, while Jackson and I reported from Uganda. After many months of research, reporting, and survey design, Al Jazeera came on board as our publisher, editing the story and designing the graphics and multimedia format.

We learned to plan for much more time and money than our initial budget allowed, due to working with a new method in a difficult region of the world. We faced immense challenges gathering the survey data. Drop-off rates — how quickly people hung up on the surveys and stopped responding — were high. It was a problem we’d anticipated, but not to the extent we faced. At the same time, the country’s economy was collapsing so the cost to run the survey kept increasing. We had to reduce the number of questions, change the order of the introduction, and ask for more funds to ensure our sample size was big enough.

An Honest Methodology

It was also crucial to ensure that our story was honest about the limitations of its methodology, and the reasons we chose it. When it came time for us to interpret the data, we wanted to be certain that our analysis didn’t misrepresent our flawed — but necessary — approach. We made every effort to back-up and corroborate our data with other reporting, including interviews on the ground, public records such as reports from the United Nations or Human Rights Watch, and research using satellite imagery. We also decided not to treat the data as statistically significant, considering the challenge of determining South Sudan’s demographic make-up, and knowing that our respondents skewed toward male, younger, and people in urban areas with better network coverage. We made all those decisions and weaknesses clear in our methodology, so that people would read the story with that knowledge.

We told readers that we called more than 35,000 numbers through random dialing, with more than 2,900 people listening to the introduction and selecting a language. A total of 405 people completed the entire 14-question survey, which was designed in consultation with South Sudan land rights experts and statisticians. Of the 405 people, more than 40% said they had been forced off their land or out of their home since December 2013; nearly half blamed government soldiers.

We worried people would question our reporting, so we invested many weeks of work in the corroboration of the evidence and survey data.

Verification & Source Protection

Another lesson was to make every effort to corroborate the evidence using multiple reporting and research techniques; this is especially important in the case of a highly-sensitive project.

We gathered submitted photos of the damage to properties and used open source investigative techniques to verify it. We looked at the image metadata to confirm it was taken on the phones of our sources. We matched submitted images to satellite imagery, and found signs of destruction that aligned with the timelines described in our interviews. We confirmed data on destruction from the survey through geolocation, by finding primary schools that we’d asked people to name, and looking up satellite imagery to see whether there was also visual evidence of destruction in the images. In nearly every case, the evidence lined up. We found patterns of widespread destruction in villages and towns across the country.

We took into account source protection and security throughout the process. We weren’t sure how capable the government was of interrupting our research or listening in to responses. To counter that, we added a detailed introduction and reminders that the respondents could hang up at any time. We decided against including my name in the introduction, to protect sources from being affiliated with a banned journalist.

We also researched the government’s surveillance capabilities, even considering switching to running the survey in refugee camps rather than South Sudan, although we eventually decided that wasn’t necessary. There were technical barriers Viamo had to resolve at the beginning; in some of our first tests there was interception of an automated nature that wasted phone time (and budget) without providing usable responses. It’s not clear whether that was the government or not, but it caused delays.

Gratifying Results

It was a long process, and many moments felt discouraging. We worried our data was too limited to be meaningful, and delayed while adjusting the survey to ensure the drop-off rate improved. And we worried people would question our reporting, so we invested many weeks of work in the corroboration of the evidence and survey data. We also wanted to ensure that the story was compelling and spent considerable time on the storytelling and finding examples of real people whose lives were affected, so that it wasn’t all about the data.

It was more than worth it — the story reached people across the country and put numbers to a topic we’d heard only regionally or anecdotally. It led to discussions in the South Sudanese community around land rights and the peace agreement, and won third place in the Investigative Reporters and Editors’ Philip Meyer Award^[7] for the best use of empirical methods in journalism.

Perhaps our methods could be replicated in other countries undergoing similar challenges during and after the coronavirus pandemic; countries where people cannot safely move or authorities want to block information, but where there are many people just waiting to be heard.

Carolyn Thompson^[8] is a freelance data journalist based in Nairobi, Kenya. She reports primarily on displacement, human rights, economy, and politics. Her work has appeared on Al Jazeera, the Canadian Broadcasting Corporation, France 24, and the Washington Post, among others.

References

1. ^{^} How They Did It (gijn.org)
2. ^{^} Lagu Joseph Jackson (www.linkedin.com)
3. ^{^} Forced Out (interactive.aljazeera.com)
4. ^{^} Pulitzer Center on Crisis Reporting (pulitzercenter.org)
5. ^{^} Viamo (viamo.io)
6. ^{^} African Defence Review (www.africandefence.net)
7. ^{^} Investigative Reporters and Editors’ Philip Meyer Award (www.ire.org)
8. ^{^} Carolyn Thompson (twitter.com)