"Africa is, indeed, coming into fashion." - Horace Walpole (1774)


how social scientists think: anecdotes aren't evidence

Every time I write about the lack of systematic evidence that there is a causal relationship showing that mining is the root cause of violence in the DRC, I get a long list links to stories, articles, and advocacy materials that purport to show just that. But the links to which passionate readers direct me almost never prove their point. Why? It has to do with the nature of evidence and the way it's gathered. Social scientists are a diverse bunch, but if there's one thing we can agree on, it's this: anecdotes aren't evidence.

The most important thing to understand about social scientists is that we are just as obsessed with how information is obtained and analyzed as we are with what that data tells us. We call the process of obtaining and analyzing data our research method.

Research methods are important because they determine the quality, kind, and extensiveness of the data we get. The method by which we analyze our research also validates (or invalidates) that we are actually answering the research question we set out to answer.

It's extremely important that the method match the research question. While we'll fight over this to high heaven on boring panels at obscure conferences, deep down, most social scientists agree that certain methods are better for certain kinds of questions. We divide methods into several types; some are numbers-based (those are called quantitative), others are based on descriptive information (qualitative methods). There are also ways of using logical reasoning and advanced mathematics to explain or predict the decisions leaders or states might take - that falls into a broad category called "game theory."

Before your eyes glaze over too much, remember that you don't need to understand the different types of methods. All you really need to understand is that the important thing is that the method used to obtain data will allow the researcher to answer the original research question. If I'm studying how young women perceive militias in the DRC in relation to their personal religious and political beliefs, for example, my method needs to allow me to answer that question. In this case, I'd probably use interviews, focus groups, and possibly a survey to get an answer to those questions, which provide me with the kind of data I need to explain that relationship. If, however, I want to know whether conflicts in the DRC have an effect on school enrollment, attendance, and completion figures, I would use a very different method that involves lots of numerical data and statistical analysis. Interviewing families might give me some useful background on the subject, but it won't really provide the data necessary to get an answer. The method has to match the question.

It's also important to get all the information about a subject possible. This means that not only do you have to read everything that's ever been written on the topic, but also that you have to make a good faith effort to gather all available data. That means finding and evaluating everything you can possibly get. If you're thinking, "That sounds really inefficient," you're right. It takes forever. It took me four years to gather the data for my dissertation. Now that it's a book project, the data-gathering continues, well into year seven, with no end to the data-gathering in sight.

Why do you need so much information? Because having a lot of information is the only way to be sure you've covered all the bases and teased out every possible explanation to be sure that yours is the right one. The goal of social science is to develop theories (more on that later this week). For a theory to be useful, it needs to tell us generalizable information, that is, information that can be applied in more than one situation. If information can't be applied at a general level, across contexts, space, and time, it's of limited utility. We don't just want to know how to end the use of rape as a weapon of war in the DRC; we want to know how to end the use of rape as a weapon of war everywhere. More importantly, we want to know what strategies don't end the use of rape as a weapon of war, as well as what effects don't explain why rape stopped being used as a weapon of war in a particular situation. So we need to find generalizable information, which means that we have to eliminate every possible alternative explanation, which means that we need to analyze as much information as we can.

How does this make us different from advocates, 0ther than that we have the luxury of time and self-set deadlines? It means that we can't rely on anecdotal evidence. Why not? Because anecdotes aren't good predictors of general phenomena - that is, we can't say that because something happened once, we know the cause of every similar event. Just because one woman was raped by LRA soldiers does not mean that all rapes in the DRC are committed by the LRA.

Now, I can hear some advocates protesting, "But we do gather systematic evidence!" And I know that's true. In some cases. However, far too often, the advocacy materials that end up in the general public's view rely far too much on anecdotes and far too little on systematically-gathered, methodically-analyzed evidence.

Why does this happen? Part of it is because of time constraints. Most advocates don't have the luxury of spending 2-10 years on a single project. Many have to do research on quick, in-and-out trips to the field that last at most two or three weeks. It's impossible to gather solid evidence in that time. But it's easy to gather heartbreaking, soul-stirring anecdotes that will move campaigners to action. More importantly, it's necessary for a good advocate to do so. Would you give money and time to a cause that didn't make you feel something, or that you could make a difference?

That's part one. Tomorrow we'll talk about why from whom evidence is gathered matters. For now, do you perceive differences in the ways that academic social scientists and advocates gather information? What are the strengths and weaknesses of each approach? I'm particularly interested in hearing from advocacy people on this.



Anonymous Anonymous said...

Please don't put this on the test. Just kidding. Question how much does the cultural biases of the researcher come into play? Or more importantly when doing such research what tools do you use to step "outside" of yourself. I would think a god researcher must be part "Jedi Knight" regarding disciplining your mind.

Tuesday, October 19, 2010 5:07:00 AM

Anonymous Anonymous said...

Please don't put this on the test. Just kidding. Question how much does the cultural biases of the researcher come into play? Or more importantly when doing such research what tools do you use to step "outside" of yourself. I would think a good researcher must be part "Jedi Knight" regarding disciplining your mind.

Tuesday, October 19, 2010 5:08:00 AM

Anonymous IdealistNYC said...

But if it takes social scientists up to 10 years to answer a single question - assuming they can get the data in the first place - then how much use is social science in affecting change? Political situations change over a decade. What was true at the beginning of the century might not hold true now. This is one of the major failings of social science that academics often don't like to address.

Tuesday, October 19, 2010 8:39:00 AM

Anonymous Michael said...

My sense is that many times advocates draw out anecdotes/personal stories/etc. from systematic research to enhance their campaign. Maybe this is naive, but if this isn't case, then the really interesting question to look at is why advocates would choose to rely on anecdotes that stand in contradiction to systematic research (in the cases where the systematic work has been done)?

Tuesday, October 19, 2010 8:40:00 AM

Blogger texasinafrica said...

@anon at 5:08 - That's a great question and a very hard one to deal with. If you're dealing with hard numbers, it's a little easier (the population is the population and it's demographic and other characteristics don't change regardless). But it gets complicated when you start asking questions: what you mean by "democracy" or "good governance" or "religious belief" or "individual identity" may not be at all what your subjects think you mean by those terms. So it's very, very important to involve local scholars in the process of developing surveys, translating them, etc. It's an imperfect method of control, but it does help.

I find that the hardest part of doing research in the DRC is remaining emotionally uninvolved. It's really hard to interview a doctor who deals with hundreds of rape cases each year and who doesn't have enough in the budget to effectively treat them all without wanting to give him $100 on the spot.

Also, it helps to learn as much as possible about the culture before going in to do the research. Knowing the languages, the history of the region, the history of the organization, and the particular quirks of the situation really helps with understanding what people are actually telling you, vis-a-vis what they think they're telling you.

But, yeah, it's messy. And difficult.

Tuesday, October 19, 2010 8:52:00 AM

Blogger texasinafrica said...

@IdealistNYC - No arguments from me on that. It's clearly problematic. One hopeful sign is that research is bubbling its way to the top more quickly now with the advent of blogs and websites. You don't have to wait for research to be peer-reviewed in the traditional process to get it out there, anymore (although there are serious disincentives to go through the new means of dissemination of information, namely the tenure process).

Another thing to consider is the generalizability issue. It may be worth it to spend years on a single question if it gives us information that can be applied in other situations. One of the frustrations academics have with advocates is that because their job is to effect change, they often act on the basis of whatever information they've got at the time. That's understandable, but what if the information is wrong or incomplete? They could actually do more harm. But waiting around for solid data means people might continue to die. It's a dilemma.

Tuesday, October 19, 2010 8:55:00 AM

Blogger texasinafrica said...

Michael, I'm not (and have never been) an advocate, so it's hard to know. The most problematic cases I've seen have chosen the anecdotes because they were awful situations - the advocates encountered these horrible, horrible things that had happened, felt a need to do something to alleviate that suffering, and extrapolated generalizations based on very limited evidence.

One thing I'm hoping to conclude by the end of this series is that we'd all be better off if academic research were more focused on getting solutions to problems, and if advocates had better training in research methods. I can dream, right? :)

Tuesday, October 19, 2010 8:57:00 AM

Anonymous Julie G. said...

I think also it's important to think of social scientists collectively. People are doing thorough work all the time. There is a value to getting accurate information and generating valid conclusions that can be applied to other circumstances.

That social scientists aren't good at providing quick answers is an advantage as far as I am concerned rather than a constraint. While a single social scientist's work might not be put to quick effect (like TXinAfrica), the collective work of social scientists at any one time are broad. It might take one out of a specific country for analysis for force one to look in different geographical areas for similar policy or practical experiences.

Tuesday, October 19, 2010 10:28:00 AM

Anonymous Julie G. said...

That's all to say: social scientists work at a snail's pace that is sure to be frustrating to advocates. But doesn't/shouldn't the advocacy community capitalize on the work produced by social scientists as a mechanism to generate policy approaches?

Tuesday, October 19, 2010 10:40:00 AM

Anonymous Anonymous said...

The problem with the current campaign on Congo being run by the Enough people is that their methods are weak--basically relying on work others have done and repackaging it as their own. The result has been to vastly oversimplify the conflict and to create minerals=rape and minerals=war perceptions that don't necessarily help the situation. But on top of that, Enough claims to speak on behalf of the Congolese people, and raises who knows how much money in their name, to save them from themselves, through policy changes in the USA.
Social scientists have been challenging Enough, but not with sufficient force and openness. Hopefully that will change.

Tuesday, October 19, 2010 1:19:00 PM

Blogger dcat said...

I'd be aware of overstatement. "Anecdotes" may not be evidence in the social sciences, but stories of actual lived experiences, what some might call anecdotes, is certainly historical (and sociological) evidence. Historical events are not repeatable and subject to the sort of experimentation of science, but something that happens just once can be pretty significant. Let's not get too hung up on trying to turn lived experience into a math problem.

Let's not conflate dumb use of evidence with the fact that lived experience can be a form of data. A historian or sociologist can presumably draw a great deal from any number of discrete historical events.


Tuesday, October 19, 2010 6:01:00 PM

Blogger texasinafrica said...

Great point, Derek. Of course, you're very familiar with these debates. I'm emphasizing the pattern-driven side of the social sciences here because I think that's what distinguishes those disciplines from history and other discrete events. But I do think there's much more of a need for context in the social sciences. Ceteris paribus drives me nuts.

Tuesday, October 19, 2010 6:09:00 PM

Blogger texasinafrica said...

Great point, Julie G.

Tuesday, October 19, 2010 6:10:00 PM

Blogger dcat said...

TiA --
I also think that what you are trying to say, and that what I'm trying to say, is that use of anecdote needs to be earned. And this ends up sounding really, well, arrogant and prickish.
Let me make the case by way of analogy: I try to beat using contractions and first-person out of my students' writing. And inevitably they come up with a piece of writing (sometimes my own) that uses first person. So to preempt that I explain that using first person or contractions is earned. That you need to be able to write formally, without contractions and first person, before you can abuse the rules, something every good writer learns to do.
The same holds with anecdote. If George Clooney knew a bit more, I'd feel ok with him using anecdotes to illuminate the whole. The problem is that for too many of the fly-in celebrity activists, the anecdote is the whole. And when the anecdote is the whole you get the ZOMG EVERY TIME YOU USE A CELL PHONE A WOMAN GETS RAPED! sort of stuff.
Anecdote can serve the evidence. It isn't the sum total of the evidence. Knowing that difference tends to be earned. And it shows why a little bit of knowledge is a dangerous thing. It also helps to explain why for all of the reputation about academics being radical squishy-headed leftists, to the activist crowd we seem hopelessly hard-hearted and even "conservative." Knowing more tends to lead to complexity, and complexity tends to lead to moderation.


Wednesday, October 20, 2010 8:07:00 AM

Anonymous Daniel said...

Sorry to come a bit late to the party on this post...

Thanks for putting these up. I wouldn't overstate the difference that there *should* be between social science research and advocacy, though.

I teach in a school of public policy, which I guess puts me somewhere in the no-man's-land between social science and advocacy (and I'm trained as a philosopher, so you can fit what I know about proper methodology on a postage stamp, probably). But I routinely have students present policy recommendations based on reading one source (and often a clearly slanted one, like a report on a program's effectiveness by that program). I have to work hard to get them to understand how important it is to look broadly at what's been said about an issue, consider alternative explanations for what's gone wrong (and hence alternative policies for how to fix it), etc.

My sense is that there *are* two important distinctions. First, policy advocacy isn't research and so, as anon (1:19) mentions, it's inevitable that advocates will be relying on the primary research of others to a large extent. Enough may be doing it badly - but that's why a big part of my job is to make my students good, critical, *consumers* of research.

The second is that, like historians, policy folks are concerned with the particular problem at hand more than with generalization (in the Aristotelian dichotomy, I guess we're historians and the social scientists are poets?). So certain kinds of methodology, like process tracing, may be more appropriate than they would be for a political scientist.

But the anecdotal approach, if it goes beyond pulling out stories to tug at the heartstrings to drinking your own kool-aid, is no better for advocates than "pure" researchers. (Actually, I even worry about it as a tactic for getting "political will," b/c you're asking your audience to not engage in the critical thought that you have... but that's my moral philosopher background peeking through).

Wednesday, October 20, 2010 9:14:00 AM

Blogger Coen J H van Wyk said...

I must disagree with the absoluteness of the condemnations. Often donors and major NGO's work through Government institutions, which ties them into structures that marginalise certain groups.

Look at Research in Bangladesh, an initiative to provide rigorous scientific methods to local groups with which to determine their needs and develop solutions. While this initiative does enjoy some foreign funding, it is the despair of the donor community because often the scientifically based solutions to these developmental problems turn out not to need much money to solve, but rather access to training, markets, the normal mechanisms of society!

We are now looking at a project to integrate a gypsy-like population group, used to living on and beyond the margins of society, into a farming community through training and basic skills provision, all at minimal cost, and outside of the formalised structures that normal donors require.

Thursday, November 04, 2010 11:28:00 AM

Blogger texasinafrica said...

Coen, that's great, but you do see that you're using an anecdote to make your point, right? I can agree that one group is doing good work in one place, but that's not systematic proof that all groups doing work in all places are in fact helping.

Thursday, November 04, 2010 9:32:00 PM


Post a Comment

<< Home