how social scientists think: anecdotes aren't evidence
Every time I write about the lack of systematic evidence that there is a causal relationship showing that mining is the root cause of violence in the DRC, I get a long list links to stories, articles, and advocacy materials that purport to show just that. But the links to which passionate readers direct me almost never prove their point. Why? It has to do with the nature of evidence and the way it's gathered. Social scientists are a diverse bunch, but if there's one thing we can agree on, it's this: anecdotes aren't evidence.
The most important thing to understand about social scientists is that we are just as obsessed with how information is obtained and analyzed as we are with what that data tells us. We call the process of obtaining and analyzing data our research method.
Research methods are important because they determine the quality, kind, and extensiveness of the data we get. The method by which we analyze our research also validates (or invalidates) that we are actually answering the research question we set out to answer.
It's extremely important that the method match the research question. While we'll fight over this to high heaven on boring panels at obscure conferences, deep down, most social scientists agree that certain methods are better for certain kinds of questions. We divide methods into several types; some are numbers-based (those are called quantitative), others are based on descriptive information (qualitative methods). There are also ways of using logical reasoning and advanced mathematics to explain or predict the decisions leaders or states might take - that falls into a broad category called "game theory."
Before your eyes glaze over too much, remember that you don't need to understand the different types of methods. All you really need to understand is that the important thing is that the method used to obtain data will allow the researcher to answer the original research question. If I'm studying how young women perceive militias in the DRC in relation to their personal religious and political beliefs, for example, my method needs to allow me to answer that question. In this case, I'd probably use interviews, focus groups, and possibly a survey to get an answer to those questions, which provide me with the kind of data I need to explain that relationship. If, however, I want to know whether conflicts in the DRC have an effect on school enrollment, attendance, and completion figures, I would use a very different method that involves lots of numerical data and statistical analysis. Interviewing families might give me some useful background on the subject, but it won't really provide the data necessary to get an answer. The method has to match the question.
It's also important to get all the information about a subject possible. This means that not only do you have to read everything that's ever been written on the topic, but also that you have to make a good faith effort to gather all available data. That means finding and evaluating everything you can possibly get. If you're thinking, "That sounds really inefficient," you're right. It takes forever. It took me four years to gather the data for my dissertation. Now that it's a book project, the data-gathering continues, well into year seven, with no end to the data-gathering in sight.
Why do you need so much information? Because having a lot of information is the only way to be sure you've covered all the bases and teased out every possible explanation to be sure that yours is the right one. The goal of social science is to develop theories (more on that later this week). For a theory to be useful, it needs to tell us generalizable information, that is, information that can be applied in more than one situation. If information can't be applied at a general level, across contexts, space, and time, it's of limited utility. We don't just want to know how to end the use of rape as a weapon of war in the DRC; we want to know how to end the use of rape as a weapon of war everywhere. More importantly, we want to know what strategies don't end the use of rape as a weapon of war, as well as what effects don't explain why rape stopped being used as a weapon of war in a particular situation. So we need to find generalizable information, which means that we have to eliminate every possible alternative explanation, which means that we need to analyze as much information as we can.
How does this make us different from advocates, 0ther than that we have the luxury of time and self-set deadlines? It means that we can't rely on anecdotal evidence. Why not? Because anecdotes aren't good predictors of general phenomena - that is, we can't say that because something happened once, we know the cause of every similar event. Just because one woman was raped by LRA soldiers does not mean that all rapes in the DRC are committed by the LRA.
Now, I can hear some advocates protesting, "But we do gather systematic evidence!" And I know that's true. In some cases. However, far too often, the advocacy materials that end up in the general public's view rely far too much on anecdotes and far too little on systematically-gathered, methodically-analyzed evidence.
Why does this happen? Part of it is because of time constraints. Most advocates don't have the luxury of spending 2-10 years on a single project. Many have to do research on quick, in-and-out trips to the field that last at most two or three weeks. It's impossible to gather solid evidence in that time. But it's easy to gather heartbreaking, soul-stirring anecdotes that will move campaigners to action. More importantly, it's necessary for a good advocate to do so. Would you give money and time to a cause that didn't make you feel something, or that you could make a difference?
That's part one. Tomorrow we'll talk about why from whom evidence is gathered matters. For now, do you perceive differences in the ways that academic social scientists and advocates gather information? What are the strengths and weaknesses of each approach? I'm particularly interested in hearing from advocacy people on this.
Labels: how social scientists think