"Africa is, indeed, coming into fashion." - Horace Walpole (1774)


how social scientists think: what your driver says isn't evidence

We all do it. Fly into a random capital in the developing world with only a couple of days to get what needs doing done, find our taxi driver, and, in a fit of jet lag and the need to get critical information quickly, we ask, "So, what's going on with the election/president/other serious political issue?"

Taxi drivers can be a great source of information about traffic patterns, road quality, and even politics. But to a social scientist, the information any one random individual provides is not in and of itself evidence. No matter how much talk radio the driver listens to.

Why not? I've already written about the need to gather huge amounts of data and the necessity of using appropriate methods to gather and assess information. But how do you know that the data you've gathered is an accurate reflection of reality?

One way to be more certain that the data you've gathered will actually tell you something useful is to be deliberate and systematic about from whom you get information. Whenever a social scientist uses research methods that involve asking people questions (be that through interviews, surveys, or participant observation), it's very important to be sure that you are asking those questions of different kinds of people. Why? Because while it's impossible to talk to every single person who is part of or affected by the issue you're studying, you want to be certain that you're getting a reasonably accurate portrayal of what's really going on.

Just like doctors running medical trials for a new drug don't usually test that drug only on one gender, racial group, or age group, social scientists aim to get information on what we're studying from representatives of all demographic groups in society. We call the process of choosing all different kinds of people "random sampling," and the group from which we eventually gather data is the "representative sample." Without getting too far into the nitty gritty details of how this is done, the main thing is that the representative sample needs to look as much like the general population as possible. If the general population is 60% Ethnic Group A and 40% Ethnic Group B, the sample should reflect that, as it should reflect age, race, gender, education level, religious affiliation, and any number of other demographic characteristics.

Not every study requires a completely random sample, of course. When I'm interviewing civil society leaders in the Congo about their organizations' activities, I don't need to get information from all the housewives in a village that's not connected to those organizations' activities. What I do need to do, however, is talk to as many civil society leaders as I can find. Remember, the goal is to get all possible information.

While this isn't always possible, an ideal social science study has what we call a "large n." Large n is a shorthand way of referring to a large number of cases, or subjects of study. Individuals can be cases, but so can organizations and events. For example, people who study the causes or likelihood of civil wars want to be sure that their answers apply across a wide range of cases, so they often compare what happened in hundreds of civil wars.

When you do a thorough search for data, you're almost inevitably going to get contradictory information. How do you know who's right? The method by which you gather that information helps. If you've done a truly thorough search, then you can be fairly confident that you've gathered a wide range of opinions. You can also compare answers and decide who is a more reliable source on a particular piece of information. But there's always uncertainty, and it's best to acknowledge that up front (more on this later in the week).

How do you know whether a source of information is reliable? You can't always know, but it's a good idea to seek out people who are known to be honest and reliable. Reliability of information is another reason it's extremely important to talk to as wide a range of sources as possible - and to not rely on a driver or fixer to find all your subjects. Like most of us, when asked who would know about a subject, drivers, research assistants, and enumerators will typically refer me to the people they know. In Africa, I've found that these people are almost always family members, neighbors, members of their religious group, and members of their ethnic group or community. And there's nothing wrong with that; just because someone is related to your driver doesn't mean she won't be a helpful source.

But it's not enough. Your driver and his acquaintances (or any other individual) do not constitute a random sample. Particularly if you're studying a place in which the politics and dynamics of ethnicity matter, it's extremely important to make an effort to interview or survey everyone.

But everyone doesn't mean everyone. Like all academics, social scientists are bound by laws and ethical guidelines regarding research involving human subjects. We have to have all our research - including the questions we ask - approved by ethics review boards at our institutions. These boards - usually known as IRB's, or Institutional Review Boards - are there to protect vulnerable individuals from being exploited, endangered, or otherwise put in harm's way. IRB's typically require us to get what is known as "informed consent" from subjects before we ask them any questions. Informed consent procedures let subjects know what research is about, why it is being undertaken, and the specific steps that will be used to protect their identities. IRB's also limit the populations we can research depending on the goal and circumstances of the project.

In my research, for example, I can't interview children, identify any of my interview subjects by name, or interview anyone without getting their informed consent. This is to protect the individuals who are kind enough to provide me with information. Especially when dealing with people who might not fully understand how research is disseminated in the modern world (how would you if you've never used the internet?), it's extra important to ensure that they know what is going on. Is it a hassle to get informed consent from every single research subject? Yes. But is it absolutely necessary in order to protect people from being harmed? Definitely.

So there you have it: from whom you gather information and how matters. How do social scientists' procedures here differ from those of advocates? Do advocates use informed consent procedures? Should they?



Anonymous Bradford said...

Thanks for this very interesting series of posts.

I think that it is a bit misleading to suggest that your caricature of an advocate's epistemology (stuff the taxi driver said) is the alternative to social science epistemology. It seems that there are some valid forms of understanding (and evidence) that lie between the two. I'm thinking of journalistic epistemology a la Kapuscinski; sociological grounded theory a la Glaser; and professional experience a la Autesserre (who, in "Trouble with Congo", includes her time working for MSF-Es in DRC as part of her "ethnographic work", even though I doubt she had committee-cleared questionnaires, etc.) (Thanks, by the way, for the recommandation - it's a great book!)

These all seem to be valid and useful ways of learning about and explaining a situation, although of course each produces different kinds of information useful for different ends.

I know that it isn't your purpose to describe all epistemologies, only your own. But I think it would be useful to recognize that other valid epistemologies exist, and that advocates (and others) might be using them. It might be useful in part because they provide standards by which to judge people using different methods. Are advocates really just relying on taxi driver wisdom, or are they at least satisfying one of the different valid epistemologies? What do those epistemologies tell us and not tell us? At what point are those epistemologies "not enough", and theories need to be tested by other methods? It seems like those questions are important and are more useful than the question of (now I will do the caricaturing) whether the method meets the standard of a peer-review board.

But again, thanks for explaining your view in detail, and for giving this question the attention it deserves!

Wednesday, October 20, 2010 4:58:00 AM

Anonymous Anonymous said...

One difference between advocates and researchers may be that advocates approach their work looking to validate pre-drawn conclusions (or campaign slogans), whereas researchers are (hopefully!) entering with a hypothesis that they rigorously test.

Wednesday, October 20, 2010 12:17:00 PM

Anonymous Ian said...

Laura - I'm very much enjoying these posts and the comments as well as the opportunity to tease you about them on twitter :-)

I'd just like to echo the point made by Bradford from the point of view of the discipline I work in, that of Knowledge Management.

People frequently ask me what we mean by "knowledge" and I'd have to say it is a very multifaceted thing. Evidence coming from formal research is an important source of knowledge but it's far from being the only one.

A lot of useful knowledge is also what we call "tacit" knowledge i.e. things people know from experience, personal interactions, skills etc. that isn't written down anywhere and some of which it might not even be possible to write down. We use this knowledge to make decisions every day, even if we don't recognize it.

Some types of problems don't lend themselves well to research either because of lack of data, complexity or their immediacy. Politics and how to navigate it is a good example. (By the time you understand why a warlord acts a particular way it might well be too late to use that knowledge to change it - in fact you might never be able to objectively describe it - only understand it through your own feelings and relevant experience). We also have to avoid thinking that research results = "reality". Any research is just a sample or simplified model of reality. Anyone who has worked in data collection knows that we don't really know for example how many rapes there were in the Congo - all we have is an estimate based on a set of assumptions and a methodology: much better than a set of crafted anecdotes - but imperfect nontheless.

This also doesn't mean that there is no science to tacit knowledge - there is an expanding body of learning around this - but it is much less objective and quantifiable, and that's OK.

So for me a good advocate will combine hard data and research with tacit knowledge and "skill" in order to determine what action to advocate for and how best to do it.

P.S. I have to say that the plural of anecdote IS evidence - but any good detective needs to consider a range of evidence not just a single source or type of it that backs up an initial hunch.

Wednesday, October 20, 2010 12:48:00 PM

Anonymous Bronwyn said...

On some level, though, isn't it important to hear what the taxi driver has to say - along with everyone else? Can he be part of your large n?
Really interesting post, by the way - it makes me think about who I should actually talk to for information

Wednesday, October 20, 2010 2:32:00 PM

Anonymous Ian said...

A quick additional thought on who we collect information from.

If I've got a lot of time and have the technical skill and access to do research then my best bet might be to ask a representative sample of people a set of carefully designed standardized questions or use some other well established research method.

If I don't have this then what I'll probably do is ask someone who I think knows something about what I want to know, preferably but not always someone I trust or respect, ideally a few of them (but likely n <5) and then come up with my own opinion of the answer based on that. (e.g. I've never been to DRC and so if I wanted to know something about DRC and conflict minerals I'd ask Laura)

Since most people are not able to do research themselves, or possibly even read it in detail and understand the technicalities they tend to mostly ask "experts" (or read Kristof) to get their answers. Senior decisions makers particularly need to do this - they trust that others have done research and/or know what they are talking about.

This is not a bad thing. What needs to be worked on is improving the interface between research and those who need to use it but can't "get it" directly and have to rely on conversations, relationships and trust to get the information they need. (and this is why I like "knowledge Management" since that's really what KM is all about.

Wednesday, October 20, 2010 3:13:00 PM

Anonymous Anonymous said...


Wednesday, October 20, 2010 6:11:00 PM

Anonymous Jason Hahn said...

Thought you would like this post re taxi drivers http://financialaccess.org/node/3576

Thursday, October 21, 2010 12:36:00 AM


Post a Comment

<< Home