how social scientists think: what your driver says isn't evidence
We all do it. Fly into a random capital in the developing world with only a couple of days to get what needs doing done, find our taxi driver, and, in a fit of jet lag and the need to get critical information quickly, we ask, "So, what's going on with the election/president/other serious political issue?"
Taxi drivers can be a great source of information about traffic patterns, road quality, and even politics. But to a social scientist, the information any one random individual provides is not in and of itself evidence. No matter how much talk radio the driver listens to.
Why not? I've already written about the need to gather huge amounts of data and the necessity of using appropriate methods to gather and assess information. But how do you know that the data you've gathered is an accurate reflection of reality?
One way to be more certain that the data you've gathered will actually tell you something useful is to be deliberate and systematic about from whom you get information. Whenever a social scientist uses research methods that involve asking people questions (be that through interviews, surveys, or participant observation), it's very important to be sure that you are asking those questions of different kinds of people. Why? Because while it's impossible to talk to every single person who is part of or affected by the issue you're studying, you want to be certain that you're getting a reasonably accurate portrayal of what's really going on.
Just like doctors running medical trials for a new drug don't usually test that drug only on one gender, racial group, or age group, social scientists aim to get information on what we're studying from representatives of all demographic groups in society. We call the process of choosing all different kinds of people "random sampling," and the group from which we eventually gather data is the "representative sample." Without getting too far into the nitty gritty details of how this is done, the main thing is that the representative sample needs to look as much like the general population as possible. If the general population is 60% Ethnic Group A and 40% Ethnic Group B, the sample should reflect that, as it should reflect age, race, gender, education level, religious affiliation, and any number of other demographic characteristics.
Not every study requires a completely random sample, of course. When I'm interviewing civil society leaders in the Congo about their organizations' activities, I don't need to get information from all the housewives in a village that's not connected to those organizations' activities. What I do need to do, however, is talk to as many civil society leaders as I can find. Remember, the goal is to get all possible information.
While this isn't always possible, an ideal social science study has what we call a "large n." Large n is a shorthand way of referring to a large number of cases, or subjects of study. Individuals can be cases, but so can organizations and events. For example, people who study the causes or likelihood of civil wars want to be sure that their answers apply across a wide range of cases, so they often compare what happened in hundreds of civil wars.
When you do a thorough search for data, you're almost inevitably going to get contradictory information. How do you know who's right? The method by which you gather that information helps. If you've done a truly thorough search, then you can be fairly confident that you've gathered a wide range of opinions. You can also compare answers and decide who is a more reliable source on a particular piece of information. But there's always uncertainty, and it's best to acknowledge that up front (more on this later in the week).
How do you know whether a source of information is reliable? You can't always know, but it's a good idea to seek out people who are known to be honest and reliable. Reliability of information is another reason it's extremely important to talk to as wide a range of sources as possible - and to not rely on a driver or fixer to find all your subjects. Like most of us, when asked who would know about a subject, drivers, research assistants, and enumerators will typically refer me to the people they know. In Africa, I've found that these people are almost always family members, neighbors, members of their religious group, and members of their ethnic group or community. And there's nothing wrong with that; just because someone is related to your driver doesn't mean she won't be a helpful source.
But it's not enough. Your driver and his acquaintances (or any other individual) do not constitute a random sample. Particularly if you're studying a place in which the politics and dynamics of ethnicity matter, it's extremely important to make an effort to interview or survey everyone.
But everyone doesn't mean everyone. Like all academics, social scientists are bound by laws and ethical guidelines regarding research involving human subjects. We have to have all our research - including the questions we ask - approved by ethics review boards at our institutions. These boards - usually known as IRB's, or Institutional Review Boards - are there to protect vulnerable individuals from being exploited, endangered, or otherwise put in harm's way. IRB's typically require us to get what is known as "informed consent" from subjects before we ask them any questions. Informed consent procedures let subjects know what research is about, why it is being undertaken, and the specific steps that will be used to protect their identities. IRB's also limit the populations we can research depending on the goal and circumstances of the project.
In my research, for example, I can't interview children, identify any of my interview subjects by name, or interview anyone without getting their informed consent. This is to protect the individuals who are kind enough to provide me with information. Especially when dealing with people who might not fully understand how research is disseminated in the modern world (how would you if you've never used the internet?), it's extra important to ensure that they know what is going on. Is it a hassle to get informed consent from every single research subject? Yes. But is it absolutely necessary in order to protect people from being harmed? Definitely.
So there you have it: from whom you gather information and how matters. How do social scientists' procedures here differ from those of advocates? Do advocates use informed consent procedures? Should they?
Labels: how social scientists think