"Africa is, indeed, coming into fashion." - Horace Walpole (1774)


how social scientists think: edition Kristof

I've avoided reading and writing about The Kristof lately as I've really been trying to get the stress in my life under control. But this merits a response as apparently somebody still doesn't understand the difference between evidence and anecdotes.

In case you've been hiding under a rock, here's a quick rundown on what happened. The Kristof wrote a long piece extolling the virtues of Do-It-Yourself Aid projects, in which amateurs circumvent large aid agencies to implement programs on their own. Dave Algoso wrote a very measured and kind response pointing out that, actually, aid is a very difficult profession and not one that amateurs are equipped to undertake and do well. Tales from the Hood also wrote an excellent response in which he noted that aid is not about us, while DIY aid is often all about the do-er.

The Kristof responded to this barrage of criticism in a late Friday afternoon blog post, which seems to be his preferred time to respond to criticism. In it, he wrote:
Compared to professionals, amateurs tend to be more, well, amateurish. Accountability can be a real problem. But on the whole, I think this concern is misstated. I’ve generally found that grassroots, locally owned aid projects have a better record than large scale, top-down ones that don’t always have the same buy-in. And the truth is that DIY aid projects are more likely to be modest, grassroots efforts undertaken with strong local partners. They often keep their ear to the ground and tinker with their model more than the larger projects. Aid projects often succeed at the experimental level and then have difficulty going to scale, but that’s less of an issue with DIY investments that are never meant to scale up (that’s a separate problem with them, and a legitimate concern).
Tales from the Hood tackles, brings down, and sends to the locker room the idea that amateurism isn't that big of a problem in two wonderful posts on professionalism in aid here and here. What I want to focus on is the problem with Kristof's claims here in terms of research methods, namely that there's not any evidence for his claims. "I’ve generally found that grassroots, locally owned aid projects have a better record than large scale, top-down ones that don’t always have the same buy-in" may be true based on Kristof's limited exposure to local aid projects, but it's not one that's supported by any systematically gathered evidence that I've seen. Has anyone else?

The problem here is that Kristof is relying upon anecdotal evidence (NGO's he has encountered) rather than systematically gathered evidence. Even though Kristof has more anecdotes than your average observer, it's still not evidence. As @WrongingRights tweeted while quoting her dad, "The plural of anecdote isn't data." Data that isn't gathered systematically isn't data at all.

Why is this problematic? Because there are exceptions to every rule. In the social sciences, we call these exceptions outliers. You can't base a theory on outliers, because then you'd be wrongly explaining general phenomena based on an unusual case. Because of this, we generally place outliers in what is called the "error term." The error term is kindof like the remainder in a long division or algebra problem. We leave those cases out of our studies in order to avoid tainting our results. We do so in order to get the right answer - the one that explains what happens most of the time under given conditions.

Because Kristof's only research method is his personal observation, we can't be sure that he's not simply making general claims on outliers. He's not using data; he's using anecdotes. And anecdotes are a slippery slope on which to base claims about the kind of aid work that will best aid the world's poor.

Also, I have to point out that there's a HUGE difference between "grassroots, locally owned aid projects" and the sort of DIY aid projects conceived and executed by well-meaning foreigners. I actually think he's probably right that the former work much better than many other projects, because they're grounded in the community. But that's not what Kristof wrote about in the piece. His article was entirely about Whites in Shining Armor, not grassroots, locally-conceived projects. And there's no evidence that I know of that shows that projects conceived by well-meaning, idealistic foreigners work better than professional, INGO-supported aid.

I met Kristof once in the eastern DRC. Based on that one encounter, I could claim that Kristof's standard research method is to go to the best local NGO a city has to offer and then to take the word of a few officials at major international NGO's and UN agencies as truth. But I can't make the claim that that's how he always works. Why? Because I only have one case. And that case could be an outlier. It could be that Kristof was having a bad day, or was scared to death of the eastern DRC, or that he accidentally drank the tap water at the Ihusi Hotel. I don't know. What I do know is that we shouldn't be making decisions based upon unreliable, anecdotal evidence. And if you want to be an aid worker, you'd better know what you're doing.

(Photo: Bernardo Guzman, via Inside Higher Ed)

Labels: ,


Blogger ewaffle said...

From a quick scan of Kristof's blog post (I would have to be paid to read the entire thing for a second time)

"But on the whole, I think..."

"I’ve generally found that..."

"...but overall I’d have to say..."

Sorry, Bill Kristof, member of the Harvard board, you flunk Rhetoric 101.

These quibbles are in addition to the major problem with his post which your and others point out so well--despite its title he doesn't answer the questions posed by readers and instead tries to redefine DIY as local grass-roots based organizations.

Monday, November 01, 2010 2:10:00 PM

Blogger Kim Dionne said...

We should also point out a problem in the data collection of Kristof's anecdotes -- there is likely a huge selection bias. Do you think he goes to every mom-and-pop DIY project? I think it more likely that he goes to the success stories he's heard about so he can write a narrative that will attract readers.

For the academic types (or even the wannabes), check out Geddes's Paradigms and Sand Castles to learn more about selection bias in social science.

Monday, November 01, 2010 2:28:00 PM

Anonymous Anonymous said...

Is there any evidence that large aid projects work better than small "amateurish" projects?

Monday, November 01, 2010 4:41:00 PM

Blogger Jeff Mowatt said...

Here's how we've done ours working on the ground, networking with citizen activists and leveraging government support for social enterprise;


Tuesday, November 02, 2010 1:40:00 AM

Anonymous Anonymous said...

Get'em Doc.Remember your a Maroon Tiger now!

Tuesday, November 02, 2010 5:36:00 AM

Anonymous Anonymous said...

"Then again, I’m writing for a mostly American audience, and I’m writing not about development as such but about Americans doing work in development."

"And the truth is that it’s already extraordinarily difficult to get readers to focus on needs half a world away; one way to do so, is to use Americans as bridge characters. Without them, even fewer people will focus on these issues.

Frankly, lack of interest is likely to become more serious. I think America is turning inward again because of our economic difficulties."

Think you are missing some major obvious points Texas. Your are judging Kristof on how good a social scientist he is. He is not a social scientist and does not want to be social scientist.

"how social scientists think: edition Kristof"

"in which amateurs circumvent large aid agencies to implement programs on their own."

"not one that amateurs are equipped to undertake and do well."

"Tales from the Hood tackles, brings down, and sends to the locker room the idea that amateurism"

"I met Kristof once in the eastern DRC. Based on that one encounter,"

Qualifications of a professional
Commitment. Just a one-word answer.
Dave Algoso

You are on a loser defending Dave Algoso original argument as after his qualifying what he said, he does not seem to have the same point of view as you, or maybe he has multiple points of view and the whole thing is more like a intellectual excise to him.

"This is my attempt to process what I’m learning, both in school and in general. I’ll keep it as focused as my intellectual wanderlust will allow. International development is a pretty broad topic, and it connects tangentially with politics, management, economics, war, human rights, public health, ethics, and much more. I’ll probably share some half-formed thoughts."

"My undergraduate studies were in philosophy and physics, which made sense at the time but I’ve forgotten why."

Not a lot of point of jumping all over him as he is still learning, not a lot of point of criticizing Kristof as being a poor social scientist when he is not one and not a lot of point Texas of you defending a previous point of view of Dave Algoso's when he seems to largely abandoned it, and may have a totally new point of view come his next blog.

Sorry about your stress level was tempted to not even reply as any reply from me was bound to bump it up!

I would not be to upset though if you want to move on to some other issue as this one seems a bit fruitless now for the reasons i have laid out.

That would only be my anecdotal opinion as a white amateur with poor Academic qualifications though Lol.

Tuesday, November 02, 2010 11:19:00 AM

Blogger texasinafrica said...

Anon @11:19, as always, if you don't like the content here, then I'd suggest that you find other blogs to read.

Tuesday, November 02, 2010 2:04:00 PM

Blogger Matt Davies said...

Oh boy, Kristof's argument is the classic "if A=B and B=C then A=C".

Or put another way, "If Trabbants (DIY projects) are small" and "small cars (local, community projects) are more reliable than large mercedes (big-NGO led projects), "then trabbants are more reliable than mercedes"!

I think the evidence from Eastern Germany tells another story!

Wednesday, November 03, 2010 10:16:00 AM

Anonymous Anonymous said...


G+M = $50 Billion bailout

Wednesday, November 03, 2010 10:27:00 AM

Anonymous Roxanne said...

First of all, thank you for contributing to this debate and for the content you publish on this blog. It is a wonderful resource for development practitioners worldwide and I always enjoy turning to it.

As far as this post is concerned, I disagree that Kristof's anecdotal, story-based methodology is erroneous. I believe in "the power of the story" and these individuals can inspire and motivate. I do not think Kristof posits that this is THE way to do development work or that any development practitioner can be as successful as the individuals he profiles. But if these profiles can cause debate and stir individuals and promote thinking about involvement in the field of development, then there are potential gains for the readers and the communities in which they choose to be involved alike.

I have further elaborated on this here: http://stagonastithalassa.blogspot.com/2010/11/in-defense-of-nicholas-kristof.html.

Thank you once again for the insightful perspectives you add to this debate!

Wednesday, November 03, 2010 10:56:00 AM

Anonymous Bradford said...

Thanks for this post.

I feel like your post raises again a question I had during your great series of posts last week on methodology. What do we do with the observation that argument A lacks evidence? And in particular, what do we do when argument B also lacks evidence?

Your post strongly criticizes Kristof for his lack of evidence. OK. But you endorse the posts of Algoso and TFTH, none of which have any evidence by the standard you're looking for, either. And your post doesn't offer any evidence on either side of the question.

So where does that leave us? I'm not saying that we should quit checking for evidence. But methodological critiques based on lack of evidence just feel sort of hollow when no other evidence is offered, and when alternate views are endorsed that don't provide evidence, either.

I feel like a methodological analysis of this debate would have to go something like: "nobody in this debate can offer evidence, so we are left arguing based on our impressions, which is really too bad since we're just going to end up going in circles. Let's all try to get some real evidence to bear on this debate. Until then, I instinctively disagree with Kristof but I recognize that neither of our positions are really verifiable right now."

Without that sort of analytical transparency, methodological critiques start to feel like substantive critiques made without actually having prove any substantive points. Which sort of feels like cheating.

I say this without trying to weigh in on the merits of Kristof's point. Your post is about methodology, and I just wanted to comment on that.

Again, thanks for the HSST series, it's been really thought-provoking!

Saturday, November 06, 2010 3:58:00 AM


Post a Comment

<< Home