I’m pleased to be invited as a guest blogger! I actually don’t know many of the other insightful bloggers here, which is a shame since there are so many interesting ideas. I stay holed up in East Africa running the GlobalGiving Storytelling Project and doing whatever it takes to turn >13,000 stories into meaningful community information for the hundreds of local NGOs we serve. Being in the communities (but not involved in story collection) allows me to see whether the process is going awry.
The most valuable piece of information for GlobalGiving (as a non-implementing, non-funding facilitator for community organizations) is not what is said in the community stories about an organization, but how the organization reacts when given a set of relevant information about itself. It helps us identify community-focused organizations, good listeners, and potential innovators – and generally break through the self-report bias that prevents international development from having a larger impact in the world (or in public policy). While we haven’t built a tool to track how NGOs react to and use this information yet, we have done a lot to track the flow of information from one party to the next – which helps us measure the reliability of the information. Here are some examples of tracking this flow of information.
We reached out to all our local partner organizations and asked them to to recommend a dozen or so young people (ideally: women just after high school but before finishing college), whom we trained as scribes. After many dozens, we now have over 1000 scribes in Kenya and Uganda, each asking their friends and neighbors to share two stories about a “time when a person or organization tried to help someone or change something in your community.”
A few of these scribes wanted particular organizations talked about. That’s why we now have over 400 stories about World Vision – entirely from one region in Uganda. We know they work everywhere, but people rarely want to talk about elsewhere. Luckily, most scribes are not choosing the organizations, but instead are working for the $0.10 we reimburse them for completed story.
Sometimes people in one country avoid saying that people in neighboring countries are willing to say. Over thousands of stories, we find that Ugandans just don’t call anything a bad idea:
Kenyans are on the left, Ugandans on the right.
So we were lucky in that (a) we didn’t test this survey only in Uganda and (b) we offered people the choice of a softer answer: “good idea that failed.”
We allow every person to share exactly two stories, including the scribes we train. Looking again at stories from Uganda, stories a scribe collects from another person (blue, left) offer a much more diverse viewpoint than stories the scribe writes himself (red, right):
Either this pattern means that Ugandan scribes like staying positive more than their friends, or a lot of the people whom we recruited have a vested interest in telling positive stories about certain organizations.
Minimal Tech: We found that the best way to ensure we involve the broadest cross section of society as possible was to distribute paper forms and not use SMS, smart phones, computers, smart pens, tape recorders, or any other kind of technology. This means even people without a phone participate, yet many of these people still have a friend or relative with a phone – which can be used for tracking them. However, with a team of 15 transcribers, some interpret the corners of triads differently from others:
Individual transcribers are color-coded. The orange person and the purple person see the “edge” of the triangle differently. We reduced this indexing bias by giving each indexer performance feedback: this picture. But it is much harder to detect and correct a similar variation among the authors of the stories, who may interpret the triads differently.
SenseMaker(R) often relies on questionnaires that use triangles with three answers at the points:
This allows the user to choose any answer, even if it falls in the gray areas between two or three choices. Not everybody likes thinking in shades of gray. (I suspect Former President George W. Bush would stick to the corners on any traid-question based survey.)
Since our story form includes four triads, I can assume that any person who fills out all four questions by sticking to the corner may not have seen triads as a question that allows one to answer in shades of gray. If it can be defined logically, you can tag it in SenseMaker(R) and use it in your analysis.
If that assumption not conservative enough for you, you could look at all the stories from a scribe, and if every answer falls in the corners, I would tag that whole batch of stories as “scribe didn’t understand triads.”
Add it up
Combining each of these story meta-data should allow anyone to filter 13,000 stories down to a smaller number that are a reliable reflection of community opinions. This is what we want:
- Stories come from many people
- Stories come from many places
- Scribe and storyteller understood the questions
- Scribe and storytellers provided mobile phone numbers as identification
- Scribe collected stories about many NGOs (my threshold for “ok” is that no one NGO is talked about in more than a quarter of the stories.)
- Storyteller followed the “2-stories per person” rule