I'm pleased to be invited as a guest blogger! I actually don't know many of the other insightful bloggers here, which is a shame since there are so many interesting ideas. I stay holed up in East Africa running the GlobalGiving Storytelling Project and doing whatever it takes to turn >13,000 stories into meaningful community information for the hundreds of local NGOs we serve. Being in the communities (but not involved in story collection) allows me to see whether the process is going awry.
The most valuable piece of information for GlobalGiving (as a non-implementing, non-funding facilitator for community organizations) is not what is said in the community stories about an organization, but how the organization reacts when given a set of relevant information about itself. It helps us identify community-focused organizations, good listeners, and potential innovators - and generally break through the self-report bias that prevents international development from having a larger impact in the world (or in public policy). While we haven't built a tool to track how NGOs react to and use this information yet, we have done a lot to track the flow of information from one party to the next - which helps us measure the reliability of the information. Here are some examples of tracking this flow of information.
A few of these scribes wanted particular organizations talked about. That's why we now have over 400 stories about World Vision - entirely from one region in Uganda. We know they work everywhere, but people rarely want to talk about elsewhere. Luckily, most scribes are not choosing the organizations, but instead are working for the $0.10 we reimburse them for completed story.
Sometimes people in one country avoid saying that people in neighboring countries are willing to say. Over thousands of stories, we find that Ugandans just don't call anything a bad idea:
Kenyans are on the left, Ugandans on the right.
So we were lucky in that (a) we didn't test this survey only in Uganda and (b) we offered people the choice of a softer answer: "good idea that failed."
Either this pattern means that Ugandan scribes like staying positive more than their friends, or a lot of the people whom we recruited have a vested interest in telling positive stories about certain organizations.
This allows the user to choose any answer, even if it falls in the gray areas between two or three choices. Not everybody likes thinking in shades of gray. (I suspect Former President George W. Bush would stick to the corners on any traid-question based survey.)
Since our story form includes four triads, I can assume that any person who fills out all four questions by sticking to the corner may not have seen triads as a question that allows one to answer in shades of gray. If it can be defined logically, you can tag it in SenseMaker(R) and use it in your analysis.
If that assumption not conservative enough for you, you could look at all the stories from a scribe, and if every answer falls in the corners, I would tag that whole batch of stories as "scribe didn't understand triads."