At the Global Voices Citizen Media Summit 2010 , GV Executive Director Ivan Sigal  led a panel discussion on Measuring and Evaluating the Impact of Citizen Media  with –
- Anne Nelson , Professor of Journalism at Columbia University
– Lova Rakotomalala , GV French Editor, and Foko-Madagascar  Co-founder
– Rosental Alves , Professor of Journalism at University of Texas at Austin, and GV Board Member
As citizen media initiatives evolve, the role of citizen media should move from making your voice heard to moving the needle. In this session, we will discuss how we can measure the impact of citizen media initiatives and improve their effectiveness.
In her pre-conference notes , Anne charts the lifecycle of a citizen media project and the difficulties in measuring its success –
The past decade of growth in citizen media has… yielded a tremendous bounty of both individual expression and data. Once new media platforms become available, new communities use them in unexpected ways. Many of them have obvious social utility.
So here we have a logical progression: good project emerges, donors offer seed money, project grows.
The crunch arrives at the next phase of development. Good project grows, grant expires, next proposal is submitted. Now the bar is raised: it's not enough to have an ìinteresting idea.î The project, however worthy, must either make the leap to being ìsustainableî… or develop a ìbusiness modelî.
This is where the ìimpact evaluationî rubber hits the road. For any projects that participate in a commercial sphere, the metrics are taking shape, logically enough, around the principles of commerce. It's now easy to measure web traffic, clicks, and sales. It's hard ó if not impossible ó to measure ìdemocratization.î
Government and corporate funding agencies have different approaches to measuring the impact of citizen media projects they are funding.
Governments have foreign policy and economic agendas that guide the choice of projects they fund, so, it's important that the grantees understand and share these objectives.
Several corporate funding agencies like the Gates Foundation, Skoll Foiundation, and Omidyar foundation see citizen media projects as a means to the end and insist on measurement, while other funding agencies like the Knight Foundation insist less on measurement.
In general, funding organizations have a history, a track record, and activists need to understand who they are themselves, who their funders are, before they seek funding.
Measurement approaches might involve quantitative approaches based on web metrics, qualitative approach based on anecdotes, and moving the needle on policy. In the end, however, it's about visualizing the change you are trying to bring in the world, and making it happen.
The BBC World Service Trust , for instance, is very rigorous in measuring its projects with an intent to improve their effectiveness. It uses a structured approach including formative research, pre-testing, audience feedback, and impact evaluation using a combination of multiple tools.
In his pre-conference notes , Lova shares the background of Foko-Madagascar –
The Foko-Madagascar group blogging project was set up to highlight… (initiatives for) positive social and environmental change in many Malagasy disenfranchised communities. Unfortunately, a political crisis changed the context of our work and many of the trained bloggers turned their focus.
Although the political crisis is still ongoing in Madagascar, the Foko project has mostly returned to its initial purpose… (and documented) children tree-planting… reinsertion programs for HIV+ people and disenfranchised adolescents… and ICT for development workshops… (Now, we are) getting ready to help oversee the scheduled national elections in May with the local Ushahidi platform.
In Madagascar, web traffic or worldwide media attention won't be sustainable, apart from a crisis, like a coup. So, how do you make the case for a sustained citizen media project?
The first step is not finding the funding, but finding the need to share. Then, we define objectives, build scenarios and analyze what would have happened if we hadn't done the projects. Finally, we use a combination of quantitative and qualitative measures and share them with the authors so that they become better over time.
The traditional media ecosystem was like the desert dominated by a few media organizations. The new media ecosystem is like the rain forest with an abundance of voices.
Donors are trying to understand how we are moving from one kind of media environment to the other, so measurement is complicated.
Donors need to realize that things are destroyed before new things are built, so we can't measure new media by old standards.
Citizen media projects need to become better at articulating what they are trying to achieve because the funders are coming from a radically different environment.
It is in this context that the open-minded funding approach of Knight New Challenge is very important.
The participants were actively involved and asked several questions.
Some questions were only tangentially related to the topic of measuring citizen media projects –
- Independent media has always existed, so isn't the citizen media revolution actually an evolution? Yes.
– Shouldn't software development projects also be considered citizen media? Yes, Ushahidi is a good example.
Other questions pushed the panel for some hard answers –
- If you are trying to measure the impact of your citizen media project, to improve the project, and not to raise funds, does your approach to measurement change?
– What is the goal of citizen media: to get the voices out, to bring about social change, or to bring about a specific type of social change?
– There's an assumption that citizen media is a normative good. How do you measure both the positive and negative outcomes of a citizen media project?
Anne: The activists who start citizen media projects often don't think of measurement, until a funding agency asks them to. It's important that they see measurement as a means to achieve what they set out to.
Ivan: It's important to measure against a prescriptive norm, but also important to measure emergent benefits, so that you are not only trying to achieve what you measure.
Rokutamo: It's important to measure both quantitative and qualitative outcomes and give constructuve feedback to the contributors so that they can become more effective. Sometimes, your objectives might change in response to your context, but you must continue to measure yourself against the objectives.
Rosental: It's important that we protect the organic nature of citizen media projects and the passion driving it. So, storytelling plays an important role in defining the success of an initiative.
Ivan: The urgency of funders to understand and support the citizen media ecosystem may have both positive and negative effects, because sometimes the very nature of small volunteer-based citizen media projects change when they get big funding.
My own view:
My own view is that we are so intent on finding and celebrating successful citizen media projects that we might be blinding ourselves to the lessons we can learn from failures. It's critical, then, that we ask hard self-reflective questions about how to measure the success of our own citizen media projects and celebrate failures as well as successes, in public. I applaud Failfare  for starting this important conversation and was delighted that 50-plus participants joined me in an open session at #GV2010 to share the lessons they have learnt from their failures.
These notes have been cross-posted on Gauravonomics .