How we evaluate civil society organisations

With its goal of recommending effective civil society initiatives, one of the main challenges for Power for Democracies from the outset was how to systematically evaluate relevant organisations. Central to this was designing the tools to help us track, gather, and assess information about each organisation we evaluated.

We share the tools here to provide greater insight into the results of this process, and as instruments that institutional funders, organisations, and practitioners might find useful in their own evaluation processes.

The steps we took

The evaluation phase in our pilot project, Effectively Countering Authoritarian Playbooks, came after an in-depth prioritisation exercise where we selected countries and categorised democracy tactics. This produced a list of countries and tactics we identified as potentially effective in countering authoritarian playbook moves in the respective contexts.

Next, we used expert interviews and desk research to create a pool of potential organisations that aligned with our criteria. We then reached out to each organisation on our shortlist to invite them to a call to discuss their work. We evaluated those that aligned with our criteria and consented to participate.

The tools we produced

For the civil society organisation evaluation phase, we used three primary tools: a companion spreadsheet, a data request form, and an evaluation rubric. The companion spreadsheet is primarily a tracking document used in our project management; we're sharing a blank copy so as not to disclose any private information of the organisations we evaluated.

For our data collection process, we used a tool that we refined over multiple versions. The first version was a Word document. We used it to collect data on eight criteria, each with two or three indicators per criterion. We focused on three areas: alignment with our goals of countering authoritarian playbook moves, operational and organisational capacity, and cost-effectiveness and impact potential.

We evolved it further to combine a Microsoft Form and Excel sheet to place greater emphasis on track record and theory of change.

After collecting information from an organisation, we used this rubric to assess each organisation. This process involved referencing the data provided and assigning each indicator a score using a Likert scale. Each researcher then justified their score with reference to information provided by the organisation.

In this anonymised evaluation rubric completed by our team (and shared with the consent of the organisation), two team members evaluated each organisation independently and then met to discuss and reach a consensus on scores.

If an organisation passed the evaluation rubric, the requisite information was aggregated into a ‘defence document,’ which serves as the primary document our researchers use when presenting each organisation to senior management, at which point senior management decided whether to include the organisation in our portfolio of recommendations.

Once ‘passed,’ our researchers draft short recommendations and a full evaluation report, both of which are published on our website.

What we've learned

Two lessons learned during the evaluation phase are worth sharing. The first deals with the operational application of the tools described above. The second involves a broader challenge in the democracy space that was revealed when only a few organisations we contacted had impact assessments demonstrating the effectiveness of their work.

First, it should come as no surprise that the design of a tool has implications for not only the information gathered, but also for how it is assessed.

In the first iteration of our data collection tool we shared a link with each civil society organisation so our team and their team could work on the same document. This allowed us to add comments that the organisation could immediately see and respond to as they entered information. The key benefit of this approach was the ability to quickly respond to information and collaborate on ideas.

However, a critical drawback was the volume of information provided to us to assess. Some organisations, unfamiliar with such a process, as we were, were unclear on what was needed for any given criterion; as a result, they provided us with hundreds of pages of information, but left some indicators unfilled or linked to external documents rather than providing the information directly. The result was too much information, and not the right information in the right place for an efficient evaluation.

To address this issue, we’ve moved to a Microsoft Form and Excel spreadsheet, which in many ways resembles a grant application. This limits what can be submitted for each indicator and narrows the request to specific pieces of information. We are still testing these documents, but so far the volume of information is more manageable and better tailored to our evaluation criteria.

Second, after looking at 131 organisations across Argentina, Indonesia, Italy, and Turkey, we found no rigorous measurements quantifying the impact of these organisations’ work. (Note: Our recommendation in the United States, Freedom2Vote, does have impact measurements. Voter mobilisation in general is an exception to our conclusion).

This had several implications for our evaluation of organisations. Drawing on the work of other organisations that do similar work – albeit with different areas of focus, often on poverty alleviation or animal welfare – we made cost-effectiveness a key criterion for assessing organisations. Without impact evaluations, direct cost-effectiveness analysis is not possible. We therefore concluded that we should rely on ‘cost-efficiency’ – an assessment of cost per outcome without counterfactual analysis – and place greater weight on organisations’ theory of change and track record as indicators of potential impact.

The lack of rigorous impact evaluations presents a significant opportunity that organisations like ours are well placed to address. As we partner with civil society organisations working on numerous tactics that strengthen democracy, we have a real opportunity to add to the evidence base for measuring impact. We will report back on progress in this area.

We encourage readers to use or share these tools. They are not perfect, but we believe they offer a strong starting point for collecting data and assessing organisations’ work – particularly for organisations that are working on topics that are challenging to measure, or where impact measurement approaches are traditionally less utilised.

We welcome suggestions on how to improve our process or tools. Please email me your thoughts and ideas.