How to plan an evaluation?
1. What to clarify before contacting an evaluator?
First of all, discuss with your project team and ideally with key stakeholders (donor, other institutions, community):
- Background: project name, identification, history, objectives, results, key activities, progress over time (add logical framework if you wish – you get more tailored proposals), organisational, social and political context of the evaluation, main stakeholders involved in the project including target groups, beneficiaries, partners, donors.
Purpose: What would you like to get out of the evaluation? What would you like to learn? Why do you do the evaluation now? (for example, you plan a subsequent project on a bigger scale, or you would like to apply the approach elsewhere, or you would like to find out why your approach did not work in a certain context)
- Use: What are the expected evaluation outputs? (debriefing, presentation, report, videos, infographics, posters – printed or on-line…) Who would use the evaluation outputs, when and how? Who else should know? Shall we make the evaluation report public? How? Why not? For example, if you work with a remote community without access to technology, publishing the report on-line would probably not contribute to their awareness of the project achievements.
- Scope and focus: What are the project details? Which components, geographical areas, period of time etc. would you like to evaluate? What is its logical framework or theory of change? What are key stakeholders? What monitoring and evaluation data are available?
- Evaluation criteria and questions: What exactly do we want to know? OECD/DAC evaluation criteria are still the most frequently used ones (required also by the European Commission, together with additional ones). OECD/DAC also suggests some very general evaluation questions. Ideally start without them and do a brainstorming. Try not to ask too many, 7-10 are just fine. There are three main types of evaluation questions – descriptive, normative and cause-and-effect. Be careful to choose appropriate evaluation design as per the evaluation question! (Evaluators will help you.)
- Methodology: How would the evaluation questions be answered? In an external evaluation, you would probably expect an evaluator to propose this. Nevertheless, it is useful to think of the indicators or answers for each evaluation question, sources of information (documents, stakeholders), data collection methods (quantitative like surveys or qualitative like interviews) and data analysis / reporting. It gives you an idea of evaluation timeline, budget and other aspects.
- Timeline: Who needs to participate in the evaluation planning? Who will be approached by the evaluator during the inception to get a project overview and to plan methodology and logistics in detail? Who will be involved in data collection, such as interviews, group discussions, surveys etc.? When are important events that evaluator can join? When are key stakeholders (not) available? When can the evaluation team and the actors do a joint sense making and draw joint recommendations? Is this realistic?
- Budget: Given all the above, what is the estimated budget for evaluator's remuneration and expenses as well as expenses incurred by you and other stakeholders? What budget do you have available and thus what is realistic?
- Human Resources: Do we have the capacities (expertise, money, time) to do the evaluation internally? If we have an external evaluator, what should be the key requirements? Who will coordinate the evaluation with the evaluator, project partners and others? Who is the evaluator accountable to?
Based on the above, you would draft the Terms of Reference, ToR and share them with potential internal and / or external evaluators.
It is also possible to create a TOR together - just let me know.