MAC-Team

Elaborating an Evaluation Plan

Evaluate Europe Volume 1 finalWhat is a project plan?
Evaluation can be defined as the systematic appraisal of the success of a project. Success refers both to the quality of the project (whether the outcomes meet the needs of the target grops) and its results (whether the project objectives have been achieved).
Depending on the purpose of the evaluation, a distinction is usually made between process (or formative) and effect (or summative) evaluation. Process evaluation is done during the project, and aims to monitor the implementation process, improve the work in progress and increase the likelihood that the project will be successful.
Effect evaluation is usually done towards the end of the project, and aims to verify if the project objectives have been achieved.

Planning the evaluation

Although the effects of a project are usually achieved at the end, evaluation must be planned from the outset and conducted throughout the project life time. An evaluation plan specifies which aspects will be focused upon for the evaluation, and explains how the quality of the project implementation and the achievement of the project outputs and outcomes will be assessed. The evaluation plan will serve as a basis for the project manager(s) and project team to monitor the project’s progress and evaluate the effects.

Key elements of an evaluation plan

Evaluation questions
Evaluation planning starts with the formulation of specific evaluation questions. These questions relate to both the quality of the implementation (process evaluation questions) and the success of the project (effect evaluation questions).

  • For process evaluation, the questions should be linked to the planning and organisation of the project activities, focusing on whether the activities are implemented according to plan, how obstacles and difficulties will be identified during the implementation and dealt with, and how the quality of the project implementation will be assured.
  • For effect evaluation, the evaluation questions should be linked to the specific objectives, and verify if the stated objectives have been achieved.

In stating the evaluation questions it is important to incorporate the stakeholders’ views, and focus on what they want to know. This may be assessed directly by asking the stakeholders which questions they would like to see answered for the evaluation, as part of an "evaluability assessment". It is also important to make sure that the questions can be answered unambiguously.

Evaluation indicators
Indicators are variables that measure the performance of a project and the level to which the objectives are reached. By quantifying the evaluation questions, they provide a possibility to measure and monitor the progress of the project and assess the extent to which the objectives are attained.

  • Indicators measuring the progress of the project are process indicators. They verify the accuracy and timeliness of the steps foreseen for the project implementation.
  • Indicators measuring the project outputs are performance indicators. They relate to the level of participation on the project, user satisfaction, efficiency, take-up, etc.
  • Indicators measuring the project outcomes are effect indicators, and relate to the achievement of the objectives. If the objectives have been formulated SMART (specific, measurable, achievable, realistic, timed), it should be possible to specify one or more variable measuring the level of achievement of each objective.

Ideally, evaluation indicators are simple metrics that are easy to measure, and should be objective, valid, reliable and repeatable.

Evaluation targets
Indicators can also specify target values (e.g., numbers expected, level of quality aimed for) to serve as a standard to compare the process or results of the project with. These target values may be difficult to define at the start of the project, but setting targets will give the project team something to aim for. Examples of targets are:

  • 1,000 users per day will visit the web site
  • 80% of participants in a training will express satisfaction with the content
  • patient uptake will improve by 10% in two years
  • 4 out of 5 institutions approached say they will adopt the guidelines

Evaluation methods
Once evaluation questions have been formulated and indicators specified, it is possible to specify the way in which data will be collected for the evaluation. Methods must be specified for each evaluation question. These methods can be quantitative or qualitative.

  • Quantitative methods include:
    • Questionnaires and surveys – Closed or open-ended questionnaires can be used to ask opinions of participants or target groups in a systematic way. Questionnaires can be mailed out, sent by email, or posted on the web.
    • Records – Records of project activities contain important information about the way the participants react to the project content. Analysing these records allows to identify trends and patterns.
    • Web server logs – Analysing the use of the project website informs about user trends.
  • Qualitative methods include:
    • Interviews – Structured, semi-structured, or unstructured interviews in person or by phone are useful to explore opinions and issues in depth on a one-to-one basis.
    • Focus groups – Group sessions with small groups of people allow to explore different views on an issue, or clarify issues to complement other data collection methods.
    • Observation – Observing specific components of the project can be a powerful way to learn about the participants’ responses and uptake of the project.
    • Expert opinions – The opinion of experts on specific components of the project can yield objective information on whether the project meets quality criteria.

The adequacy of these methods does not depend on the methods themselves, but on whether or not they will answer the evaluation questions. Moreover, the methods chosen should also match the context for the project implementation and the expectations of the target group and stakeholders.

Timing
Evaluation is done throughout the project. An evaluation plan needs to specify which evaluation aspects will be focused upon at which time. At the start of the project, the focus is likely to be on assessing the stakeholders’ views, the establishment of indicators and the measurement of baseline data for the effect evaluation. During the project, process and performance indicators will be measured at various moments, specified as milestones in the implementation process. Towards the end of the project, effect indicators measures will be assessed to evaluate the project outputs and outcomes.


Source: original news by the European Executive Agency for Health and Consumers
R
eport: A Project Manager’s Guide to Evaluation (1.01 MB)

more info: 
 - Project Management Infokit. Jisc Infonet. www.jiscinfonet.ac.uk
 - Wholey JS, Hatry HP, Newcomer KE (Eds) (2004). Handbook of Practical Program Evaluation (2nd Ed). San Francisco: Jossey-Bass.
 - Hughes J, Nieuwenhuis L (Eds) (2005). A Project Manager's Guide to Evaluation. Evaluate Europe Handbook Series. Bremen: ITB Institute Technology and Education.