Curriculum
---------------------------------------------------------
E. Evaluation |
What is evaluation? Evaluation involves assessing “ ‘effects’ or outcomes, as measured against predetermined criteria.”[1] Academic freedom defenders may evaluate the extent to which any given action they have taken has helped to avert an immediate crisis. Actions and strategies may also be evaluated with regard to their contribution to increasing respect for academic freedom generally or for the human rights of scholars.
Why evaluate? In order to ensure that they are implementing the four strategies described earlier in this section (as well as the standards discussed in Part B and the actions discussed in Part C) as effectively as possible, academic freedom defenders are urged to undertake regular evaluations of their work, including overall goals, strategies used to achieve them and specific tools and actions deployed in each situation of concern. What is working well? What is not? What else can be tried? The importance of building an evaluation component into all strategies to promote and protect academic freedom cannot be overstated. Regularly evaluating your strategy will help to improve the effectiveness of actions and identify best practices and lessons for the future. Evaluation is a means to measure achievements, assess cost-effectiveness, and identify early warning signs or signs of progress that may alert you to the need to try alternative responses. Undertaking regular evaluation of your organisation or committee’s work will have long-term benefits both for the organisation or committee, as well as direct benefits for your ‘clients’.
What to Evaluate? The first step of the evaluation process is to establish the criteria that you wish to evaluate. Examples of evaluation criteria might include:
- numbers of individuals who have benefited from your actions in relation to numbers of individuals who sought your assistance
- the different groups or categories of stakeholders who benefited from your actions
- whether your actions had any unintended adverse effects or did any harm
- which actions had the most impact
- reasons why these actions proved more effective than others
- how visible your actions were to targeted stakeholders and the wider public
- the strengths and weaknesses of your committee or organisation as it goes about its work
- the extent to which your committee or organisation is fulfilling its goals and staying ‘on mission’
- whether political, economic, social or cultural conditions have changed significantly over the course of the period you are evaluating
- the extent to which any urgent actions feed into an overarching strategy to promote academic freedom
- whether your strategy is sustainable and your actions replicable in different contexts
How to Evaluate? Academics and advocates should begin by developing a simple methodological approach for the evaluation. The best approaches combine both qualitative and quantitative elements and may involve some or all of the stages outlined below:
Assign Responsibility: It is important to give some thought to who the best person in your organisation or committee to conduct the evaluation may be. Depending on availability of resources and the evaluation criteria (e.g. if the evaluation focuses heavily on your own organisation or committee’s internal strengths and weaknesses), you may prefer to have an external evaluator conduct the assessment. If your evaluation involves the development of questionnaires or interviews with stakeholders, it is important for the evaluator to have the right aptitude and skills for such work, e.g. good communication and negotiation skills, good research and methodological skills, and an understanding that data is rarely ‘neutral’ and may be used selectively by stakeholders to present a point of view.[2]
Data collection: Before beginning any interviews with stakeholders or circulating questionnaires, a certain amount of desk research can likely be done for initial data collection purposes. Examples include: research on numbers of incidents reported or requests for assistance to you and your committee over a given period; numbers of situations in which you and your committee were able to provide assistance or intervene; types of challenges reported; source of problems (e.g. university faculty or administration, government agencies, other state or non-state actors); mapping exercises to identify the stakeholders in your response work; reports on legislative or policy changes relating to academic freedom and university values that have come about as a (direct or indirect) result of your activities; changes in work environment that have had a direct impact on your committee’s activities, etc.
Questionnaires: If your committee has been able to intervene in a significant number of situations it may be helpful to prepare a number of evaluation questions to present to either a portion of the people you have assisted or to all. To ensure a good response rate, make it as easy as possible for the recipient to complete the questionnaire. When developing or formatting the questionnaire, consider the work you will need to do to collate responses. Let participants know whether the questionnaire is or is not anonymous. Use a combination of ‘open’ and ‘closed’ questions. Open questions leave greater space for the participant to elaborate on his/her opinions on a matter (e.g. “Tell me about…” or “Describe…”). Closed questions (e.g. Yes/No questions, or scales of agreement 1-5) may help you gather important data or facts, or may make results easier to process, especially when anticipating large numbers of responses.[3] It is a good idea to also include an ‘Other’/’Have we missed anything?’ section to the questionnaire to leave space for participants to address an area you may have overlooked. Finally, always include room for participants’ suggestions for improvement or changes to enhance impact and outcomes of your actions or strategies.
Interviews: Conducting phone or face-to-face interviews with stakeholders will be more time consuming than asking them to complete questionnaires, but there may be significant pay-offs associated with this approach. People will often say things in person that they might prefer not to put in writing and such information may provide additional important insights or add new dimensions to the evaluation. This is particularly true when asking for sensitive or personal information from participants in relation to any challenges they are facing. Interviews should include more open-ended questions. This allows time for the interviewer and interviewee to establish a personal connection (important if the interviewee is being asked for sensitive information) and allows the interviewee to provide greater detail on each issue.
Group discussions: With careful consideration to any privacy or security issues or concerns, it may be helpful to gather a small group (5 to 8 people at most) together for a focused discussion with an experienced moderator. Participants who have had similar experiences might find it helpful to share their views with each other and with your committee about the actions you have taken to address their concerns. Arrive with a series of questions for the participants to discuss, building up gradually to the more sensitive questions. Participants may be reticent at first but, with a thoughtful and well-prepared moderator, the discussions should prove fruitful.
Site visits: As discussed in Part C, a site visit is a pre-arranged trip to a country, university or other location of interest by a group of concerned individuals for the purpose of sharing their concerns and gathering information. Site visits or follow-up visits can also be used as a means of evaluating the success of a past or ongoing actions or strategies. In most cases, an evaluation site visit consists of interviews, meetings and events with persons familiar with the situation or issue. Because of the time and expense required, site visits should only be undertaken when the situation or issue is of particular importance for evaluation purposes. After the visit, the group generally prepares a report which includes recommendations for reforms, improved practice or further discussion. For further information on site visits, see Part C.
Evaluation report: Once the information has been gathered, findings should be made and summarised in an evaluation report. The report might include the following sections:
- Goals or mission of your committee or organisation
- Evaluation criteria: A description of the aspects of your work or organisation being evaluated
- Evaluation methodology: A description of the methods you used to collect quantitative and qualitative data, e.g. interviews, questionnaires, site visits etc.
- Examples of any questionnaires or interview questions
- Summary of findings: A summary of the key data or other information gathered in the course of the evaluation
- Recommendations: Using the findings as your starting point and in discussion with your committee or organisation, develop a set of recommendations on how to improve on the process, outcomes or impact going forward. Clearly assign responsibility and include a timeline for implementing recommendations.
- Conclusion: Make a clear commitment to repeating the evaluation process at regular intervals and to following up on implementation items.
Once it is prepared, a draft report might be shared for comments within the organisation or committee, or with participants in any questionnaires, interviews, discussions or site visits. After it is finalised, the evaluation report might remain confidential for use internally within the organisation, or it might be intended for wider distribution, such as with clients, partners, funders or other stakeholders or the public.
Useful Resources:
- Amnesty International, “Evaluation, A Beginner’s Guide,” July 1999, available at: http://www.amnesty.org/en/library/asset/POL32/003/1999/en/923cbe2e-e0f0-11dd-be39-2d4003be4450/pol320031999en.pdf
- “Evaluation in the Human Rights Education Field: Getting Started” , Felisa Tibbitts, Human Rights Education Associates, available at: http://www.hrea.org/erc/Library/display_doc.php?url=http%3A%2F%2Fwww.hrea.org%2Fpubs%2FEvaluationGuide%2Findex.html&external=
Conclusion
Regular evaluation can strengthen the capacity of academics and advocates to respond to urgent crises. It can also improve efficiency and effectiveness of the strategies outlined earlier - standard-setting, coalition-building, public education initiatives, and organisational development - which may help to prevent or mitigate future crises and to establish the conditions where academic freedom can flourish.
NOTES
---------------------------------------------------------
Curriculum
