The European Union s ENPI Programme for Georgia. Evaluation of the EIDHR Programme in Georgia Letter of Contract N 2009/226183

Transcription

1 The European Union s ENPI Programme for Georgia Evaluation of the EIDHR Programme in Georgia Letter of Contract N 2009/ FINAL REPORT Prepared by: Pierre Robert Lucy Mincheva Nino Khurtsidze March 2010 The project is financed by the European Union The project is implemented by IBF International Consulting In collaboration with AFC Consultants International 1

2 This report has been prepared with the financial assistance of the European Commission. The views expressed herein are those of the consultant and therefore in no way reflect the official opinion of the Commission 2

3 Acknowledgements This report was written by three independent consultants: Nino Khurtsidze, Lucy Mincheva and Pierre Robert. The consultants would like to thank all those who gave their time and contributed information during the evaluation. They would like to thank in particular the staff of the NGOs visited across Georgia who generously gave their time and shared insights, often at short notice. The consultants also extend thanks to the staff and project beneficiaries who joined activities and briefings, and whose dedication and commitment are essential to the success of the projects. They also thank the staff of the European Union Delegation in Georgia for facilitating the mission and providing a wealth of relevant information and guidance. 3

4 TABLE OF CONTENTS List of Acronyms EXECUTIVE SUMMARY Background Findings Impact on minorities and integration Impact on the fight against torture and ill-treatment Impact on the rights of people with disabilities Impact on democratic election processes Impact on ecological migrants Recommendations to the EU Relevance to conflict context Complementarity with other EU programmes Project design Human rights advocacy/awareness raising Training for NGO representatives Inclusiveness and gender equality Opportunities for collaboration across EIDHR-supported projects Use of operational grants; sub-granting Recommendations on priorities Recommendations to NGOs Beneficiaries buy-in and feedback Engagement with government and other state institutions Coalition-building INTRODUCTION Objectives of the evaluation Scope of the evaluation: EIDHR in Georgia Impact of the Georgian social and political context on EIDHR METHODOLOGY RELEVANCE OF THE PROGRAMME Consistency with EC and government policy Response to needs Quality of the proposals Participation of beneficiaries in project design Promotion of gender equality EFFECTIVENESS AND EFFICIENCY Qualitative aspects Risk management Outreach to stakeholders Project management Examples of best practices IMPACT OF THE PROGRAMME Impact on minorities and integration Impact on the fight against torture and ill-treatment Impact on the rights of people with disabilities Impact on democratic election processes Other areas of impact

5 7 SUSTAINABILITY OF THE PROGRAMME COHERENCE OF THE PROGRAMME Overview EC added value Operational modalities Programming Application process Feedback and backstopping Complementarity RECOMMENDATIONS Relevance to conflict context Complementarity with other EU programmes Project design Human rights advocacy/awareness raising Training for NGO representatives Inclusiveness and gender equality Opportunities for collaboration across EIDHR-supported projects Use of operational grants; sub-granting Recommendations on priorities Recommendations to NGOs Beneficiaries buy-in and feedback Engagement with government and other state institutions Coalition-building ANNEX 1: METHODOLOGY NOTE ANNEX 2: EVALUATION TERMS OF REFERENCE ANNEX 3: LIST OF PEOPLE MET

6 List of Acronyms AEC CSP DCI EIDHR EUD IDP IfS IIZ/DVV ISFED GBP GIP GCRT GoG GYLA MES MoLHSA ngni NGO NMP NSA&LA OPCAT PRI PSA RRM TACIS TI TOR UNICEF Adult Education Centre Country Strategy Paper Development Cooperation Instrument European Instrument on Democracy and Human Rights European Union Delegation Internally Displaced People Instrument for Stabilisation Institute for International Cooperation/German Adult Education Association International Society for Fair Elections and Democracy Georgian Public Broadcasting Geneva Initiative on Psychiatry Georgian Centre for the Rehabilitation of Victims of Torture Government of Georgia Georgian Young Lawyers Association Ministry of Education and Sciences Ministry of Labour, Health and Social Affairs New Georgia New Initiative Non-Government Organisation National Preventive Mechanism Non-State Actors and Local Authorities Optional Protocol to the Convention against Torture Prison Reform International Public Service Announcement Rapid Reaction Mechanism Technical Assistance to Commonwealth of Independent States Transparency International Terms of Reference United Nations Children s Fund 6

7 1 EXECUTIVE SUMMARY 1.1 Background The European Union Delegation (EUD) in Georgia has commissioned a consortium led by IBF International to carry out an evaluation of its European Instrument for Democracy and Human Rights (EIDHR) programme from 2005 to The evaluation took place from January to March 2010, with a field visit in Georgia from 8 to 25 February. The terms of reference (TOR) required a comprehensive evaluation of the impact and sustainability of outcomes of the EIDHR programme in Georgia. The specific objectives of the evaluation were: To assess the programme as whole focusing on outcomes and their sustainability, and on overall programme impact; To provide an assessment of the strengths and weaknesses of the current programming and operating modalities of the Programme and their capacity to achieve stated objectives; To make strategic recommendations aimed at improving impact (e.g. choice of priorities, choice of sectors, operating modalities, etc.), including identification of opportunities for increased complementarity with other EC-funded instruments and programmes. In the period under consideration, the EC supported 28 projects: - Micro-projects, with grants amounting to up to 100, Macro- and targeted projects, with grants normally amounting to up to 1m. Within these, the evaluation focused on a sample of 12 micro- and 3 macro-projects, selected in consultation with the EUD to be representative of the sectors covered and of the geographical spread of activities funded. A list of the evaluated projects is included in the methodological note appended to this report. A note on methodology, appended to this report, was submitted to the EUD in January 2010 prior to the mission to Georgia. This included a discussion of the terms of reference and outlined the evaluation team s understanding of the needs. 1.2 Findings In general, the relevance of the projects was high, in terms of the needs they were responding to and of alignment with EIDHR priorities. The projects largely addressed issues on which current resource allocations were given comparatively low priority or where civil society had a key role to play rather than, or complementary to, government. In general terms, the projects have been relevant to EIDHR priorities and to underlying needs in Georgia. Weaknesses in relevance were related to the following factors: 7

8 Some issues within the mandate of EIDHR were not covered, despite the existence of an underlying need, such as economic, social and cultural rights, including labour rights. The relevance of some projects was hampered by weaknesses in project design, such as lack of realistic objectives and achievable milestones (intermediate results). In some cases correctly identified needs were not directly addressed by project activities, and the project design did not clarify sufficiently what indicators of success would be used. The evaluators found that the projects were generally effective, in the sense that they were on the way to achieving their objectives to an adequate degree. All projects have largely implemented the planned activities. It was difficult to assess the efficiency of the projects in view of the limited time the team could devote to each project visit. However information on attendance and participation suggested that activities reached the intended beneficiaries, despite occasional logistical difficulties. While most projects were able to carry out the activities they had planned, some experienced problems in this respect, which may be ascribed to organisational capacity and risk management difficulties. These challenges could have been anticipated through more thorough project design, and mitigation approaches could have been adopted - such as working with different partners than those selected. Projects generally had an appropriate level of engagement with civil society stakeholders. However, engagement with government seems generally to have been less effective. Some projects have had fruitful engagement with the Ombudsman s Office and with local authorities at various levels, and there were some examples of effective lobbying of parliament and ministries on legislative issues. However, many of the projects lacked a well-designed, concerted advocacy strategy. It was the general view of the team that project budgets were reasonable and consistent with prevalent practices in Georgia and internationally. Despite this generally positive picture, there were two concerns related to financial aspects: Some equipment costs seemed excessive. Conversely, some budgets included unrealistically low staff costs. The team identified some valuable good practices, which seem to have been effective and could be emulated in future projects. These included: Challenging and working with local government; Developing private-public partnerships; Innovative approaches to promote integration Generally, the projects appear to have achieved the expected impact, although they have done so at varying levels. In those cases where impact was the clearest, the reasons were, in essence, the following: 8

9 Appropriate analysis of the situation to be addressed. Appropriate selection of entry points in the exercise of advocacy and/or awareness-raising. Provision, where necessary, of a high level of skills and competence: this enhanced the projects credibility with their target groups Impact on minorities and integration Minorities integration turned out to be the leading area in terms of number of projects funded and activities implemented. The EIDHR programme had successfully addressed various aspects pertaining to the human rights of the minority and ethnic groups in Georgia, and fostered the development of culture of tolerance and respect to diversity. Some good initiatives also addressed the long term issue of integration Impact on the fight against torture and ill-treatment The EIDHR has consistently supported the efforts to fight ill treatment and torture in Georgia through a series of macro and micro projects and longer term support. The few actors working in this area have also been quite successful in ensuring the continuity of their activities either with new EIDHR grants or support from other donors Impact on the rights of people with disabilities EIDHR has rendered support to a number of projects which strengthened the role of civil society in fighting discrimination on any grounds, promoting the rights of vulnerable groups and fostering their integration into society. It has thus facilitated the process of integration of persons with impairments in society and contributed to the improvement of their quality of life and self-esteem Impact on democratic election processes The micro-project Monitoring the Use of Administrative Resources in the 2008 Presidential and Parliamentary Elections helped promote free and fair elections in Georgia, by supporting the processes of democratisation through monitoring, awareness-raising, and empowerment of individuals and civil society to protect their rights and prevent further violations Impact on ecological migrants The EIDHR generated impact in a very specific area, eco migrants, through its project in Ajara Sustainable Development Program for Ajara Ecomigrants (people displaced as a result of environmental degradation). The key added value of the EIDHR programme is its unique focus on human rights and democracy. The EIDHR is unique in that it covers the broad range of human rights, and is framed in terms of support to the implementation of all rights covered in the International Bill of Human Rights. 9

10 1.3 Recommendations to the EU Relevance to conflict context The EUD should ensure that EIDHR applicants that propose to engage in peacebuilding or similar activities, should enhance the conflict analysis which underpins the design of their project, so that activities and indicators of success are clear. To that end, they should confront their analysis with that of other organisations with relevant expertise and/or credible research institutions Complementarity with other EU programmes While the EIDHR programme already operates in a manner that is complementary to other EU funding programmes, complementarity could be enhanced by ensuring that NGOs applying for EIDHR grants are aware of the modalities of support under the Non-State Actors and Local Authorities programme, and can either apply for funding under that programme or develop linkages with organisations funded under that programme. Similarly, linkages should be encouraged between organisations working on IDP issues with EIDHR and IfS funding Project design The EUD should: Explicitly remind applicants for EIDHR funding of the eligibility for EIDHR support of the promotion and protection of labour rights and other economic, social and cultural rights; Closely review the design of projects before signing the grant contacts, to ensure a realistic, appropriately narrow statement of objectives. It is important also to ensure that results, including intermediate results or milestones, are achievable and consistent with the project objectives; Ensure that the advocacy (or campaigning or lobbying) element of projects is explicitly stated, and that it is appropriately designed to enhance the visibility and clarity of expected results. Review project budgets prior to signing grant agreements, to make sure that costs are neither inflated nor underestimated Human rights advocacy/awareness raising The EUD should amend the guidelines to applicants to remind them of the importance of the advocacy element, and/or awareness raising of human rights to the fulfilment of EIDHR criteria. All projects should enhance the element of advocacy for human rights, particularly in relation to authorities (local at all levels, and national) and in relation to stakeholders able to exercise an influence on the situation they are dealing with (such as opinion leaders, etc). 10

11 1.3.5 Training for NGO representatives The EUD should continue its practice of training NGO representatives on financial reporting issues. However it should also consider expanding the training to other issues, in particular: Project design, to ensure that NGOs develop project proposals with more realistic objectives, results and milestones, and acquire better project management skills. Logical framework, to ensure that NGOs understand better the rationale of using log frames, and the way a log frame can be used in monitoring progress and identifying implementation challenges. Advocacy and lobbying (and more generally campaigning techniques) to ensure that the advocacy component of projects is more explicitly set out and that organisations acquire the necessary understanding of what is required for effective government lobbying Inclusiveness and gender equality The EUD should amend the guidelines to applicants to ask them to specify how they will ensure that people from a range of backgrounds are involved in the project as beneficiaries, participants, staff and managers. In addition, the guidelines should require applicants to demonstrate how they ensure equal gender representation at all levels of their organisation Opportunities for collaboration across EIDHR-supported projects The EUD should consider ways of encouraging collaborative activities between NGOs implementing projects with EIDHR funding, including the possibility of establishing a small fund to support such activities. In particular, collaboration should be encouraged in relation to advocacy and lobbying, between organisations with similar or complementary objectives, and bring together organisations with a credible track record on specific concerns and organisations with expertise in the legislative and policy-making processes Use of operational grants; sub-granting The EUD should review the possibility of allocating operational grants in certain cases, where they can play a significant role in reinforcing the sustainability of beneficiary organisations. The EUD should also review the possibility of allowing grant beneficiaries to provide and administer sub-grants of less than 10,000. This could help small organisations without increasing the transaction costs at EUD level (provided the recipient of the main grant took responsibility for the sub-grant s oversight) Recommendations on priorities In the context of the forthcoming EIDHR Call for proposals, the evaluation team recommends that the EUD should take account of the following recommendations for thematic priorities: Integration. EIDHR projects targeting integration have demonstrated some effectiveness in relation to vocational training and mediation between minority groups 11

12 and mainstream authorities. Projects supporting cultural activities could effectively prolong these approaches, especially as instruments to support integration processes. Children s rights and juvenile justice. There appears to be an unmet need to address the prevention of social problems in a school setting (corresponding to the number of children s rights), as are: - Primary prevention of mental health problems; - Primary Prevention of drugs misuse; - Prevention of bullying in public schools and school violence. Human rights promotion, advocacy and education. EIDHR should continue its emphasis on human rights as it is one of the few, and to our knowledge the most effective, instrument that allows NGOs to independently work in the sphere of human rights. Within this, the EUD should consider highlighting to applicants the broad range of human rights, including economic, social and cultural ones - labour rights in particular. EIDHR-funded NGOs have identified the following needs in relation to human rights support: - IDP rights - Access to Justice in environmental matters - Freedom of expression - Combating torture and ill-treatment - Combating domestic violence - Work on psycho-social rehabilitation of people traumatized as a result of natural or man-made disasters - like war, displacement, earthquake, violence, etc. 1.4 Recommendations to NGOs Beneficiaries buy-in and feedback Organisations should systematically seek feedback from beneficiaries on the design and implementation of the projects (especially those involving training in language or skills), and regularly inform them of developments, achievements and challenges. This would enhance accountability and effectiveness Engagement with government and other state institutions As part of the stakeholders analysis and project strategy, applicants should be required to specify how they will engage with government authorities at all appropriate levels, and with state institutions relevant to their field of activities. 12

13 1.4.3 Coalition-building NGOs should, as a matter of course, systematically consider building coalitions to apply for EIDHR funding, ensuring that coalition members have complementary skills and experience. This approach could enhance in particular the advocacy aspect of projects. 13

14 2 INTRODUCTION The European Union Delegation (EUD) in Georgia has commissioned a consortium led by IBF International to carry out an evaluation of its European Instrument for Democracy and Human Rights (EIDHR) programme from 2005 to A team of three independent consultants has prepared the present draft report on behalf of the consortium. The evaluation took place from January to March Following a short preparation period, an inception note, appended to this report, was submitted to the EUD in January. A field visit was conducted in Georgia from 8 to 25 February, concluding with a workshop organised by the consultants, bringing together representatives of nongovernmental organisations (NGOs) supported by EIDHR in Georgia, as well as Delegation staff members and representatives of other donor institutions. 2.1 Objectives of the evaluation The terms of reference (TOR) require a comprehensive evaluation of the impact and sustainability of outcomes of the EIDHR programme in Georgia in the period , in order to provide the Delegation with recommendations concerning strategic programming and operational choices for EIDHR in future. The specific objectives of the evaluation are: To assess the programme as whole focusing on outcomes and their sustainability, and on overall programme impact; To provide an assessment of the strengths and weaknesses of the current programming and operating modalities of the Programme and their capacity to achieve stated objectives; To make strategic recommendations aimed at improving impact (e.g. choice of priorities, choice of sectors, operating modalities, etc.), including identification of opportunities for increased complementarity with other ECfunded instruments and programmes. 2.2 Scope of the evaluation: EIDHR in Georgia According to the EUD, EIDHR has supported about 120 projects in Georgia since its introduction in According to the EC Cooperation Report for Georgia (2005), the EIDHR allocation between 2000 and 2005 was about 2m/year. According to the Georgia Country Strategy Paper (CSP) for , Georgia was an EIDHR focus country from 2002 to 2004, with projects implemented in fields such as the fight against torture; combating discrimination; rule of law and democratisation/elections. That CSP foresaw the use of EIDHR in particular in the fields of governance/rule of law and conflict prevention and resolution, complementing programmes under Technical Assistance to the Commonwealth of Independent States (TACIS) for the period. Following the 2003 Rose Revolution (see below) the EC also implemented programmes under its Rapid Reaction Mechanism (RRM), particularly in relation to 14

15 rule of law and elections preparedness and monitoring fields also related to the mandate of EIDHR. In the period under consideration, the EC supported 28 projects, which fell under the following categories: - Micro-projects, with grants amounting to up to 100, Macro- and targeted projects, with grants normally amounting to up to 1m. The evaluation focused on a sample of 12 micro- and 3 macro-projects, selected in consultation with the EUD to be representative of the sectors covered and of the geographical spread of activities funded. A list of the evaluated projects is included in the methodological note appended to this report. As indicated above, the main objective of the evaluation was essentially to assess the impact of the EIDHR programme as a whole. This was done by drawing information obtained from the study of the 15 projects in the evaluation sample. 2.3 Impact of the Georgian social and political context on EIDHR EIDHR has been used in Georgia initially to contribute to the development of civil society, at a time when the government, then led by President Eduard Shevardnadze, was stating its support for democracy but was widely seen as failing to follow through on its rhetoric. In its CSP, the EC highlighted continuing human rights violations, such as widespread torture in pre-trial detention, criminal procedures inconsistent with international human rights standards, and violent assaults against members of religious communities. These concerns, compounded by weak rule of law and pervasive corruption, led to such political and social instability - and economic stagnation - that some observers were referring to Georgia as a failed state. The increasing instability also led to civil unrest, which culminated on the occasion of the November 2003 parliamentary elections (the period of the Rose Revolution ). Protests against the fraudulent elections led to the resignation of President Shevardnadze and to the presidential elections of January 2004, which were won by Mikheil Saakashvili and were followed by new parliamentary elections in March, which gave a strong majority to President Saakashvili s supporters. Mikheil Saakashvili won a new five-year term in January 2008 when elections were held ahead of their normal term as a result of public protests against his government in late The government stemming from the Rose Revolution has taken numerous steps in the years following its assumption of power to address major human rights concerns. These included reforms concerning the fight against corruption, legislation on legal aid, integration of ethnic minorities and support to returnees from forced exile. The police forces were reformed, leading to a dramatic improvement in the image of the police and to a significant reduction in complaints of ill-treatment in police custody. However concerns remain in relation to the protection of human rights and the development of a democratic society. Key areas of continuing concern, as expressed 15

16 by international human rights organisations and the Council of Europe, include the following: Protection of civilians in conflict zones. Amnesty International reported allegations that military operations during the August 2008 conflict with Russia in South Ossetia were not conducted with appropriate precautionary measures. Internally displaced people (IDPs), mostly former residents of South Ossetia and Abkhazia displaced by the successive conflicts with Russia, are in continuing need of support. Although the Georgian authorities have provided IDPs with new housing and other support, their livelihoods remain insecure, as is their access to social services. While conditions in police custody have improved markedly, NGOs remain concerned about conditions in prisons, particularly for vulnerable groups such as juveniles. Media freedom also remains a concern: although information and news can be found in an increasingly lively Internet-based scene and the printed media does enjoy significant freedoms despite their critical reporting, TV media continue to experience difficulties from time to time, not only through government attempts to curb reporting, but also from opposition political forces. The Council of Europe, in a resolution in January 2008, called on the Government of Georgia (GoG) to implement further reforms in relation to eradicating corruption, reforming the penal code and criminal procedure legislation, and enhancing the fairness of the electoral system. Some of these reforms are in process, in particular concerning criminal laws and procedures. 16

17 3 METHODOLOGY A note on methodology, appended to this report, was submitted to the EUD in January 2010 prior to the mission to Georgia. This included a discussion of the terms of reference and outlined the evaluation team s understanding of the needs. To complement the methodology, the following points provide an overview of the team s approach: Evaluation criteria. In keeping with the TOR the evaluation considered impact and sustainability issues. However it also considered the relevance and effectiveness of the programme, as well as some management aspects falling under the criteria of efficiency. Sources of information. The evaluation primarily used information from project documents and from its own investigations. Project documents included project descriptions agreed at the time grants were given (project outlines annexed to grant contracts, logical frameworks, budgets) as well as interim and final reports and, where available, evaluation reports. In addition each of the organisations was asked to respond to a questionnaire (available in English or Georgian). That questionnaire was also used as a broad interview guide during meetings and field visits. Data collection methods. The questionnaire used is appended to this report. In addition, interviews were held where possible with project beneficiaries, who were asked to describe the difference, if any, that the project made to them. The team also met representatives of other donor agencies implementing programmes in areas similar to those covered by EIDHR. The objective of these meetings was to compare the approach of the EC with that of other donors to draw lessons for future practice. Reference indicators. The key indicators of achievement used by the team during its consideration of the projects were those provided under each project s logical framework. In addition, when looking at the achievements and shortcomings of the programme as a whole, the team relied on EC guidelines and legislation concerning EIDHR. Key limitations. It was not the objective to carry out 15 distinct project evaluations, and the team had neither the resources nor the time to do so. As a result, it could not carry out its own investigation into the achievements and shortcomings of individual projects, and largely relied on the accounts given on these points by representatives of the organisations concerned. As in any sampling approach, the team assumes, but cannot be certain, that the fifteen projects it selected were representative of the overall programme. Stakeholder participation. The team was able to meet a wide range of organisations operating from the capital Tbilisi as well as from key regional centres. Its meetings with other donor agency representatives indicated that the range of projects under consideration was consistent with the range covered by other donors active in similar fields. The team was not able to meet with beneficiaries for most of the projects, due to time constraints, but it did take into account publications and other documents (DVDs, published 17

18 statistics, etc), which gave a broad sense of any changes in the situation of beneficiaries in many of the projects concerned. 18

19 4 RELEVANCE OF THE PROGRAMME In general, the relevance of the projects was high, in terms of the needs they were responding to and of alignment with EIDHR priorities. The projects largely addressed issues on which current resource allocations by the GoG and donor governments were given comparatively low priority (as in the case of support to people with disabilities and eco-migrants displaced as a result of environmental degradation) or where civil society had a key role to play rather than, or complementary to, government (as is the case of support to media freedom, monitoring of transparency, etc). 4.1 Consistency with EC and government policy All the projects fell within the approach set by the Georgia CSP, which highlights the priority given by the EC to supporting the promotion and protection of human rights. On the other hand, there were no clear links between the projects and policies of the GoG: when projects aimed at influencing GoG policy on a particular issue (media freedom, integration of minorities, education) it was often unclear what aspect of government policy the implementers expected to see change: many could not say clearly whether they were targeting a particular law, for example, or whether they were working towards a more general change in attitudes. The relative lack of clarity in this regard is explained partly by the small scale of the projects, and partly in some cases by the sensitivity of the human rights issues addressed by them. It is also to be noted that the EIDHR is an EU funding instrument that does not involve consultations on its use with the host government. This is not to say that the programme is entirely separate from actions undertaken by the GoG. Many of the projects cooperate with Georgia s Public Defender s Office in areas such as juvenile justice, integration of ethnic minorities and freedom of expression. Several project implementers also work with local authorities, for example on the integration of people with disabilities and on social inclusion of members of minority groups. As a result, it can be concluded that the projects were consistent with EIDHR priorities, and therefore also with EU strategy in Georgia as expressed in the successive strategy documents. However, while the projects were aligned with priorities, neither the sample of 15 projects nor the other projects funded during the period covered the entire range of rights included in the scope of EIDHR. While civil and political rights and issues related to governance were covered by funded projects, there were almost no projects covering economic, social and cultural rights. In particular, labour rights and health were not addressed in the range of projects (except, in the case of health, incidentally as part of projects focusing on other issues). It might be appropriate in future EIDHR calls for proposals to include the protection and promotion of economic, social and cultural rights as part of the priorities, in particular with regards to labour rights. 4.2 Response to needs In addition to alignment with EIDHR priorities, relevance can also be assessed in terms of the projects response to underlying needs in Georgia. In this respect relevance was 19

20 generally adequate, though sometimes weak. It was adequate because all the projects sought to address issues of democracy and human rights where the current situation in Georgia is not fully satisfactory, and where civil society input can help make a difference. However the relationship between the identified need and the approach taken by the projects was sometimes unclear. For example a project focusing on youth participation through civic education correctly identified a need for more civic education of young people - but it was unclear how the activities were supposed to contribute in any significant way to respond to this need. Another project based outside Tbilisi, while broadly consistent with the identified need to improve governance, was of such a general nature that it was difficult to clarify how it could bring specific benefits to citizens and build the capacity of the target institutions. 4.3 Quality of the proposals The project outlines that form the basis of the grant contract agreements between the EC and implementers were of unequal quality. While some outlines were well designed and strategically sound, others presented significant design weaknesses. These often led to difficulties in subsequent implementation and to misunderstandings among stakeholders. In theory, weaknesses in project design should be identified at proposal approval stage, and addressed during the grant contract finalisation stage. In practice, however, there was not enough time for the EUD to do so, because it had to deal with a complex approval process, involving liaison with many grant applicants at the same time and with Brussels, within a relatively short period. The key weaknesses of the proposals were the following: Goals were often too general and vague, not always clearly connected to activities. For example one project sought to increase media freedom and provide the public with information about corruption in Georgia. However, the planned activities concerned only the second of these objectives, and the proposal did not say how the production of information about corruption would enhance media freedom. In another example from a macro project, the project outline had no less than four major objectives: to strengthen the existing NGO network contributing to the rehabilitation of torture victims; to increase the effectiveness of services to such victims; to implement international standards on the prevention of torture; and to enhance the capacity of the beneficiary and its partner NGOs. These four objectives were not realistic: some were not under the beneficiary's control (an NGO cannot implement international standards). Others were not compatible with the amount of resources available and the size of the tasks at hand. The relationship between goals and actual activities was often unclear because applicants were insufficiently familiar with the distinctions between goal, objectives and activities that the logical framework approach mandates. Widespread failure to build action capacity through meaningful coalitions. Many of the grantees were relatively small and specialised organisations. They had genuine expertise and credibility in their core area of work, but many tended to underestimate the expertise needed to deliver on some aspects of their proposals. For example, a project based outside Tbilisi, which planned to provide legal advice to repatriates did so, putting beneficiaries in touch with some lawyers. However the quality of the advice may not have been commensurate with repatriates expectations and needs, particularly in relation 20

21 to the complex property and citizenship issues at stake. In addition, the organisation itself did not have the capacity to conduct the sustained lobbying in Tbilisi that was necessary to increase the likelihood that the law on repatriation was amended. Its project would have gained in relevance and effectiveness if an appropriate coalition had been set up with a relevant legal and advocacy organisation, able to interact on an on-going basis with representatives of the GoG and Parliament. 4.4 Participation of beneficiaries in project design None of the projects substantively involved future beneficiaries at the project design stage. In each proposal, the project design was developed on the basis of the applicants past experience with similar target groups. As a result, while it cannot be said that beneficiaries were explicitly consulted, lessons from previous similar projects were taken into account in project design. The projects appear generally to have been responsive to beneficiaries needs during implementation. 4.5 Promotion of gender equality Gender seems to have been an important component of EIDHR projects over the period Five micro projects supported under this programme dealt specifically with promoting gender equality. Furthermore, among the 15 projects reviewed by the team, many have been able to successfully address various aspects of women s rights, including the cultural stereotypes that are impeding gender equality. The project Folk High Schools in Samtskhe-Javakheti a Chance for Integration of Minorities was able to generate tangible impact in this area by achieving integration of women and girls in social and cultural life. Through the Adult Education Centres of Akhaltshikhe and Akhalkalaki, women and girls were able to improve their knowledge and skills and in this process become more integrated in the life of their community, which is a significant achievement given the conservative and patriarchal traditions of the remote and strongly Armenian communities in the region. Much like this macroproject, the micro-projects reviewed by the Team did not have a separate gender component in their activities. However, some addressed this issue through partnering with women s rights NGOs and others did so through making special efforts to reach women beneficiaries. Another EIDHR macro project reviewed by the team Fostering Civil Integration through Education and Freedom of Expression did have a separate gender component, which tried to address women s rights issue from two fronts. One was society s approach to women and another is empowerment of women so that they are more proactive in realizing their modern role and opportunities. The most notable achievement of the project was facilitating an agreement among experts and NGOs working on gender issues to come together under the umbrella of the Civil Lobby on Gender, in order to jointly advocate for gender equality. 21

22 5 EFFECTIVENESS AND EFFICIENCY The evaluators found that the projects were generally effective, in the sense that they were on the way to achieving their objectives to an adequate degree. All projects have largely implemented the planned activities: where delays have occurred they were manageable and in those cases the implementing NGOs were actively working on plans to catch up and implement all planned activities during the project period. It was difficult to assess the efficiency of the projects in view of the limited time the team could devote to each project visit. Information on attendance and participation suggested that activities reached the intended beneficiaries, despite occasional logistical difficulties. 5.1 Qualitative aspects Whereas the projects have been effective in terms of implementing planned activities, the evidence about the quality of these activities was contrasted and of varying nature from one project to another. While this made it difficult to give a synthetic assessment, the view of the evaluators was that quality was generally good. For example, the evaluators were told about the contents of training modules implemented by some of the organisations which appeared to be of appropriate quality, and delivered by appropriately trained personnel. Some of the organisations also appeared to have invested in training their own staff, which had a positive effect on the delivery of results. Thus, although the evaluators did not witness many activities, they believe that the projects have been implemented to a satisfactory degree of quality, sometimes to an outstanding degree. This view is, to some extent, lent further support by the assessment of the projects impact (see below). 5.2 Risk management While most projects were able to carry out the activities they had planned, some experienced problems in this respect, which may be ascribed to organisational capacity and risk management difficulties. For example, one project was not able to implement training sessions on HIV and drugs for law enforcement officers, apparently because of lack of access to appropriate government departments and levels. This component had to be removed from the project. In hindsight, the difficulties experienced were predictable, and might have been avoided through a more appropriate choice of partner. Similarly, a project which had planned to target members of the judiciary with information about corruption and lobbying on freedom of the media, did not have the lobbying expertise, contacts and track record that would have made such high-level advocacy possible. In effect, this part of the project was shelved. Both of these examples indicate that some implementing organisations faced difficulties related to unforeseen practical implementation challenges. However, these challenges 22

23 could have been anticipated through more thorough project design, and mitigation approaches could have been adopted - such as working with different partners than those selected. 5.3 Outreach to stakeholders The issues of gender and inclusiveness highlighted in the relevance section above are also important to the projects effectiveness. Some projects have made specific efforts to address particularly poor and vulnerable individuals within their target groups. Projects generally had an appropriate level of engagement with civil society stakeholders. Indeed, engagement with other civil society actors is generally of a high standard in all cases: schools, local communities, religious groups, etc, were involved appropriately to each project. In some cases the EIDHR-funded project is, in effect, helping other projects through coordination and exchange of information - or simply by providing a convenient meeting place. However, engagement with government seems generally to have been less effective. Some projects have had fruitful engagement with the Ombudsman s Office and with local authorities at various levels, and there were some examples of effective lobbying of parliament and ministries on legislative issues. However the successes in this field were relatively limited. Reasons for this appear to have included the following: Many of the projects lacked a well-designed, concerted advocacy strategy. Project implementers seemed to have assumed that their activities would generate enough publicity or visibility for their concerns to influence decisionmakers without recourse to such a strategy. This, in practice, was not the case: contrary to this assumption, effective advocacy of government requires specific materials to be produced and presented to authorities. The implementing organisations did not always have the relevant advocacy or lobbying skills. In addition to thorough understanding of the issue at stake (and track record of addressing it, which many NGOs have), effective advocacy requires a good understanding of legislative and policy-making processes. Many NGOs lack experience in these fields. To address these shortcomings, NGOs should include in project proposals explicit advocacy strategies that take account of their organisational strengths and weaknesses. Furthermore NGOs should be more pro-active in developing advocacy coalitions bringing together organisations with a range of different skills, including the capacity to analyse government policy, contribute to legal drafting and address legislative organs as well as individual government departments at appropriate levels. Such coalitions would help circumvent the relatively weak organisational capacity of many Georgian NGOs, and would ensure that the different aspects of advocacy and lobbying (from selection of target audiences to development of messages, campaigning techniques and follow-up) are carried out by appropriately skilled actors. In addition, NGOs should consider seeking the support of pro bono experts in such fields as law and training. 23

24 5.4 Project management It was not the objective of the evaluation to consider project management in detail. The team did not attempt to audit or otherwise assess the sincerity of financial reports provided by the implementing organisations. However, it was the general view of the team that project budgets were reasonable and consistent with prevalent practices in Georgia and internationally. One reason why the NGOs were able to present reasonably tight budgets was that some of them at least were able to spread project management costs across different projects. Another factor in the relatively low project budgets is that (as mentioned above) several of the projects were the continuation or reiteration of approaches already tested by the implementing organisations, thus reducing the start-up costs. Despite this generally positive picture, the study of some individual project budgets provided a reminder that the EUD should carefully review project budgets prior to the signature of grant agreements. The evaluators found an instance of apparent overestimate of computer costs, and another instance of apparent underestimate of salary costs: these issues should be identified at an early stage to avoid difficulties in project implementation and reporting. In other respects, project management seems to have been of good quality, as indicated by the level of detail given in interim narrative and financial reports. Many NGOs have noted that the strict financial reporting requirements of the EC are sometimes difficult to follow, but the reports seen by the evaluators suggest that the requirements were generally met satisfactorily. Other than reports to the EC, information appears to have circulated appropriately within the NGOs themselves, and staff met by the evaluators were aware of the projects, the EC s funding, and of activities carried out. A further sign of the quality of project management was that all the project managers could provide satisfactory answers to the evaluators on most aspects of the projects, and had a clear understanding of the strategic approach and risks faced by their projects. While some projects missed some reporting deadlines, the delays were manageable and were explained by managers workload. The evaluators heard no reports of significant staffing problems related to the projects. One area where there was unmet demand for more project-related information concerned EIDHR micro-project NGOs themselves. NGOs praised the opportunity given to them by the EC to meet each other to discuss issues of interest which they had done on some occasions before the meeting organised by the evaluators. Some (not all) NGOs felt that there was scope for EIDHR-funded NGOs to cooperate with each other, either in the context of the current projects or otherwise. In the evaluators view, the effectiveness of projects might be enhanced if the EIDHR-funded NGOs could carry out joint work on specific areas. For example, synergies could be found, to carry out joint public advocacy and government lobbying work, among EIDHR-funded NGOs working on similar issues (such as those related to juvenile justice and other aspects of criminal justice) or complementary ones (such as media freedoms and government transparency). 24

25 However, the EUD should exercise care when encouraging synergies among organisations it supports: while there is no reason not to encourage coordination amongst EIDHR-funded NGOs, the EC should ensure that it does not give rise to the assumption that further funding will only be available to NGOs that cooperate in that way. It is also important to avoid perceptions that EIDHR-funded NGOs constitute a club in which new entrants would be unwelcome. To dispel any such perceptions, it is important for the EUD to continue organising information sessions on EIDHR open to all NGOs. The issue of enhancing cooperation among beneficiary NGOs is discussed further in the section on coherence, below. 5.5 Examples of best practices The following are examples of approaches used by projects, which seem to have been effective and could be emulated in future projects: Challenging and working with local government - Rather than providing direct financial support to the families who suffered from environmental degradation, one of the beneficiaries aimed at changing state policies toward this vulnerable group, which resulted in increased sustainability. As a result of consistently challenging the local authorities to address the grievances of those affected by erosion, raising awareness of citizens rights and working with the Ajaran government to devise solutions, the project was able to institutionalise support for the ecomigrants and has ensured that the impacted community is not splintered when resettlement takes place. Furthermore, this approach had a multiplier effect in that it contributed to the government devoting sizeable amounts of state funds to alleviate the socio-economic plight of ecological migrants and led to the development of a state-funded mechanism to assess coastal erosion problems and devise solutions. - Similarly, a project implemented in Zugdidi challenged the local government to make public buildings handicap accessible and managed to achieve a measurable impact through working with the authorities. Developing Private-Public Partnerships - To achieve project objectives, one beneficiary maintained a successful referral system, where the Ministry of Justice Department for Probation and Non-Custodial Punishment, Prisons Republican Hospital, and medical units in partner prisons worked with the NGO to assist the prisoners or probationers in receiving the needed services. Medical hospitals, private doctor practitioners, and NGOs provided needed medical, psychological, and legal services to those who needed. This has proven to be a successful private-public partnership which could be duplicated in other projects. 25