Save the Children Somalia Karkaar Model Primary Education Project Terminal Evaluation

Save the Children Somalia / Somaliland Programme

Karkaar Model Primary Education Project Terminal Evaluation

Project Terminal Evaluation

Terms of Reference (TOR)

For Final Evaluation of Developing a Local Model for the Delivery of Primary Education in Karkaar Region of Puntland

1. Introduction

1.1. Education Situation in Puntland/Somalia

Somalia has one of the lowest enrolment rates of primary education in Sub-Saharan Africa. The gross enrolment rate (GER) in Somalia stood at only 30% at the primary level in 2006/07, with 36.6% for boys and only 24.6% for girls (UNICEF PES, 2006/2007).

The education system exhibits very low access opportunities and poor quality of education at all levels, with striking disparity between male and female enrolment rates, among administrative regions, and amongst various social and economic groups. Communities such as minorities, pastoralists and rural settlers, internally displaced people, and returnees are highly marginalized by the system.

Relevance of the education system to the dynamic realities of the different regions is also low.

The prolonged civil war has not only destroyed what had once existed but undermined the resource base of the state to address the growing education needs of its citizens. Limited state resources, lack of relevant policies and poor implementation of the existing policies, limited human resource capacity, meagre incentives to children’s education are some of the root causes of the failing education system.

The Karkaar primary education project aimed to develop a replicable model for the provision of quality primary education in Somalia at the regional level. This was achieved by building the capacity of existing Government and community education management structures in the Puntland Region of Karkaar.

With improved capacity and systems, partners would be enabled to deliver the highest possible quality of primary education in such a highly resource-constrained setting. Critically, the project
would also establish mechanisms to lever greater external funds into education at the District and community levels.

1.2. Developing a Local Model for the Delivery of Primary Education in - Karkaar Region, Puntland, Somalia

1.2.1. Project Purpose

This project purpose is to develop a local model for the delivery of primary education in Somalia, which contributes to the overall goal of increasing the number of children accessing and completing inclusive, quality and protective basic education in Puntland. This project works with the Puntland Ministry of Education (MoE) and all regional stakeholders in education in Karkaar to strengthen the provision of quality primary education.

This includes improving the infrastructure and learning environment of existing primary schools; building the capacity of and linkages between the Regional and District Education Offices as well as the Community Education Committees (CECs) attached to each school.

The project also aims to increase the funding available for education from a range of sources
(communities, governments and donors).

1.2.2. Specific Purpose

The specific purpose of the project is to support the Puntland education authorities and communities to establish a regional level model of good practice in the delivery of quality primary education. The project supports and complements other MoE and SC education activities and programmes in Puntland.

1.2.3. Planned Results of the Project

* A 120% increase in the number of children in Karkaar region accessing and completing quality basic education
* The capacity of Education Authorities and Communities to manage and monitor the delivery of quality primary education is strengthened
* A protective environment is in place in all SC supported schools
* Resources available for basic education in Puntland increased

1.2.4. Project Beneficiaries

Direct Beneficiaries:

* An additional 7,437 school aged children (45 % girls) who will gain access to primary education.
* Approximately 20 education staff and officials in the Regional and District Education Offices in Karkar.
* Approximately 460 members of the 66 CECs currently active in Karkar Region
* Approximately 2,350 pupils and teachers in at least 25 schools that will benefit from new and/or renovated school buildings and premises.
* Approximately 2,350 pupils who will have access to sports and other equipment for use in school based clubs

Indirect Beneficiaries:

* The estimated 13,635 (1) primary school children enrolled in schools in Karkaar, 460 CEC members

(1) Figures based on the current figures (2006) in primary school in Karkaar increased by 120%

1.3. Implementing Partners

The project was implemented by Save the Children in partnership with the MoE of Puntland, Karkaar Regional Education Office, and District Education Offices of the five districts of Karkaar region and CECs of schools included in the project.

UNICEF has been technically supporting SC and the MoE in addition to the financing of the project.

2. Purpose and Objectives of the Evaluation

The evaluation is intended principally for learning and accountability purposes. It is expected to
generate relevant findings, lessons, and recommendations which will be shared with key stakeholders of the project and used by the implementing agencies to guide and inform future similar projects and programs in Puntland and elsewhere.

The evaluation will assess the performance of the project against key parameters including the project’s relevance, effectiveness, efficiency, sustainability, timelines of activity implementation, and its strengths and weaknesses.

In addition, the strengths, weaknesses and challenges during implementation will be analysed with a view to extract lessons and best practices for the future. This information will be shared with key stakeholders of the project, most importantly UNICEF, MOE, local education authorities, and other actors in Puntland.

2.1. Objectives

2.1.1 The 7 standard OECD and UNICEF evaluation citeria will be used to assess the performance of the project.

These include

* Efficiency
* Effectiveness
* Impact
* Relevance
* Sustainability
* Coverage and
* Coherence

2.1. 2 To identify the strengths and weaknesses of the project and come up with lessons and
recommendations to guide and inform future programme work with a special focus on the delivery of Alternative Approaches to Basic Education (AABE), formal, adult education (various modes of delivery) and capacity building component at national, regional and local levels.

2.1. 3 Examine and give recommendations on the future expansion of quality primary education
taking into consideration the national policies and priorities and relevance of education to child
development and child rights and needs

3. Scope of the evaluation

The evaluation will cover the accomplishment of all the expected results as outlined in the project document and detailed in the annual work plans during the period of 1st April 2008 – 30th August 2011. All the different project components and activities stated in the approved logical framework matrix will be assessed.

3.1 Evaluation Criteria

The evaluation will concentrate on the five evaluation criteria elaborated below, namely; relevance, efficiency, effectiveness, impacts, sustainability, coverage and coherence of the project.

i) Project Preparation and Design

* How appropriate was the project design?
* What preparatory activities were carried out?
* Were they relevant?
* Was there any outstanding omission in the design?
* Were they reflected in the final project design?
* Were any necessary preparatory activities omitted?
* Was the initial intervention plan/logic appropriate?
* Recommendations for improving future project design/s.

ii) Relevance: The main focus will be on the appropriateness of the project’s concept and design to the improvement of access to education in Karkaar region of Puntland. In particular the:

* Extent to which the stated objectives addressed the problems of the primary target groups re: (school children, MOE, and schools/ABE centres, REOs, DEOs)
* Relevance of the project design within the framework of the MOE programmes and policy guidelines
* Relevance of the project design within the framework of the UNCRC principles.
* Achievements of the project against the original results (outcomes), outputs, and activities specified in the project log frame.
* Level of significance of the results to address the primary needs of the beneficiaries.
* Level of awareness of the children, implementing partners and other relevant stakeholders on the results of the project.
* Positive/negative changes, intended/un-intended that the project produced on the target groups both past and present

iii) Efficiency: The main focus will be on how well the project activities transferred the available resources into the intended results in terms of quantity, quality, and timeliness through sustainable and participatory processes.

In particular:

* How economical were project inputs (funds, expertise, time, and other resources etc) converted into results in the required quantity and quality and in good time?
* Was the use of the project resources cost-effective?
* Could the project activities have been done more efficiently?
* Whether similar results or more could have been achieved at a lower cost in the same amount of time?
* Whether project activities were done right i.e. on time, in expected quantity and quality, and through participatory processes.
* General implementation and management of the project in terms of quantity, quality, and timeliness of inputs and activities, adherence to work-plans, action-plans, logical framework, and budgets.
* Adequacy of management arrangements as well as monitoring and evaluation, technical backstopping support given to the project by all parties concerned coordination and communication.
* Institutional set-up through the Central Ministry, the Regional Education Office, District Education Offices, and the schools , and the degree to which the project has encouraged full involvement of all the stakeholders;
* Responsiveness of project management to changes in the environment in which the project operates;
* • Co-operation among project partners (SC, UNICEF, and the MOE) and other key stakeholders in achieving project results.

iv) Effectiveness: The main focus will be on the extent to which the project achieved its stated results and purpose in a sustainable way. In particular:

* The progress made in achieving the results of the project at all levels. The project Logical Framework will be used as the basis for analysis and assessment; whether the intervention logic actually worked or not.
* Efforts made in capacity building of the MOE, REO and other local project stakeholders and whether the strategies worked or not, and why.
* Project’s management of risks taking into account the stated risks and assumptions.
* Adequacy of management arrangements as well as monitoring and evaluation, technical backstopping support given to the project by all parties concerned; coordination and communication.
* Responsiveness of project management to changes in the environment in which the project operated.
* Co-operation among project partners (SC, UNICEF, MoE, REOs, CECs and other key stakeholders in achieving the project results.
* The degree of support given by the Ministry of Education in integrating the project objectives and goals into the National Education Policy and the National Legal Framework and long/short term strategy plans, and how well the project fits into the national goals of education
* The level of ownership of the project by the MOE and its commitment to support the project after the expiry of the project period.

v) Implementation Process

* To what extent were the activities of the project implemented in participatory and empowering manner? How involved were the key stakeholders of the project in planning and execution of the project activities, steering the project, and in monitoring of the project?
* What worked well?
* What did not work well?
* What should have been done differently?

Vi) Gaps in the implementation of the project will also be analysed

vii) Sustainability: The main focus will be on whether the outcomes and changes brought about by the project are likely to continue after the end of the project and whether they can be sustained at the sector level and in each implementation region (Somaliland and Puntland).

In particular the:

* Extent to which the MOE, REOs, children and other key stakeholders own the objectives and achievements of the project.
* Policy support available to the project from the MoE to sustain the achievements of the project.
* What sustainability measures have been put in place?
* Institutional capacity of the MoE and its local structures to continue the project activities after the end of the project, including the good will and financial and technical capacities.
* Conduciveness of the socio-cultural factors to the continuity of the project activities especially those that relate to support to the girl child

viii) Impacts - The main focus will be on whether the project has contributed to the overall project goal, specific changes that the project has brought about in the lives of target groups (teachers, children, etc), and impacts realized or likely to be realized in relation to cross cutting issues such as gender, environment and child protection. In particular;

* What sustainable changes – positive/negative, intended/un-intended – did the project produce on the target groups – (past and present) as well as on the MOE, children, CECs, REOs, the general community and other stakeholders?).
* Achievements of the project against the original results (outcomes), outputs, and activities specified in the project log frame.
* Level of awareness of the stakeholders, particularly the MOE, REOs, DEOs, DEBs, CECs, teachers/head teachers and children.
* Level of ownership of the project by the education Authorities (MOE) and their commitment to support the project after the expiry of the project period.
* Degree of support given by the Ministry of Education in integrating the project objectives and goals into the National Education Policy and long/short term strategy plans, and how well the project fits into national goals of education
* Extent to which the project has impacted on teaching and learning.
* Project impact on enhancing co-operation among education sector and project partners.

3.2 Evaluation sites: The project was implemented in Karkar region of Puntland (all the 5 districts of the region).

The evaluation team will select sample districts, schools, and communities from the project region in consultation with relevant stakeholders and using statistically acceptable parameters and select offices/places of various stakeholders identified as sources of information.

3.3 Specific Tasks: Working closely with the Evaluation Team (ET), the evaluator will undertake the following tasks:

* Develop evaluation framework and methodology and refine these with the project team.
* Develop a sampling frame that will be discussed and adopted by the SC ET
* Develop the evaluation plan
* Undertake extensive document review to familiarize with the project
* Assess the capacity development interventions the project has undertaken with the MOE and schools.
* Prepare the evaluation draft report and present the findings in workshops in Garowe (Puntland) for key national and international agencies operating in the zones;
* Prepare and submit final report to the Lead Agency in three hard copies (bounded) and CD-ROM.
* Present the findings of the evaluation to the ESC members

4. Evaluation Approach

The evaluation will be undertaken through a combination of qualitative and quantitative techniques that will be used to gather data and information from main stakeholders of the project.

This will mainly constitute the target groups, parents, MOEs and other actors engaged in similar programs, and project staff of SC. The data and information gathered will be systematically analyzed and presented following the evaluation criteria mentioned above.

Evaluation method

The evaluation methodology will be detailed during the first stages of the assignment as listed in the tasks above. It is envisaged to comprise the following data collection techniques:

a) Documents Review: Desk study review of all relevant Project documentation, including but not limited to:

* Project document and logical framework matrix
* Interim project reports submitted and approved by UNICEF
* Various implementation documents (guidelines, monitoring tools and job profiles) developed through the project:
* Government policy documents: National Education Policy, NFE policy, education sector strategy document

b) Qualitative: Focus Group Discussions, Semi-Structured Interview (SSI), will be made with the students, teachers and Community Education Committee members. Additionally Observations, Key Informant Interviews, and most significant change stories will be used for data collection.

c) Quantitative: The consultants are also expected to use and apply quantitative research techniques such as questionnaires (structured way of data collection). In addition, data will be collected against project indicators to measure progress/achievement of targets

Data Analysis:

Data gathered using various methods and tools have to be collated, analyzed, and interpreted
systematically. The evaluators will be expected to come up with detailed data/information analysis methods/framework. The analysis and interpretation shall be made along with the key issues of the evaluation. It is suggested that the evaluators employ triangulated data analysis procedures.

Deliverables

The consultants should deliver the following outputs and services:

* Detailed evaluation framework/design and implementation plan agreed with the Evaluation team.
* A comprehensive sampling frame
* Development and finalization of the evaluation tools.
* Orientation /training of the evaluation team.
* Draft evaluation report.
* Facilitate validation workshop in Garowe on the basis of the draft reports.
* Produce final draft evaluation report
* Finalise the reports of the zones and submit merged reports for the two to the lead agency
* The final report shall be submitted in three hard copies and CD-ROM to the Lead Agency in Nairobi.
* PowerPoint presentation to the Education Sector Committee

6. Expertise Required

The following are the qualification requirements for the evaluation consultant:

* Postgraduate qualification (Masters or above) in Education, international development, economics or in a discipline relevant to this assignment with a minimum of 10 years experience.
* Experience in the evaluation of technical assistance project, preferably, particularly those under the education portfolio;
* Experience with UN agencies preferred
* Evidence of evaluation of similar programmes.
* Excellent report writing skills.
* Competence and adequate experience in the use of qualitative and/or quantitative methods of data collection and analysis including: sampling, desegregation of data, structured and semistructured interviewing, focus group discussions, and observation and triangulation research methods.
* Ability to interpret and analyse complex qualitative and quantitative data, and to present findings and recommendations in a clear and concise way.
* High quality writing skills (in English).
* Excellent inter-personal communication skills including experience of facilitation and presentation.
* Ability to work equally well with communities and international organisations.
* Knowledge and sensitivity to political and social contexts of Somalia/Somaliland communities.
* Experience working in Somalia is an added advantage

7. Time Frame

The evaluation will be carried out from 1 – 20 May 2011. Consultants will be required to submit a technical proposal indicating the number of days and rate for the consultancy work with a realistic action plan.

8. Support Available

SC will facilitate and coordinate the evaluation process with the MoE including establishing an
evaluation team and assisting in the collection of the required data; paying the professional fee and other related costs from the contract; provide logistical support (transport, accommodation, food, etc during the assignment), and review/input methods and other outputs of the evaluation.

9. Supervision and Management

The overall supervision of the evaluators will rest with SC. The consultant will report to the Deputy Country Director (or a designated manager).

The consultant will be expected to work closely with, and interact a lot with implementing agency and the MOEs.

The Regional Education Coordinator and Primary Education Programme Manager based in Hargeisa will provide day-to-day supervision and support to the consultants.

10. Terms and Conditions

Logistics: The consultant’s travel from base to the field and back after the end of the contract (including airport tax), food, and accommodation will be covered by SC

Professional fee: The consultant will come up with their own rate as part of their financial proposal.

Tax and insurance: The consultants shall be responsible for their income tax and/or insurance during the assignment.

Code of conduct: Because Save the Children's work is based on deeply held values and principles, it is essential that our commitment to children's rights and humanitarian principles is supported and demonstrated by all members of staff. Save the Children's Code of Conduct sets out the standards which all staff members must adhere to. The consultant is bound by the principles and conditions of Save the Children’s Code of Conducts.

A contract will be signed by the evaluator upon commencement of the evaluation which will detail additional terms and conditions of service, aspects on inputs and deliverables.

11. Application Requirements

All expressions of interest should include:

* Cover letter: A short (maximum three pages) letter addressing the evaluation criteria.
* A detailed curriculum vitae
* Technical Proposal (maximum eight pages) interpreting the understanding of the TOR, detailed methodology of executing the task, as well as draft evaluation framework and plan.
* Financial Proposal: The financial proposal should provide cost estimates for services rendered including daily consultancy fees excluding: accommodation and living costs; transport cost; stationeries, and supplies needed for data collection; costs related to persons that will take part from consortium partners and government authorities during evaluation process, enrichment workshop.
* The financial proposal should also provide cost estimates for services rendered including daily consultancy fees related to the consultant and/or associate consultant who will take part in the final evaluation of the project in Puntland.
* Data collection and data processing costs are on the account of the Consultant. The Consultant should also use his own computer.

Application Criteria

Interested and eligible applicants can send copies of their CVs, testimonials and at least two copies of reports of similar work, Technical and Financial proposal via e-mail to;

Human Resources Manager
Save the Children Somalia/Somaliland
email: vacancies@scsom.org

Last date for receiving applications is on or before 25th Apr 2011