Zenex Foundation:Terms of Reference: Evaluation services for the Early Grade Reading Programme

Zenex Foundation
Please note: this opportunity closing date has passed and may not be available any more.
Opportunity closing date: 
Sunday, 13 December, 2020
Opportunity type: 
Call for proposals

This is a call for the services of an agency/ consortium of agencies/ individuals available to extend evaluation support to the Department of Basic Education’s (DBE) Early Grade Reading Programme (EGRP). The programme is a reading improvement intervention that seeks to support the instruction of Home Language and English First Additional (EFAL) reading outcomes of Foundation Phase classrooms in the North West province of South Africa. This document sets out the details of the project and the evaluation services required from 2021 to 2023. Interested applicants must submit their proposals to the Zenex Foundation by 13 December 2020 at 17h00. The total budget for the evaluation should not exceed  R6,5 million.

1.           Introduction
This is a call for the services of an agency/ consortium of agencies/ individuals available to extend evaluation support to the Department of Basic Education’s (DBE) Early Grade Reading Programme (EGRP). The programme is a reading improvement intervention that seeks to support the instruction of Home Language and English First Additional (EFAL) reading outcomes of Foundation Phase classrooms in the North West province of South Africa. It is jointly funded by the Hempel Foundation, the United Nations International Children’s Fund South Africa (UNICEF-SA), Zenex Foundation and the North West Department of Education (NWDE) in collaboration with the national DBE. The three-year intervention will roll out high-quality teacher professional development support across Grades 1 to 3 in all eligible Setswana Home Language instruction non-fee paying primary schools of the Dr Ruth Segomotsi Mompati (DR RSM) district from January 2021 through to December 2023. This document sets out the details of the project and the evaluation services required from 2021 to 2023. Interested applicants must submit their proposals to Zenex Foundation by 13 December 2020 at 17:00pm. The total budget for the evaluation should not exceed  R6,5 million.
2.           Background
The DBE has been conducting research into the acquisition of reading through the Early Grade Reading Studies and Reading Support Project (RSP), across two provinces since 2015. The EGRS studies aim to provide this evidence in an iterative manner, with each study responding to existing and emerging research questions. Overall, these studies aim to systematically provide evidence on alternative models of teacher support for the teaching of reading. The First Early Grade Study (EGRS I) commenced in 2015 – 2017 by working in 230, quintiles 1-3, no-fee schools catering for the most disadvantaged learners in the North West province (districts of Ngaka Modiri Molema and Dr Kenneth Kaunda). After two years of implementation, formal impact evaluation results revealed that the basic programme of lesson plans, integrated reading materials and up-front training had positive effects on reading outcomes. A follow-up data collection one year after the interventions had ended, found that the impacts on the original cohort of learners persisted into Grade 4. The magnitude of the advantage held by learners in the coaching group in 2018 was similar to that after two years of intervention, at 40% of a year of learning. It must be noted that whilst coaching proved to be a successful mechanisms for quality outcomes, coaching was done externally and was quite costly.
The second EGRS study focused on establishing whether the model is effective for English outcomes.  The Second Early Grade Reading Study (EGRS II) was implemented in Mpumalanga (2017 – 2019) to measure the impact of the package of lesson plans, materials and coaching of EFAL, and to investigate the use of electronic lesson plans and virtual coaching. More information on the EGRS designs and findings of the evaluation reports is available here.
The EGRP is the latest study in the EGRS studies. This intervention leverages off the success evidenced by previous DBE-led impact studies that revealed that the basic programme of lesson plans, integrated reading materials and up-front training/coaching have positive effects on reading outcomes in the country. It focuses on exploring different models of coaching support to teachers, with a view to establishing how to embed coaching support into the system.  This role can be embedded within subject advisors or HODs. This project explores HOD coaching option. This project has been co-developed collaboratively by the Curriculum Policy, Support and Monitoring branch, the Teachers, Human Resource and Institutional Development branches (Branches C and T), and the Research Coordination and Monitoring and Evaluation (RCME) directorate of the DBE. Furthermore, strategic support was also received from UNICEF South Africa.
3.           Intervention overview
The EGRP intervention involves the provision of a structured reading programme as the base programme for all eligible schools. This programme focuses on both English and Setswana. This is further layered with innovative coaching models implemented in 80 priority schools in order to improve the teaching practice of reading. Ultimately, the improvement of reading outcomes of Foundation Phase learners in both Setswana Home Language and EFAL is anticipated.
The three main implementation components are
3.1         Training
All teachers from Grades 1, 2 and 3 across all participating schools will be trained every year for three years at the start of every term. Terms 1 and 3 will include an overnight residential component where each grade is trained for 2 days. Term 2 and 4 will be term training for a single day each term.
3.2         Materials
All Grades 1, 2 and 3 teachers in participating schools will be provided with the required printed Learning and Teaching Support Materials (LTSM) packages for Setswana and EFAL instruction. This includes lesson plans and readers for learners. In addition they will be provided with tablets containing the lesson plans for Setswana and EFAL.
3.3         Coaching
Dedicated Literacy Coaching will be provided by 8 coaches in 80 randomly selected schools for all Grade 1, 2 and 3 teachers in the sample using a blended ICT model. The 80 schools will receive coaching for the 3 year period, however the ratio will change in year 2 and 3 as summarised in the diagram below.
In year 1, each coach will support 10 schools. Based on the current project beneficiary numbers, a single coach would support approximately 6 teachers per school, and roughly 60 teachers overall per term per year.
In Year 2 and 3, the coaching schools will be split into 2 groups of 40. In the first group of 40 schools teachers will receive a high dosage on-site coaching support from an external provider. To enable this, 6 coaches will be allocated to this group. Four (4) of these coaches will work with an estimated coach school ratio of 1:7 schools. This reduces the overall number of teachers supported to 42 for these coaches. The other two (2) coaches will transition to a lower on-site dosage and have an estimated coach to school ratio of 1:6 schools. The estimated number of teachers these coaches will support is 36.
In the second group of 40 schools the HOD will receive on-site coaching and support. This onsite support will be augmented with virtual coaching support. It is intended that HODs then provide the relevant coaching support to teachers. In this model, external coaches provide support to HODs to set up and run COPs, they support HODs to effectively support teachers. Over time the coach will reduce their support to the HOD.  
Figure 1: Summary of the structure of the intervention
 

 
Scope of WorkThe main research questions the evaluations will be answering using a mixed method approach are:
4.1 What has been the impact of the coaching interventions on learning outcomes for Setswana and        
       EFAL at the end of the first year?
4.2 What has been the impact of each of the two coaching interventions on learning outcomes for  
       Setswana and EFAL at the end of the third year?
4.3 What is the extent to which that core methodologies targeting language instruction were   
       Implemented in classrooms?
4.4 How successful are HODs as coaches compared to external coaches?
4.5 How well have PLCs been functioning and why?
 
A Steering Committee convened by the DBE guides the department’s work in this regard, overseeing both implementation and evaluations of these ongoing reading research activities in fulfilment of the EGRS I Improvement plan. An appointed Project Management Team (PMT) oversees the commissioning of service providers and expert researchers to undertake the various components and is made up of representatives from the DBE, UNICEF South Africa and the NWDE. Zenex Foundation will be the contractual body for this evaluation service provider and also forms part of the PMT.
A team of researchers at the DBE form part of the PMT and will undertake the analysis, write up and communication of the programme evaluation. The services required in this TOR are in support of the activities needed to enable the collation of evidence through the following:
 
 Quantitative Data CollectionThe main measure of success for the proposed intervention is improved reading outcomes for learners in the Home Language and EFAL in the Foundation Phase. Thus, an experienced data collection survey company with sound experience in Early Grade Reading Assessment (EGRA)-type administration is required to conduct the following in Years 1 (201) and 3 (2023) of the study:
Data Collection in 140 of the 190 schools consisting of 80 coaching & 60 base programme schools of training and Learning and Teaching Support Materials (LTSM).
Assess 20 learners per grade in Grade 1 and 2 in Wave 1 (October 2021).
Assess 20 learners per grade in Grade 3 and 4 in Wave 2 (October 2023). This would be the same two cohorts previously assessed.
Assess 10 learners in Grade 2 in Wave 2 (October 2023).This would be a new sample.
Learners would be assessed in both Setswana and EFAL. 
3.1.2 Specific Undertaking
Learner Test and Questionnaire Development
The appointed Service Provider is to develop and version learner test instruments in the project’s languages using PMT approved EGRA items. The main purpose of the assessments is not to benchmark learners against curriculum requirements, but rather to determine their literacy abilities at the end of milestone years although the assessments will be aligned to the curriculum. Components of the assessment should include an oral assessment to be conducted by a fieldworker with individual learners in a one-on-one setting AND for Grades 2 and 3, a written assessment with a reading comprehension to be completed by learners in a group “exam-type” setting. It is expected that the Service Provider will propose which subtasks or skills will be assessed, for example, whether the test will be assessing the skill of phonemic awareness. To test 20 learners within one school day, it is requested that oral assessment be no longer than 15 minutes and 45 minutes for the written comprehension component. 
It is expected that the instruments will be aligned with best practice in early literacy assessment internationally. It is expected that the Home Language test items will be sensitive to the dialect spoken in the respective district so as not to introduce any bias into the results due to contextual differences between learners however the assessments must be based on the standard official languages. A comprehensive yet easy to follow training manual will also be developed and disseminated to Fieldworkers for reference following the training. It will be the Service Provider’s responsibility to identify, partner, and pay the use of a translation service provider able to assist with the versioning of the instruments, should these services not exist in-house.
Furthermore, the appointed Service Provider will be responsible for electronically versioning data collection instruments as well as teacher/principal questionnaires, learner home background questionnaires and the like onto the Tangerine ® Application or similar, to enable remote data collection via Android tablets.
Pilot and Full Data Collection
The Service Provider will be responsible for conducting the piloting of the instruments during the given timeframe and provide constructive feedback as to how well the questions worked. This will entail versioning test instruments and questionnaires onto the Tangerine ® application or similar prior to pilot data collection. The appointed Service Provider will lead the recruitment, training and supervision of qualified fieldworkers and budget accordingly for all aspects involved therein. This will include but is not limited to, travel and accommodation, training venues, data collection supplies, printing of learner assessments, charts, posters, subsistence fees and so forth.  Training will include intensive simulations and an enumerator evaluation to ensure that every individual data collector is able to adhere to the required data collection standards. This should include a simulation visit to schools on a certain day of training. Training will be conducted mostly in English, although the Setswana components of the learner assessment will have to be discussed by people who are proficient in this language. Fieldworkers have to be issued with detailed fieldwork training manuals and wear name tags to identify and associate them with this evaluation. The PMT will approve the preliminary fieldworker schedule and list of fieldworkers to be used for each round of data collection.
The Service Provider will be required to produce a clear and binding work schedule that includes a proposed schedule of visits and timeline as approved by the PMT prior to each data collection. Furthermore, they will be required to:
Deliver a detailed data management plan describing standardised data collection, linking, capturing, storing, coding and labelling procedures;
Establish procedures to ensure that the linking form data is captured accurately and ensure that all the data collected can be linked; and
Deliver a detailed data quality assurance plan describing the procedures which will be established to ensure high data quality both during data collection and data capturing. 
Deliverables
The main deliverable expected is for the Service Provider to furnish the PMT with the final raw data, uncleaned, in either CSV, Excel or Stata format.
Furthermore, the following deliverables are anticipated:
Services required
Deliverables
Timeline
Learner Test and Questionnaire Development

Grade specific EGRA oral assessments & learner charts
Grade 2 or 3 written assessments & learner booklets

End May 2021 AND
End May 2023

Learner Home Background Questionnaires (administered to all parents/ guardians)
Teacher and/or SMT questionnaires
School and Classroom Observation questionnaires
Teacher Language proficiency test

End June 2021 AND
End June 2023
Pilot

Fieldworker training and manual development
Instrument versioning on Tangerine ® or similar application
Pilot data collection in 6 non-evaluation sample schools

August 2021 AND
August 2023
Full Data Collection

Fieldworker training and selection
Instrument versioning on Tangerine ® or similar application
Fieldworker schedule and list of Fieldworkers
Full data collection in evaluation sample schools

October – November 2021 AND
October – November 2023
Receipt of final raw data, uncleaned, in either CSV, Excel or Stata format.
December 2021 AND
December 2023
 
 60 – Classroom Observation Study Data CollectionMonitoring and evaluation of this intervention would entail a mixed-method approach to provide micro, meso and macro lessons and insights. The specific sub-components that will be used to measure this will be receipt of lesson plans and LTSM and successful training for all schools. While the measure for coaching will be an analysis of the records of in-classroom observations grounded in the lesson plans and LTSMs consisting of feedback and modelling of core methodologies by the coach. In addition, this data collection should also enhance understanding and practices of PLC and their functionality as well as the role of the HOD in this. Thus, an experienced evaluation/ data collection survey company is required to:
Develop grade specific lesson observation instruments in the languages of the study (Grade 1 and 3);
Develop an observation instruments for PLCs and needs-based workshops as well as propose additional mechanisms to collect data on HOD practice;
Develop appropriate and insightful teacher and SMT structured interview questionnaires; and
Pilot, recruit and lead the training of qualified emerging young researchers to collect all components of the 60 – Classroom Observation Study in Year 2. 
3.2.1 Specific Undertaking
Instrument development
The Service Provider will have the requisite observational research experience and expertise in Foundation Phase curriculum development and language instruction in order to develop rigorous and in-depth lesson observation instruments.  The PMT will work closely with the Service Provider to develop:
1x EFAL lesson observation instrument for Grades 1 and 3;
1x Home Language lesson observation instrument for Grades 1 and 3;
Structured teacher interview questionnaires for Grades 1 and 3 teachers;
A PLC observation instrument to be used to administered to capture HOD practice;
A HOD shadowing tool to be administered to capture coach practice;
A coach shadowing tool to be administered to capture coach practice;
Structured document review questionnaire (to be completed independently by enumerators); and
All required protocols and training manuals for enumerators. 
Pilot and Full Data Collection
The Service Provider will be responsible for conducting the piloting of the instruments during the given timeframe and provide constructive feedback as to how well the instruments worked. This will entail versioning instruments and questionnaires onto the Tangerine ® application or similar prior to pilot data collection and full data collection. The appointed Service Provider will lead the recruitment, training and supervision of qualified fieldworkers and budget accordingly for all aspects involved therein. This will include but is not limited to; travel and accommodation, training venues, data collection supplies, printing of instruments (if applicable), stationery and so forth.
Deliverables
The main deliverable expected is for the Service Provider to furnish the PMT with the final raw data, uncleaned, in either CSV, Excel or Stata format.
The following deliverables are anticipated:
Services required
Deliverables
Timeline
Instrument development by expert curriculum/linguistic researcher.
-              1x EFAL lesson observation instrument for Grades 1 and 3;
-              1x Home Language lesson observation instrument for Grades 1 and 3;
-              Structured teacher interview questionnaires for Grades 1 and 3 teachers;
-             A HOD shadowing tools;
-             A coach shadowing tool;
-              A PLC observation instrument to be used with HODs;
- Structured document review questionnaire (to be completed independently by enumerators);
-              All required protocols and training manuals for enumerators
March – April 2022
 
Service provider to version instruments and questionnaires onto suitable software.
Completed instrument links for review by PMT
May 2022
Service provider to recruit, train and manage oversight of fieldworkers for PILOT at schools.

FW recruitment plans
Completed fieldworker training and Pilot fieldwork schedules,

June – July 2022
Service provider to recruit, train and manage oversight of fieldworkers for full 60 school data collection at schools.

Final fieldworker schedules and list of fieldworkers
Final raw and cleaned data to PMT for pilot and full data collection

July 2022
 
August 2022
 
 Qualitative Case StudiesIn addition to learner assessment, measures of success for coaching will be a substantial number of teachers implementing the curriculum well with sustained changes in their practice. Using a theory of change on coaching developed from EGRS II as in the diagram below. The study will aim to increasing the number of teachers that are confident and effective.
In addition, case studies should enhance the understanding of the practices and mechanisms for successful HOD support as they increasingly take on the coaching role. Data gathered should include identifying enabling conditions for PLCs, the main constraints on HOD time and practice and the documenting of successful and unsuccessful PLCs as well as why this happens.
 
 
Figure 2: Stages of teacher response to coaching

Taken from K Alsofrom 2018 EGRS II Qualitative Case Study
3.3.2 Specific Undertaking
The development of data collection materials and instruments that are based on international best practice, but adapted for the local context; 

The instruments need to ensure that the capturing of information allows for a nuanced understanding of what changes in behaviour and practice have led to improved learner outcomes, and the extent these changes could impact HL and EFAL learner outcomes in the Foundation Phase at a larger scale.

On-site observations of six HL and EFAL lessons of both Grades 1 and 3 learners and teachers across the EGRP schools in Year 1 and 3 (1x case per school);
On-site observation and interviews with HODs and coaches establishing PLCs ;and
A final narrative case study report to be shared alongside other EGRP findings to stakeholders and the public at large in Year 1 and 3. The reports should include a review of relevant literature and programme documents to substantiate and interpret the findings.Research questions to guide the inquiry will be co-created with the PMT researchers, often investigating specific themes or emerging questions emanating from the larger quantitative impact evaluation. It is expected that the Service Provider will have the requisite research capacity with a proven track record of undertaking qualitative research. Should the Service Provider not have such capacity internally, proof of subcontracted consultants or academic researchers must be listed with the working relationship described in full. CVs of all intended researchers must be outlined in the proposal. All aspects relating to the case studies will be led and overseen by the Service Provider, this includes but is not limited to: travel and accommodation, researcher/ consultant data collection and report development time, school arrangements (with support from DBE) and so forth. Proposals will need to budget for all aspects of this.
 
Deliverables
The following deliverables are anticipated:
Services required
Deliverables
Timeline
6 Case Studies in Year 1 in pre-selected schools across the EGRP sample

Inception/ planning meeting w/ PMT

Questions, methodology and approved fieldwork plan
In school data collection
Preliminary case study report for PMT review
Final case study report

March 2021
 
March – April 2021
 
September 2021
 
October 2021
December 2021
6 Case Studies in Year 3 in pre-selected schools across the EGRP sample

Inception/ planning meeting w/ PMT

Questions, methodology and approved fieldwork plan
In school data collection
Preliminary case study report for PMT review
Final case study report

March 2023
 
March – April 2023
 
 
 
September 2023
 
October 2023
 
December 2023
 
 
Proposal Submission

Activity
Date
Opening of advert
27 December 2020
The proposal submission must contain:

A brief outline of how you understand the task
An outline of the evaluation design and evaluation logic. This includes outlining a process, methodology, data collection strategy, instruments and data analysis.
Specifying ethical requirements
A detailed plan. A detailed project plan for each evaluation service listed, i.e. quantitative data collection, classroom observation and case studies.
Times line with key deliverables.
A detailed budget to support activities, please clearly identify VAT and profit where applicable;
Experience and expertise of the organisation and team that will undertake the evaluation. Outline of sector experience and expertise in evaluation services required and, where applicable, early grade reading improvement programmes in Southern Africa. Experience and expertise in Early Grade Reading Assessment (EGRA)-type test development and data collection will be an added advantage. Company registration documents; and 3 Reference letters from previous clients confirming expertise and experience in listed evaluation support activities.
List of team members. This includes CVS of the team leader and researchers/evaluators
BEEE status
 

 
Non-compulsory virtual briefing click here to join the meeting
3 December 2020 at 12:00pm to 14:00pm
Submission Closing Date
13 December 2020 at 17:00pm
Shortlisting of candidates
17 December 2020
Shortlisted candidate interviews/ presentation to the evaluation team
 
18-20 January 2021
Preliminary appointment of service provider
 
29 January 2021
Revised proposal for submission to Zenex Board
 
01 February 2021
Contracting
 
February 2021
Clarification/Inception Workshop
 
March 2021
 
CompetenciesThe following competencies and capacity to undertake activities are expected:
Experience and expertise in quantitative and qualitative data collection using various instruments in multiple primary schools while providing high-quality data management and quality assurance. Preference will be given to providers able to demonstrate capacity and experience with data collection in rural areas of South Africa.
Expertise in the development of school level/SMT and/or parental background questionnaires.
Expertise in the development of the necessary online data collection tools to support timeous and accurate data for analysis.
Experience and expertise in deploying professionally trained enumerators to conduct oral and written reading assessments in the intervention’s languages of Setswana and EFAL with young children.
Experience and expertise in deploying professionally trained enumerators to conduct structured classroom observations, school questionnaires and teacher/SMT interviews in the language requirements of the study.
Willingness and availability to work closely with the Project Management Team to ensure seamless data collection. 

Criteria weighting on submission of the proposal

Functionality
Weight

Experience in large scale national/provincial data collection

A minimum of 3 years’ experience.
Provided 3 recent reference letters and details of the fieldwork undertaken for each of them.
Specific experience in the primary-school sector and especially in assessing school-going learners aged 7 – 10 will be preferred.
Specific experience in collecting data using electronic devices/ online data collection tools

 
35

Methodology:

Provided the bid response proposal demonstrating understanding of the project and plans to successfully complete the project within the prescribed timeframe.
Included all required documents for bid application.

 
25

Capacity:

3.1.List of team members and contractors/experts to be used in specific activities. This includes CVs of the team leader and key researchers/evaluators; including no of black African researchers on the team.

Names, experience and CVs of project supervisors or Team leads and administrative team to work on the project.

 
10

Financial capability:

Detailed budget with bid price specified exclusive of VAT.
Audited financial statements for the 2017/18 and 2018/19 financial years.

 
30
TOTAL
100
 
ContractingThe successful bidders proposal will be submitted to the Zenex Board for approval in February 2021.
The implementation plan, deliverables as well as the nature timing of reports and other required data/information will be finalized before the signing of the contract. The contract will be subject to performance reviews by the EGRP PMT and Zenex Foundation, who may discontinue the contract at any time based on substandard performance or non-delivery. The Service Provider commits to abiding by all Zenex Foundation policies and regulations. Zenex foundation will make payments within 30 days of receipt of the invoices based on approved deliverables. As stated in the introduction the total budget for the evaluation should not exceed R6,5 million.
Project Management and GovernanceA Steering Committee convened by the DBE guides the department’s work overseeing both implementation and evaluations of these ongoing reading research activities in fulfilment of the EGRS I Improvement plan. As stated before, an appointed PMT oversees the commissioning of service providers and expert researchers to undertake the various components on a more ongoing basis and is made up of representatives from the DBE, UNICEF South Africa and the NWDE. Zenex Foundation will be the contractual body for this evaluation service provider and also forms part of the PMT. This advisory body will advise on all key aspects of the evaluation activities.
Intellectual PropertyThe appointed service provider will assign to DBE/Zenex Foundation its copyright to all data, instruments, documents, and reports emanating out of this study.
Permission may be sought by the service provider to use any data from the research. Such permission will not be unreasonably withheld provided the data will make a positive contribution to the sector and all reports and presentations emanating from such, will acknowledge the Foundation’s role.
The results of the findings will be shared with the broader sector and other participating stakeholders such as schools, other funding agencies, and key identified role players in education including the Department of Education.
Contact Person for Queries and SubmissionQueries and Proposals must be emailed to: Ms Nompumelelo Nkabinde mpumi@zenexfoundation.org.za 

For more information, contact Fundiswa Sayo fundiswa@zenexfoundation.org.za

For more about the Zenex Foundation, refer to www.zenexfoundation.org.za

To view other opportunities, refer to www.ngopulse.org/group/home-page/other-opportunities
 

Location: 
North West Province

NGO Services

NGO Services

NGO Events

S M T W T F S
 
 
1
 
2
 
3
 
4
 
5
 
6
 
 
 
9
 
10
 
11
 
12
 
13
 
14
 
15
 
16
 
17
 
18
 
19
 
20
 
21
 
22
 
23
 
24
 
25
 
26
 
27
 
28
 
29
 
30
 
31