ARCHIVED – Enhanced Language Training Initiative: Formative Evaluation
1.0 Introduction
This report reflects the results of a formative evaluation of the Enhanced Language Training (ELT) Initiative (referred to as the “program”) carried out between November 2006 and July 2007 by Goss Gilroy Inc., on behalf of Citizenship and Immigration Canada (CIC). The report is structured as follows:
- Section 1.0 – an overview of the ELT and the evaluation methodology, including a discussion of the limitations of this methodology;
- Section 2.0 – a profile of ELT projects and participants;
- Section 3.0 – the evaluation findings;
- Section 4.0 – the implications for the summative evaluation; and
- Section 5.0 – the conclusions.
1.1 Overview of the ELT
1.1.1 Program Description
The Enhanced Language Training (ELT) Initiative was launched as a pilot project in 2003/04. Its goal is to develop and deliver higher levels of language training and job-specific language training, including labour market understanding and/or experience. As a component of the Immigrant Settlement and Adaptation Program (ISAP), ELT contributes to ISAP objectives by assisting immigrants and refugees to access and remain in the labour market at levels commensurate with their skills and qualifications. [ note 5 ]
ELT funds two different kinds of projects: development and delivery projects. Development projects support the delivery of labour market levels of language training, but do not include the delivery component. For example, projects can include the development of assessment tools, software tools, research, study guides and other learner supports, and curriculum. Delivery projects include two components: language training and bridge-to-work. In most cases, language training is delivered to attain Canadian Language Benchmarks (CLB) levels 7 to 10 (English/French), including job-specific language training. In smaller centres (e.g., where there is currently no language instruction infrastructure for newcomers such as LINC), the program funds language training at CLB levels 1 to 10, including job-specific language training. The second component, bridge-to-work, comprises a variety of employment-related activities that can include, for example, orientation to the local labour market, assistance in finding employment in the immigrant’s field of specialty, mentoring, work placements, cultural orientation in the workplace and preparation for licensure exams and internships.
1.1.2 Resources
The 2003 Federal Budget allocated $5 million per year to implement projects. This amount was increased in the 2004 Federal Budget by an additional $15 million per year. The program allocates $20M annually ($5M starting in 2003-04 and $15M starting in 2004–05) for all provinces and territories outside Quebec [ note 6 ], additional amounts specifically for Ontario as per the Canada-Ontario Immigration Agreement (COIA), which represents $10M for 2006–07, $20M for 2007–08, $30M for 2008-09, $40M for 2009–10 and on going. ELT funds are available to cover costs related to child minding and transportation.
ELT projects must include cost-sharing in the form of funds, in-kind contributions, services, tools or facilities. Eligible partners include provincial and territorial governments; municipalities, businesses, not-for-profit organizations, individuals, non-governmental organizations, community groups and colleges and universities, etc. [ note 7 ]
Initially, a condition of the program was that the project costs be shared 50%/50% between the federal government and provincial governments and/or SPOs. However, the applications received in response to the Call for Proposals early in 2005 indicated that respondents were unable to meet this requirement. CIC subsequently requested Treasury Board of Canada approval to modify the cost-sharing condition. Currently, the provinces/territories are required to contribute at least 20% of the project costs. Not-for-profit organizations (including colleges and universities) can receive up to 100% of the project costs from the federal government; however, financial or non-financial (in-kind) contributions are encouraged and are worth additional points in the assessment of proposals.
1.1.3 Program Delivery Approach [ note 8 ]
Originally, ELT management and delivery was centralized at Integration Branch, NHQ, largely due to the pilot status of the program (i.e., there were no procedures, standards, tools, partnership models yet in place). Decentralization of ELT management to the CIC regional offices began in Summer 2006 in Alberta and culminated in Spring 2007 in Ontario.
Under decentralization, the roles and responsibilities of NHQ include areas with primarily a national focus, such as negotiating and managing federal-provincial agreements, managing and coordinating national and demonstration projects, developing national policies, guidelines and tools (in consultation with the Regions), providing functional guidance to the Regions, gathering and analyzing data, gathering and sharing best practices, among others. [ note 9 ] The roles and responsibilities of regional CIC offices include liaising with SPOs and provincial government colleagues, developing and coordinating requests for proposals (in collaboration with NHQ and other Regions), assessing project proposals, negotiating and managing contribution agreements, providing functional guidance to SPOs, helping to build SPO capacity to deliver ELT, collaboration with NHQ on plans and priorities, evaluation activities, program development and design, among others. [ note 10 ]
The Governments of British Columbia (BC) and Manitoba have both signed cost-sharing settlement realignment agreements, which include ELT. Also, a cost-sharing contribution agreement was signed with the Government of Saskatchewan. In these three provinces, SPOs submit applications directly to the provincial government. In Alberta, the province participates in joint Calls for Proposals and joint selection of projects. Alberta SPOs then sign separate contribution agreements with CIC and Alberta.
1.2 Evaluation Methodology
The objective of this formative evaluation was to review the delivery of ELT across all provinces, [ note 11 ] identify, the outcomes of the program and identify the factors that contribute to, or detract from, the success of the program. As it was a formative evaluation, the evaluation was also designed to contribute to the planning for the summative evaluation. The methodology for this evaluation included specifically:
- A review of key program documents;
- In-person or telephone interviews with 26 key informants (KIs) from CIC (NHQ, regional and local offices), provincial/territorial governments, other federal government departments and other national stakeholders;
- A statistical analysis of CIC program data;
- Case studies of twelve SPOs, based on site visits to the SPOs in six cities and including a range of types of projects (development and delivery) and SPOs (immigrant serving organizations, educational institutions, provincial governments and an employer-based organization). [ note 12 ] These case studies included interviews with SPO staff (often including the instructors) and employers and focus groups with participants; [ note 13 ]
- Mini-case studies with an additional twenty SPOs, based on telephone interviews with SPO representatives; [ note 14 ] and
- An assessment by two external experts, who are faculty members or staff of the Second Language Institute at the University of Ottawa, of selected project tools used by the SPOs that were included in the site visit case studies.
As is frequently the case with evaluations, even with a comprehensive evaluation methodology, there are limitations to the evaluation results and these are outlined below. In spite of these limitations, the evaluation does provide strong qualitative information on a) the range of ELT projects being funded and their success or limitations, from the perspective particularly of the SPOs, and b) information on the challenges of the program administration from the perspective of various stakeholders, including CIC staff, provincial/territorial representatives and SPOs.
Qualitative Nature of Evidence
The core methodology for this evaluation was case studies – some carried out in-person during site visits to SPOs and some carried out by telephone. As a result, the nature of the evidence is primarily qualitative. While this allows for in-depth exploration of the evaluation issues, it makes it more difficult to extrapolate the findings to all ELT projects. However, the case studies that were conducted with 32 SPOs represent nearly one-quarter (23%) of the total number of SPOs funded by ELT.
A second challenge with the case study information is that the case studies were conducted in two phases – first the site visit case studies and then the telephone case studies were initiated to replace the proposed online survey. The initial case studies were designed primarily as qualitative sources of information. Had it been foreseen at the beginning that there were so many case studies conducted, it would have been possible to adopt a more quantitative approach to the case studies. As a result, although 32 case studies were conducted and the evaluation team has tried to quantify the results where possible, they remain primarily qualitative sources of information.
Data limitations
The expectation in the design of this formative evaluation was that the ELT database would provide considerable information on the project and its participants. However, it became clear that there were considerable limitations to the information available from the database. As a result of the different delivery structures in different provinces, data on projects and participants is not being collected in the same way in all provinces. At the time of the evaluation, the database did not include information on all projects and/or participants. [ note 15 ]
In addition to the missing participants, there are also other inconsistencies in participant data that is available – including missing information for some variables and the poor quality of some data (in part due to the fact there is no fixed format for data entry). See Appendix B for a summary of data quality issues, by key variable.
Evaluation of ELT Tools
The independent expert assessment of the tools provides very detailed information on the tools being used in ten ELT projects (i.e., those where site visit case studies were conducted and tools were available to be reviewed). But this review was undertaken by the experts in the context of the projects in which they were being used – they were not reviewed as independent, stand-alone tools. The results of these reviews need to be considered in that context.
_______________
[5] CIC RPP, 2005/2006. Part 3
[6] ELT excludes Quebec as all settlement services delivered in the province of Quebec are governed by the Canada-Quebec Accord, signed in 1991. The Quebec government has exclusive responsibility for offering to all immigrants in Quebec integration services comparable to those offered elsewhere, with financial compensation from the federal government.
[7] ELT CIC Staff Training Kit Ontario Region, updated June 9, 2006. Page 14.
[8] Adapted from the Draft Accountability, Risk and Audit Framework for Settlement Programs, CIC, February 2004.
[9] ELT CIC Staff Training Kit Ontario Region, updated June 9, 2006. Pages 34-35.
[10] Ibid. Pages 36-37.
[11] The evaluation of ELT marks the first evaluation of settlement programming undertaken in collaboration with the provinces of BC and Manitoba, thus ensuring a more national picture than previous settlement program evaluations (as with previous evaluations, activities in Quebec are not included).
[12] Note that one case study was conducted with the Vancouver Community College SPO. This SPO received funding (through the Government of British Columbia) for the development of a curriculum. The funding was for the development of the curriculum only and did not cover any costs associated with curriculum delivery. This curriculum is currently offered by Vancouver Community College with all costs covered by tuition (no part of the delivery of this curriculum is funded by ELT).
[13] Although there were twelve case studies, there were only eleven focus groups because one of the case studies was of a development project. Also, because of inability to attract participants, one group included only one participant.
[14] This line of evidence was added mid-way though the evaluation once it became apparent that the originally planned online survey would not be possible (due to the various delivery structures of ELT across provinces).
[15] The program has received data for an additional 1,500 participants from projects starting in 2005/06 that will be entered into the database shortly.
Page details
- Date modified: