2.1 Evaluation Approach
The evaluation scope and approach were determined during a planning phase, in consultation with CIC branches involved in the delivery of the Immigration Loan Program. The logic model and evaluation framework were developed in discussion with the program areas. Terms of Reference for the evaluation were approved by the Departmental Evaluation Committee in October 2013. The evaluation was conducted in-house.
It was determined that a medium-sized evaluation would be conducted with the level of effort based on several factors including:
- the Immigration Loan Program had never been the subject of an evaluation;
- the Evaluation of the Government Assisted Refugees and Resettlement Assistance Program,Footnote 28 conducted in 2010-11 included a recommendation to examine the transportation and medical loans;
- the program has undergone changes over the years in response to financial pressures;
- policy options and recommendations have been and continue to be considered; and
- there has been a high level of interest in the program among stakeholder groups.
2.2 Evaluation Scope
The evaluation assessed the relevance, delivery and performance of the Immigration Loan Program, and was guided by the program logic model, which presents the expected immediate and intermediate outcomes:Footnote 29
- Program development, management and governance effectively support program delivery and decision making.
- Individuals in need receive loans or contributions according to their need in order to cover costs associated with admissibility, transportation, landing fees and other eligible expenses related to settlement.
- Loan recipients are fully aware of and understand the meaning and nature of the loan agreement they have signed.
- Collection services are accessible, effective, and responsive to client needs, and facilitate loan repayment.
- Loans contribute to the settlement of recipients.
- Loans are repaid in full and in a timely fashion.
- The advance from the Consolidated Revenue Fund is adequately replenished to sustain the Immigration Loan Program.
2.2.1 Evaluation Questions and Issues
The evaluation questions, organized by core issue,Footnote 30 are presented in Table 6.
|RELEVANCE (Need, Alignment and Role)||Section|
|Is there a continued need to provide immigration loans?||5.3|
|Is the provision of immigration loans aligned with departmental, government-wide and international objectives and priorities?||5.2|
|Is the provision of immigration loans consistent with federal roles and responsibilities?||5.1|
|PERFORMANCE (Achievement of Expected Outcomes)||Section|
|How well do collection services and client support facilitate loan payments?||4.4|
|To what extent do individuals in need have access to the loans (or contributions)?||4.2|
|To what extent are loan recipients fully aware of and understand the meaning and nature of the loan agreement they have signed?||4.3|
|To what extent are program development, management and governance effectively supporting program delivery and decision-making?||3.1 3.3|
|To what extent do immigration loans contribute to the settlement of recipients?||4.6|
|To what extent are loans repaid in full and in a timely fashion?||4.5|
|To what extent is the advance on the Consolidated Revenue Fund adequately replenished to sustain the program?||3.1|
|PERFORMANCE (Demonstration of Efficiency and Economy)||Section|
|Are the program’s resources managed effectively to facilitate the achievement of outcomes?||3.2|
|Are there alternative approaches to delivering the program that could improve efficiency?||Footnote *|
2.2.2 Evaluation Timeframe
The timeframe for the evaluation is 2003 to 2012. It was chosen in recognition that the loan repayment period can extend up to 6 years after landing and in some cases, go beyond this period. In certain circumstances, however, a shorter timeframe was used, based on the line of evidence and data availability. Timeframes are specified throughout the report as applicable.
2.2.3 Evaluation Focus
The evaluation primarily focused on GARs and PSRs as they represented the vast majority of recipients (97.8%), and on transportation, admissibility and assistance loans, as they were the most frequently provided during the timeframe being evaluated. RPRF loans were not examined due to the very small number issued (6 between 2008 and 2012).
2.3 Data Collection Methods
The evaluation of the Immigration Loan Program included multiple lines of evidence to allow for a triangulation of information (see Table 7). A mix of quantitative and qualitative approaches was used, and an evidence matrix served as a guide throughout the process.
|Line of Evidence||Description||Purpose|
|Document Review||A review of relevant Government of Canada documents (e.g., Speeches from the Throne, budgets, policy documents); departmental documents (e.g., manuals, operational bulletins); and documents from other government departments (e.g., documents on the Canada Student Loans Program).||Used to provide context and background information (e.g. historical) on program, as well as to inform the assessment of the program’s delivery, relevance and performance.|
|Literature Review||Online search, including Metropolis Canada resources, and settlement and non-governmental organizations’ literature and releases.||Used to gain a better understanding of the global refugee context, trends related to Canada’s refugee resettlement, and the impact on settlement and integration outcomes.|
|Interviews||Interviews with individuals or groups with knowledge of the program and/or experience with its delivery. The interview list was developed in consultation with program representatives. 53 interviews were completed with 67 individuals from CIC (NHQ), visa offices overseas and regional offices and external stakeholder groups (i.e. UNHCR, IOM).||Used to gain a better understanding of the operations of the program, its continued relevance and its perceived level of performance.|
|Focus Groups||Seven focus groups with GARs and five with PSRs. Focus groups were held in Vancouver, Calgary, Lethbridge, Winnipeg, Toronto, Ottawa and Halifax. 117 individuals participated in the focus groups.||Used to obtain information on personal experiences with the program and its impact on early resettlement, settlement and integration.|
|Survey of RAP Service Provider Organizations (RAP SPO)||Online survey of SPOs (funded through RAP) that deliver settlement services to GARs and PSRs. Responses were received from 19 RAP SPOs, which represents a response rate of 76.0%.||Used to assess program delivery and performance, in particular settlement outcomes.|
|Survey of Loan Recipients||A telephone and online survey of GARs and PSRs. A total of 742 individuals responded to the survey. The margin of error is ±3.55% using a 95% confidence level.||Used to assess program performance in terms of short and long term impacts on loan recipients and their families, and to assess program delivery from a client perspective.|
|Survey of Sponsorship Agreement Holders (SAH)||Online survey of SAHs who provide settlement assistance to PSRs. A total of 20 organizations responded to the survey, which represents a response rate of 24.7%.||Used to assess program delivery and performance, in particular settlement outcomes.|
|Administrative Data Analysis||An analysis of administrative data from a variety of sources, including CIC’s financial databases, the Global Case Management System (GCMS) and FOSS.||Used to develop the profile of loan accounts and loan recipients, as well as assess program performance, in particular loan repayment.|
|Longitudinal Immigration Database (IMDB)Footnote * analysis||Analysis of the incidence of income, level of income and use of social assistance among GARs and PSRs who landed both prior to and after the introduction of IRPA.||Used to assess performance in terms of potential impact on settlement and integration.|
Where qualitative information is presented in the report, the scale shown in Table 8 is used. Note that, in some cases (i.e., where the number of respondents was too small or where the question yielded more descriptive information), the responses were not coded and a summary approach was used to analyze the information.
|Descriptive||Percentage of Respondents|
|All||Findings reflect the views and opinions of 100% of respondents.|
|Majority/Most||Findings reflect the views and opinions of at least 75% but less than 100% of respondents.|
|Many||Findings reflect the views and opinions of at least 50% but less than 75% of respondents.|
|Some||Findings reflect the views and opinions of at least 25% but less than 50% of respondents.|
|A few||Findings reflect the views and opinions of at least two respondents but less than 25% of respondents.|
These lines of evidence, along with data collection instruments, are described in greater detail in the Technical Appendices.
The evaluation contained a balance of qualitative and quantitative lines of evidence and allowed the triangulation of data; however, there are a few limitations that should be noted. These limitations were mitigated as described below, and therefore did not have a major impact on the evaluation findings.
- Various data systems have been used by CIC to record information on loans and loan recipients. As a result, some data that is only available on microfiche had to be manually re-entered, and a sampling approach was used to build the administrative dataset. In other instances where demographic information on loan recipients was required, the analysis was based on a sub-population of loans (2008 and 2012), representing 99.7% of the loans (a total of 25,858 accounts) issued during this period. Finally, as contributions data and loans data are recorded in separate systems, datasets were compiled and both systems were cross-referenced; business rules were developed.
- Loan information was not available in the landing data file and therefore could not be linked to the Longitudinal Immigration Database (IMDB). Since almost all refugees receive a loan, economic outcomes analysis was conducted on all GARs and PSRs as a proxy for the recipients.
- The survey of loan recipients could not be fully administered in all of the languages originally identified due to difficulties in finding surveyors who could speak Burmese and Somali. While the survey could not be administered in Burmese, it was administered in Somali using an online version.
- The response rate to the survey of SAHs was relatively low. Survey results were reported as qualitative evidence in conjunction with findings from other lines of evidence.
- Date Modified: