Evaluation of the Federal Skilled Worker program

1. Introduction

1.1 Purpose

Canada has adopted a comprehensive immigration policy to select applicants for immigration. The system contains provisions for several classes of immigrants: family, economic (which includes skilled workers, provincial nominees, self-employed persons, entrepreneurs, investors, and live-in caregivers), and refugees. Since June 28, 2002, Canada’s immigration program has been based on the Immigration and Refugee Protection Act (IRPA) and its regulations. The Federal Skilled Worker Program (FSWP)⁴ was designed to select permanent residents based on their ability to become economically established in Canada. This report presents the findings and conclusions of the evaluation of the Federal Skilled Workers Program, from the period following introduction of IRPA in 2002 to the implementation of the Ministerial Instructions (Bill C-50) in 2008.

The purpose of this evaluation is to assess the design, implementation and impact of the redesigned FSWP, introduced on June 28, 2002. More specifically⁵, the objectives of the evaluation are to assess:

1. Program design and implementation, including timeliness, consistency and transparency of selection; and
2. The impact of the program to date at the immediate and intermediate outcome levels, including a preliminary assessment of the economic establishment of skilled workers.

The evaluation focuses on a series of evaluation issues and questions related to program relevance, design and implementation, program impact, alternatives, cost-effectiveness, and unintended outcomes, as listed in the chart below.

1.2 Evaluation methodology

The evaluation used multiple approaches and lines of evidence. Data was collected and analyzed from a variety of primary (e.g. interviews, surveys and focus groups) and secondary data sources (document and literature review, as well as federal government databases).

A. Primary Data Sources

The primary data sources for this evaluation include interviews with key informants, surveys of FSWs and their employers, and case studies. The methodology used for each of line of evidence is outlined below:

• Interviews with key informants. The key informant interviews were designed to address evaluation questions related to relevance, program design and implementation, program impact, effectiveness, unintended outcomes and alternatives. As indicated in Table 1-1, 53 key informants were interviewed.

• Surveys of FSWs and employers. A telephone survey of 1,500 FSWs who entered Canada through the FSWP was conducted. A letter from CIC asked skilled worker immigrants to formally consent to participate in a telephone survey. The letter was sent to a random sample of 30,000 Federal Skilled Workers who were selected under the IRPA criteria and landed between 2002 and 2008. Six thousand were returned as wrong addresses. Of the 24,000 FSWs who received the letter, 2,053 consented to participate in the survey. Each of these FSWs was contacted for a total of 1,500 respondents. FSWs interviewed were asked to provide information about their current/recent employers, and employers who had participated in the Arranged Employment Offer (AEO) program. A sample of 110 current or recent employers and 53 employers who had previously made an AEO to one or more FSWs, whom they subsequently hired, were interviewed. The purpose of these interviews was to get the views of employers who have hired FSWs on the need for the program, the process of applying for and hiring FSWs with arranged employment, characteristics of FSWs and their positions, and employers’ overall satisfaction with the FSWP.
• Case Studies. The case studies consisted of field visits to 5 Canadian Visa Offices Abroad (CVOA): London, New Delhi, Hong Kong, Port of Spain, and Buffalo. These visits were designed to provide a better understanding of the program implementation, selection processes, challenges and best practices. Case studies consisted of interviews with visa office staff, including medical officers, a focus group in each mission, and review of mission documents, data and a sample of FSW files. Interviews were also conducted with stakeholders from the Canada Border Services Agency (CBSA), CSIS, and RCMP, Canadian Integration Immigration Project representatives in New Delhi and Hong Kong, a representative from NARIC (London) and staff from the British Council. Client interviews conducted by visa officers as part of the assessment process were observed. In total, 83 visa office staff and stakeholders were interviewed (8 managers, 55 visa office staff, and 20 stakeholders), 73 client files were reviewed, and 8 client interviews were observed.

B. Secondary Data Sources

The secondary data sources for this evaluation include document review, literature review, and statistical analysis of the Field Operations Support System (FOSS), the Computer Assisted Immigration Processing System (CAIPS) and the Longitudinal Immigration Database (IMDB). The following describes each of the secondary data sources used in this evaluation:

• Document Review. In the document review, particular emphasis was given to obtaining information to address evaluation questions relating to program relevance and to application processing and selection. Examples of the documents reviewed include operational manuals; mission directives; briefing notes; progress, financial, statistical, and annual reports; policy documents; operational profiles; program meeting notes; and process and procedure documents.
• Literature Review. A literature review on skilled immigrant workers both in Canada and in other countries was conducted to obtain additional perspectives and evidence regarding approaches for selection and  processing of skilled worker applicants. A comparative analysis of similar programs in Australia, New Zealand was undertaken. The selection process in the province of Quebec, which selects immigrants destined to that province, was also examined. Evidence-based information on the economic success of immigrants, changes in the labour market, and the appropriateness of the current selection system was identified through the literature review.
• Statistical Analysis of CAIPS and FOSS databases. The Computer Assisted Immigration Processing System (CAIPS) database contains information about all the immigration applications submitted to CIC and the points received by FSW applicants on each of the selection factors of the grid (see section 1.3 for data limitations). The Field Operations Support System (FOSS) contains landing information on immigrants entering Canada. The analysis of these two databases was used to address evaluation issues regarding program design and implementation, including timely and efficient entry of skilled worker immigrants (CAIPS) and to describe landed immigrant profiles (FOSS).
• Statistical Analysis of the Longitudinal Immigration Database (IMDB)⁶. The longitudinal Immigration Database (IMDB) is a database that links immigration and taxation administrative records into a comprehensive source of data on the economic performance of the immigrant population in Canada⁷. For the purpose of this evaluation, a series of fields from the CAIPS data file was added to the IMDB. This allowed for a more complete analysis of the FSWP.  The IMDB was used to assess a number of evaluation issues related to program impact, effectiveness of the selection criteria, and the mobility (interprovincial and outward migration) of FSWs. More specifically, it was used to compare economic outcomes of principal applicants (PA) admitted under the IRPA regime to the pre-IRPA selection regime, and to examine factors that have the greatest impact on their outcomes based on the selection regime and points under which they were accepted. For the purpose of the analysis, all earnings were adjusted using the Consumer Price Index (CPI) to account for inflation. This allows comparison of earnings across the different years.

1.3 Evaluation strengths and limitations

A. Evaluation Strengths

Key strengths of the evaluation approach include:

• Relationships can be inferred through the use of multiple lines of evidence, for example, the use of various sources of information and a mix of methods such as interviews, surveys, case studies, document and literature research, and statistical analyses of three different databases. This type of evaluation design facilitates triangulation of findings and improves the reliability and validity of findings.
• Significant sample sizes incorporated the perspective of the key stakeholder groups involved with the program. Over 1,800 individuals from various groups provided input via primary data sources, thus strengthening the level of evidence.
• The extensive use of secondary data covering pre-IRPA and IRPA (e.g. applications/approvals, processing times, income data) was used to develop a simple time series design and create a baseline time trend for comparison purposes. As mentioned earlier, IMDB analysis included large number of individuals arriving between 2000 and 2006, including 157,440 pre-IRPA FSWs, 28,730 FSWs arriving under dual assessment and 31,945 FSWs arriving under IRPA. A total of 218,115 individuals were included in IMDB analyses. Only individuals who made a tax filing are included in the IMDB. The number of individuals included in this database is therefore lower than the total admission numbers, as not every immigrant has filed a tax return.
• A variety of descriptive and inferential statistical analyses were utilized (e.g. cross tabulation, regression, survival analysis, etc.) to further support comparative findings and the effectiveness of selection criteria.

Triangulation of results from the multiple lines of evidence indicates strong consistency of findings across data sources, with the exception of AEOs (see section 3.4. of the report). The evaluation data obtained from each line of inquiry was analyzed to address each evaluation question, taking into account their strengths and limitations.

B. Challenges and Limitations 

Some of the challenges and limitations that should be considered in reviewing the results include:

• Potential non-response error in the survey of FSWs. Given the self-selected nature of the survey, there is concern that the characteristics of FSWs who responded may be different from those who did not. A comparative analysis indicated that there are minor differences in FSWs participating in the survey compared to the general population of FSWs. The FSWs who participated are somewhat more likely to:
  • Have landed in Alberta (15% versus 11% in the overall population of FSWs)
  • Be female (36% versus 31%)
  • Be British (11% versus 7%) and less likely to be Indian (8% versus 17%) or Chinese (7% versus 16%)
  • Be over the age of 35 at the time of landing (64% versus 50%)
  • Have English as a Mother Tongue (29% versus 19%).

In addition, the FSWs who participated are less likely to have moved. Approximately 20% of the introductory letters mailed to FSWs were returned to CIC because the FSWs no longer resided at that address⁸.

• Potential respondent bias (i.e., target groups with a vested interest in the program). Some groups included in the analysis such as employers of FSWs with an AEO, could be biased in their responses if it is in their interest to support the program. Views on the different themes (i.e., processing of applications, economic performance of FSWs, etc.) have been provided by multiple stakeholders. Therefore, potential bias was mitigated through the use of multiple lines of evidence, as well as validating findings through other primary and secondary research. How FSWs fulfill labour market needs could, however, only be addressed through the survey of FSW employers.
• The sample of employers was developed based on referrals from FSWs. This introduces a potential bias given that FSWs are more likely to provide referrals to employers with whom they have a positive relationship. As well, these findings cannot be generalized as the sample was not randomly selected.
• Challenges with the design of the data collection system (CAIPS). The allocation of points and entry into CAIPS is done in two steps. The first step, ‘paper screening’, is based on a review of the documentation in the applicant file and is generally conducted by locally engaged staff in Canadian Visa Offices Abroad (CVOA). It typically results in a paper screening decision entered in CAIPS. The paper screening points can be changed by the visa officer who makes the selection decision on the application and enters selection points into CAIPS (second step of the process). If the interview is waived (about 80% of IRPA cases) the selection decision is positive but the selection points are reset to 0. Therefore, the only information remaining for these individuals is paper screening points. The  IMDB analysis used paper screening points, unless selection points were available. Tests were conducted to ensure that paper screening points provided an acceptable alternative. For those individuals for whom paper screening points and selection decision data were available (ie: those who were interviewed), results show a low level of variability. Results indicate that for most variables, paper screening and selection results were close (language showed the greatest variation, with 72% of points remaining the same between paper screening and selection).⁹  When the values changed, it was generally to increase the number of points awarded. Therefore, it can be assumed that using the paper screening points rather than selection points (when not available) would, if anything, provide a more conservative picture of the situation.

Notes

⁴ FSWP refers to Federal Skilled Workers Program, FSWs refer to Federal Skilled Workers.

⁵ A logic model has been developed for the FSWP which outlines activities, outputs and program outcomes. The logic model is presented in Appendix III.

⁶ For more detailed information about the IMDB analysis that was done for the purpose of this evaluation, refer to the IMDB technical report. In addition to presenting the methodology used and analysis results, the report contains a section comparing the population in the IMDB to the population in FOSS for the same time period.

⁷ Note that the number of individuals who are included in the IMDB is lower than the total admissions as not all immigrants have filed a tax return.

⁸ Some individuals might have moved within the same city or province, or between provinces.

⁹ Language showed the greatest variation with 72% of the points remaining the same between paper screening and selection. The points for adaptability were the same for 82% of the cases where both sets of points were available, while they remained the same 86% of the time for both experience and education points. The age and arranged employment factors were the ones where the greatest consistency was observed, with points remaining the same in 99.5% and 97.5% of the time respectively.

• Previous
• 1
• 2
• 3
• 4 (current)
• 5
• 6
• 7
• 8
• Next