Evaluation of the Overseas Orientation Initiatives

Executive summary

Purpose of the evaluation

As per the requirements under the Financial Administration Act, an evaluation of Citizenship and Immigration Canada’s (CIC) Canadian Orientation Abroad (COA) initiative was required in fiscal year 2011/12. COA is one of three in-person pre-departure orientation initiatives funded by the department and CIC is in the process of establishing an overseas orientation strategy to frame immigrants’ orientation needs and its programming priorities regarding pre-departure services. Therefore, the evaluation was expanded to include all three of CIC’s pre-departure orientation initiatives.

The data collection for the evaluation was undertaken by CIC’s Research and Evaluation Branch (R&E) between July 2011 and January 2012.

CIC’s overseas orientation initiatives

CIC currently funds three initiatives that offer pre-departure orientation: Canadian Orientation Abroad (COA), the Active Engagement and Integration Project (AEIP), and the Canadian Immigrant Integration Program (CIIP). These initiatives are delivered by three different third-party service providers. They are offered in different locations and have distinct service delivery models, ranging from general information and awareness services to an integrated support system that includes needs assessments and referrals.

Canadian Orientation Abroad: COA is delivered by the International Organization for Migration (IOM) and is provided in over 40 locations. Eligible clients include federal skilled workers (FSWs), provincial nominees (PNs), refugees, members of the family class (FC), and their spouses and working age dependents. Live-in caregivers (LCs) are also eligible for COA. It is offered as a 1-, 3-, or 5-day session. The objective of COA is to provide information to enhance knowledge of Canada and to ensure that newcomers know how to obtain assistance upon arrival.

Active Engagement and Integration Project: AEIP is delivered by S.U.C.C.E.S.S. and is offered in Seoul, South Korea and Taipei, Taiwan. Eligible clients include FSWs, FC, PNs, business immigrants, and their spouses and working age dependents. LCs in Taiwan are also eligible to take AEIP. AEIP participants can take a 2-hour group orientation session, topic-specific workshops, have a one-on-one interview, and receive referrals to organizations in Canada. The objectives of AEIP are to support the settlement, adaptation and integration of newcomers into Canadian society and promote community and labour market integration.

Canadian Immigrant Integration Program: CIIP is delivered by the Association of Canadian Community Colleges (ACCC) via offices in India, China, the Philippines, and the United Kingdom. Eligible clients include FSWs and PNs and their spouses and working age dependents. CIIP participants can take a 1-day group orientation session, have a one-on-one interview, and receive referrals to organizations in Canada. The objective of CIIP is to help prospective economic immigrants prepare to meet foreign credential requirement and achieve labour market integration.

Methodology

The evaluation was designed to address three broad themes: relevance, design and implementation, and performance. In keeping with the requirements of the Directive on the Evaluation Function (Treasury Board of Canada Secretariat, 2009), program relevance was assessed in terms of: (1) continued need; (2) consistency with respect to federal roles and responsibilities; and (3) alignment with government and departmental objectives and priorities. Program performance was assessed by examining program results in terms of: (4) effectiveness; and (5) efficiency and economy. The evaluation used multiple lines of evidence to ensure the strength of results. Several lines of enquiry, including both quantitative and qualitative lines of evidence, were used for the evaluation:

  • interviews;
  • administrative data analysis;
  • site visits;
  • focus groups;
  • analysis of COA survey responses;
  • federal skilled worker survey; and
  • document review.

The scope of the evaluation included COA activities from 2005-2006 to 2010-2011 and AEIP activities since program inception in 2008 to 2010-2011. With respect to CIIP, the evaluation focused mainly on the first year of operation under CIC (2010-2011). However, because some of the participants to the FSW survey would have taken CIIP when it was the responsibility of Human Resources and Skills Development Canada (HRSDC), that line of evidence covers CIIP activities in 2009-2010.

Limitations

Although the evaluation included a good balance of quantitative and qualitative lines of enquiry, and allowed for the triangulation of results, there were four notable limitations to the methodology, which should be considered when reviewing the evaluation results.

  • There is confidence in the FSW survey findings overall; however the level of confidence varies according to orientation initiative. There is a higher level of confidence in the responses for CIIP participants (599 responses) and COA participants (445 responses) than AEIP participants (89 responses). Therefore, caution should be used in drawing conclusion with the AEIP survey data.
  • There was limited information available to assess the impact of COA and AEIP on LCs, therefore, results for LCs cannot be considered representative of all LCs.
  • The COA survey was not designed specifically to respond to the evaluation and therefore, did not provide information for all of the evaluation questions and indicators. In addition, given the size of Canada’s refugee population, the COA survey contained a limited number of responses from refugees.
  • CIC systems are not designed to identify which of those refugees and immigrants that arrived in Canada have taken pre-departure orientation. This resulted in some limitations with respect to calculating the proportion of individuals arriving in Canada that took pre-departure orientation—information that was needed not only to examine program results and reach, but also to establish sample size for informed consent. Therefore, certain assumptions were made with respect to how much time elapsed between taking pre-departure orientation and arrival in Canada.

Evaluation findings

The main findings associated with each of the evaluation questions are presented below.

Relevance

  • There is evidence that pre-departure orientation, as per its ‘common’ definition is needed for refugees, as it can address initial settlement and integration challenges that they face. However, there was no evidence that this type of pre-departure orientation can address gaps and challenges for non-refugees given that their needs are focused on specific employment-related issues rather than initial orientation to Canada.
  • While there is no legislative obligation to provide pre-departure orientation services, interviewees believe there is a role for the federal government in delivering these services to ensure consistent messaging overseas; however, there is a lack of clarity regarding the respective roles of the federal government and provincial governments in delivery.
  • All three pre-departure orientation initiatives are well-aligned with CIC priorities related to settlement, more specifically those related to informing settlement decisions and supporting labour market integration. The three programs are also linked to federal priorities related to humanitarian assistance and foreign credential recognition and labour market integration. With planned changes to the selection process for economic immigrants, there may be a need to examine the role of pre-departure orientation to ensure that it continues to be aligned with those changes.

Design and implementation

  • CIC’s three pre-departure orientation initiatives do not overlap with one another as they have different objectives, locations, and offerings, although there is one area of duplication with respect to COA and CIIP in the Philippines. The information provided to participants is in alignment with the specific objectives of the initiatives and the different groups that are targeted. In addition, CIC delivers pre-departure orientation services for refugees similarly to other countries.
  • Governance structures are in place to manage each of CIC’s pre-departure orientation initiatives and interviewees reported that those structures work well. However, there is a lack of coordination within CIC with respect to the overall strategic direction and management of pre-departure orientation, including the lack of a clear strategy to identify what type of information should be provided to which immigration categories and in what locations.
  • There was no clearly articulated rationale for how the locations and target groups for pre-departure orientation were selected. The fact that pre-departure orientation is being offered in some countries that do not account for a large percentage of immigrants suggests that it may not be offered in the most appropriate locations or to the right target groups.
  • While pre-departure orientation has been taken by many immigrants, the extent to which planned targets are being met vary. One of the main factors that may contribute to this variation among non-refugees is the way in which individuals are informed of the sessions, as information about pre-departure orientation is not consistently distributed. For refugees, other factors related to security and geography were cited.

Performance (effectiveness)

  • Overall, participants to pre-departure orientation were satisfied with the sessions, although not all of the enhanced services (e.g., referrals, workshops) offered by AEIP and CIIP were useful to all participants. Orientation information is provided to participants in a timely fashion and those who took it found it useful to prepare for the trip to Canada.
  • In-person pre-departure orientation helped newcomers prepare for life in Canada and ensured that they knew what to do upon arrival, including accessing settlement services. There was some slight variation between orientation programs; however, this was likely due to the fact that not all place the same emphasis on settlement-related information. Few challenges were identified in this respect, although some pre-departure orientation participants indicated that more information would have been helpful.
  • Participants to pre-departure orientation received accurate information, which helped to manage newcomer expectations, although not entirely.
  • CIC’s pre-departure orientation initiatives helped newcomers prepare for employment in Canada to varying degrees based on which orientation they took. The biggest challenges and gaps for orientation participants were employment-related.

Performance (efficiency)

  • The cost per participant for COA has been fairly stable and is in line with what was expected given that COA met its participation targets in most years. The overall cost for COA and its cost per participant are influenced by a number of factors including the fact that it serves a large number of immigrants and is delivered within the existing IOM structure, thus taking advantage of facilities and trainers that are used for purposes other than just COA. In addition, for its cost, COA has provided pre-departure orientation to about 20% of FSWs/PNs, LCs, and FC and anywhere between 31-56% of refugees in the locations in which it is offered.
  • The cost per participant for AEIP is higher than what was expected given that AEIP did not meet its participation targets for many of its offerings, with the exception of the workshops. The overall cost for AEIP and its cost per participant are influenced by a number of factors, including the fact that it has served a fairly small number of participants and has offices in two overseas locations, staffed with full-time trainers entirely dedicated to AEIP. In addition, for its cost, AEIP has provided pre-departure orientation to about 11% of the FSWs, PNs, LCs, FC, and business immigrants in the locations in which it is offered.
  • The cost per participant for CIIP was lower than expected given that it exceeded its participation targets, although for its cost, it provided pre-departure orientation about 8% of FSWs in the locations where it is offered. The overall cost for CIIP and its cost per participant are influenced by a number of factors, including its network of focal point partners and the fact that it has offices in four overseas locations, staffed with full-time trainers entirely dedicated to CIIP.

Conclusions and recommendations

CIC currently funds three pre-departure orientation initiatives with different stated objectives and depending upon location, eligible participants may include refugees, live-in caregivers, members of the family class, provincial nominees, federal skilled workers, and business immigrants. Over time, some of the initiatives have expanded delivery locations and client groups, however, there has not been a clearly articulated rationale for this expansion. There is no formal articulated common approach or framework in place for the provision of pre-departure orientation, including a definition of what is to be achieved through pre-departure orientation and what information needs to be provided to newcomers prior to departure.

Recent changes have been announced to the selection criteria regarding the economic category, which include requirements for higher language proficiency and more emphasis on pre-assessment of foreign credentials and pre-arranged employment. These changes will likely have an effect on the source countries for economic immigrants, as well as amend the type of information that might be needed by those individuals prior to arrival, and the time at which it is needed.

Recommendation #1: CIC should develop a strategy for the provision of pre-departure orientation, aligned with relevant departmental policies and programs. This strategy should consider, among other factors:

  • a definition of CIC’s objectives and expected results in providing pre-departure orientation;
  • a determination of what immigration categories and statuses (family configuration) will receive in-person pre-departure orientation and why;
  • guidelines for how to prioritize locations for the delivery of pre-departure orientation services within targeted immigration categories;
  • a determination of what and how information will be provided to each of the immigration categories prior to departure; and
  • a consideration of the cost of services and value for money.

There is no federal legislation that requires the government to provide pre-departure orientation. In addition, immigration agreements with provinces do not outline the specific responsibilities related to pre-departure orientation. However, a few interviewees felt that it was the federal government’s role to provide pre-departure orientation and to ensure that it was delivered using a uniform and nationally consistent approach. Some provinces are interested in becoming more involved in providing province-specific information and some have already provided ACCC with information, which ACCC has incorporated into its curriculum. The delivery of specific curricula to PNs destined to specific provinces means that the same level of national information is not being provided to all pre-departure orientation participants.

Recommendation #2: CIC should clarify the respective roles and responsibilities for the federal and provincial governments in the delivery of overseas orientation service, including whether province-specific information should be delivered as part of the orientation curriculum, and if so how it should be delivered.

There are governance structures in place to manage each of the pre-departure orientation initiatives, both within each of the delivery agents and between the delivery agents and CIC. While the centralization of responsibility for the contribution agreements within IPMB has added some consistency to how the contribution agreements are managed, there is a lack of coordination between the Branches responsible for the initiatives, particularly regarding decisions related to who will be served by pre-departure orientation and what information will be provided to participants.

Recommendation #3: CIC should put in place a governance structure with clear roles and responsibilities, and accountabilities to allow for effective decision-making between all CIC Branches involved in pre-departure orientation policy and programming.

One of the over-arching issues identified in the evaluation was related to the way in which the initiatives are currently promoted to economic immigrants. Depending on the initiative and location, eligible participants receive different promotional information at different times in the process. This has contributed to a lack of awareness among eligible participants regarding pre-departure orientation.

Recommendation #4: CIC should ensure that there is a consistent and whole-of-CIC approach in place for the promotion of pre-departure orientation to all eligible participants.

CIC’s pre-departure orientation initiatives have different stated objectives, are designed differently, and operate in different environments. Therefore, drawing conclusions with respect to which of the initiatives is more efficient or effective is not appropriate. That being said, in looking at each of the initiatives individually, the evaluation provides some information that can help guide the future implementation of pre-departure orientation.

Recommendation #5: Once CIC has finalized and approved its overseas strategy, it should re-examine the appropriateness of current initiatives to determine how well they align with its new strategy and make adjustments to its current overseas orientation programming as needed.

Features

Page details

Date modified: