Report on the Evaluation of the Delivery of the Canadian Orientation Abroad Initiative

Final Report
Citizenship and Immigration Canada
June 2005

Table of Content

• Executive Summary
• 1. Introduction
  • 1.1 Background
  • 1.2 Structure of the Report
  • 1.3 Overview of the COA
• 2. Evaluation Framework
  • 2.1 Evaluation Issues and Indicators
  • 2.2 Evaluation Methodologies and Data Sources
• 3. Evaluation Findings
  • 3.1 Planning and Design
  • 3.2 Logistical, Managerial and Technical Support
  • 3.3 Training Development and Delivery
  • 3.4 Monitoring and Evaluation
  • 3.5 Results
• Management Response

Executive Summary

The Canadian Orientation Abroad (COA) initiative was implemented in 1998. It provides orientation sessions abroad to assist refugees and others who have been accepted for immigration to Canada in preparing for their move to Canada and to facilitate their integration into Canadian society. The COA is the responsibility of the Settlement Division of Citizenship and Immigration Canada (CIC) and is currently delivered abroad on behalf of CIC by the International Organization for Migration (IOM), under a contribution agreement.

COA sessions are offered to all classes of immigrants and refugees who have been selected for permanent resident status. However, priority is given to Convention refugees (CRs). In addition, CIC has indicated that the participation of women and children is a priority.

An updated evaluation framework for the COA was prepared in 2003–2004. Based in part on that evaluation framework, an evaluation of the COA was conducted. This report contains the findings resulting from the evaluation. Appendix A to this report contains a summary of the issues and questions addressed in the evaluation and the key methodologies employed.

The key findings and conclusions emanating from the evaluation are organized into the following issue areas:

• Planning and Design
• Logistical, Managerial and Technical Support
• Training Development and Delivery
• Facilitation Approach and Facilitator Training
• Monitoring and Evaluation
• Results

I. Planning and Design

Policy Direction

The evaluation assessed the extent to which CIC has provided clear direction to the IOM and other partners (i.e., CIC missions) with respect to such things as COA objectives, roles and responsibilities, targets and budgets, and priority groups.

Key Conclusions

Our overall conclusions in regard to policy direction are as follows:

• In general, CIC has established a clear policy and planning framework for the COA, and the IOM understands CIC’s policies and expectations.
• Not all CIC missions abroad are equally aware of the objectives and benefits of the COA, and not all are equally clear about their roles and responsibilities for referring individuals to the COA.
• There is a lack of systematic knowledge of the orientation needs of immigrants and CRs from different countries and cultural groups.

Delivery Agency Capacity

The evaluation approached this issue in terms of the “reach” of the IOM (its capacity to provide the COA to eligible individuals in a wide range of countries and at a wide range of sites); its overall ability to meet COA attendance targets; and its success in providing the COA to priority groups (CRs, women and children)

Key Conclusions

• The IOM has placed a great deal of emphasis on reaching attendance targets and doing so within budget; this has resulted in an overemphasis on these at the expense of ensuring that CRs – the current priority for the COA – are well served by the initiative.
• While the IOM has responded quite well to year-over-year shifts in immigration movements, it has not responded as well to within-year changes, in part because it lacks consistent access to timely information on changes in caseloads from start-of-year estimates at participating CIC missions.
• The process for monitoring the COA’s progress against attendance targets, while recently made more stringent by CIC, could be further improved.
• The IOM has been successful, at least in the short term, in increasing the participation rates of women in the COA.
• Canadian privacy legislation has been interpreted in a way that prevents CIC from providing the IOM with contact information for potential participants, unless the IOM is arranging medicals or transportation to Canada for these individuals. This has had a negative impact on the participation rates of those in the skilled worker and family classes.

II. Logistical, Managerial and Technical Support

This includes the process for identifying potential participants and referring them to the COA; the adequacy of facilities and equipment; and the appropriateness of ancillary services, such as child care, offered at the various sites.

Key Conclusions

• The process for referring individuals to the COA works very well for CRs, for the most part. However, it is much less effective for other categories because the IOM cannot contact potential participants in these categories to encourage them to participate in the COA.
• While most CIC missions cooperate with and support the IOM, some have been more consistent in ensuring that all non-refugees who are potential participants in the COA are provided with information about the program.
• The equipment at COA sites meets the needs of COA. However, not all classroom facilities were adequate from a learning perspective. The average COA class size approaches or exceeds the classroom capacity at some sites. This detracts from the comfort of participants and, more importantly, from the learning process.
• The IOM’s overall approach to providing ancillary services, such as reimbursement of local travel costs, child minding and snacks, is based on an assessment of need. However, this approach is not consistent at all sites

III. Training Development and Delivery

Training program development and delivery is the issue of whether the delivery agent is providing training that reflects best practices and that supports the learning objectives of the course. To address this issue, the evaluation assessed the course content and materials, and the facilitation approach and the related issue of facilitator training.

Course Content and Materials

Key Conclusions

• The content of the COA is highly appropriate with respect to CRs. However, the course is not as well matched to the needs of SWs in terms of content, sub-topic focus or level of detail on some topics.
• The core Facilitator’s Guide is out of date, necessitating the facilitators to maintain large quantities of updated material as a supplement to the manual and creating the potential for inaccurate information to be provided.
• The main COA video is out of date with respect to both the accuracy of some of the content, and the look and feel. However, CIC is in the process of updating this videotape.
• The IOM has little support from in-Canada organizations to assist it in keeping course materials up to date, nor is there in place a process for regular review and updating of core course materials.
• The COA has, until recently, been severely limiting distribution of the COA participant’s manual. This could potentially have negative impacts on the benefits of the program, especially for CRs.
• The COA participant’s manual has been translated into only a few other languages, which impairs its usefulness, especially for CRs.

IV. Facilitation Approach and Facilitator Training

With respect to the COA, the research identified four potentially significant factors that could affect the effectiveness of the course: commitment to facilitated learning; facilitator training; class size; and course length.

Key Conclusions

• The facilitation approach to learning is known to be a highly effective approach to adult education and, as such, should be the preferred approach to learning of the COA.
• While COA staff are strongly committed to the facilitation approach, cultural differences among participants present a challenge to gaining active participation. As well, CRs may be reluctant to be vocal or questioning, out of concern for jeopardizing their status.
• Most COA facilitators have appropriate backgrounds for their roles, although a few lack direct knowledge of Canada.
• COA facilitators all have appropriate training to enable them to implement a facilitated approach to learning.
• The COA classes are too large and are an impediment to the facilitated approach to learning.
• The reduction of the COA course to one or two days’ duration at virtually all sites seriously undermines the facilitated approach and the capacity of facilitators to cover the material at an appropriate level, especially in the case of CRs.

V. Monitoring and Evaluation

The findings in relation to these questions are organized under two headings: participant feedback; and monitoring and reporting on delivery.

Key Conclusions

The Evaluation I (end of course) and Evaluation II (after arrival in Canada) questionnaires are not useful tools, in their current forms, for obtaining participant feedback on the COA for the following reasons:

• the content of the COA questionnaires does not appear to be driven by clear objectives for their use;
• the questions are too general and they are virtually identical on both questionnaires.

COA staff at some sites are skeptical of the value of the Evaluation I questionnaire and have concerns regarding the difficulties participants with poor literacy skills have in completing it. Consequently, they do not distribute it.

Only a very small percentage of course participants complete and mail in the Evaluation II questionnaire, which greatly diminishes its value as an unbiased reflection of the population of participants.

Monitoring and reporting of the delivery of the COA, while adequate, is not ideal. Reporting is not clearly focused on providing CIC with data and information useful for financial and performance management or for planning.

The IOM has had difficulty, in recent years, in providing quarterly and annual reports in a timely fashion.

CIC officials do not visit COA sites periodically as part of their monitoring of the COA, and CIC has not suggested site visits by the IOM Project Manager. This lack of site visits is an impediment to the effective management and monitoring of the COA.

VI. Results

The findings in relation to this question are focused on the contribution of the COA to improved participant knowledge and facilitated integration.

Key Conclusions

• Based on limited evidence from stakeholder interviews and analysis of a small sample of the COA participant after-arrival questionnaire, the COA does appear to improve participants’ knowledge of topics that are important to the settlement experience.
• The lines of enquiry adopted in the evaluation did not provide a basis for firm conclusions on whether or not the COA facilitates the integration of participants into Canadian society.

1.  Introduction

1.1  Background

The Settlement Division of the Integration Branch at Citizenship and Immigration Canada (CIC) is responsible for facilitating the integration of immigrants to Canada. One initiative it administers in this regard is Canadian Orientation Abroad (COA). COA, which was established in 1998, is intended to assist refugees and others who have been accepted for immigration to Canada to prepare for their move, and to support their integration into Canadian society. In support of this objective, the COA offers orientation sessions at various sites abroad.

An updated evaluation framework for the COA was prepared in 2003–2004 and an evaluation of the COA was conducted in 2004–2005. This report contains the findings and conclusions of that evaluation.

1.2 Structure of the Report

This report contains three chapters. The remainder of Chapter One contains a brief overview of the COA. Chapter Two contains the evaluation framework on which the COA evaluation was based. Chapter Three contains the evaluation findings and conclusions with respect to the evaluation issues.

1.3 Overview of the COA

Objectives

The Immigrant Settlement Assistance Program (ISAP) facilitates the adaptation, settlement and integration of newcomers to Canada so that they may become participating members of Canadian society as quickly as possible. To support this process, ISAP funds organizations to provide services such as reception, orientation, interpretation, solution-focused counselling and job search.

ISAP funds also support the COA. Under this initiative, orientation sessions abroad provide future immigrants and refugees with an idea of what life in Canada is like. With a more realistic view of life in Canada, immigrants are expected to be better prepared to face the initial demands of adaptation and settlement and to integrate into Canadian society more easily.

Eligibility

COA sessions are offered to all classes of immigrants and refugees who have been selected for permanent resident status. However, places are limited and are awarded, in order of priority, to Convention refugees (CRs), then to Independent Class immigrants (primarily Skilled Workers [SWs]) [note 1] and finally, to Family Class (FC) immigrants. Places are also made available for Live-in-Caregivers (LICs), who are issued visas under the Temporary Worker Program. These individuals are eligible for landed immigrant status after a period of time in Canada.

In the 2003–2004 IOM proposal for services, it was noted that the participation of women would be a priority and the referral of children would be encouraged where family groups can attend orientation sessions together.

COA Management and Delivery

CIC sets the overall policy and program-planning framework for the initiative, funds it and monitors the performance of the delivery agency.

Since the inception of the program in 1998, CIC has signed contribution agreements with the International Organization for Migration (IOM) to deliver orientation sessions abroad on behalf of CIC. The amount of the contribution agreement was approximately $900,000 in 2003–2004 and $800,000 in 2004–2005. The IOM enrols participants, hires and trains facilitators, delivers the courses, and provides regular reports to CIC on participant statistics and costs.

According to IOM interviewees, training modules were initially developed for Convention refugees (five days), Family Class (three days) and Skilled Workers (one to two days). However, these courses were subsequently reduced to three days, two days and one day, respectively. Topics include introduction to Canada; the settling-in period; employment; rights and responsibilities; climate; finding a place to live; living in a multicultural society; the cost of living; family life; education; communications; and adaptation to Canada. In most cases, the sessions are delivered in the language of the participants.

COA locations are determined by CIC national headquarters in consultation with the Canadian missions, which refer CRs to the sessions. At some sites, missions also provide information on COA to other classes of migrants, such as LICs, SWs and FC immigrants. Since 1998, courses have been delivered in such places as Vietnam, Kenya and the Balkans. Sessions have also been provided in Egypt, Djibouti, Tanzania and Ethiopia. New locations may come on stream and others may be discontinued, depending on the greatest need and the resources available. Table 1.1 below shows the location of COA delivery sites.

Training is delivered either at a fixed site with permanent staff, or through satellite sites. Regular satellite sites use regular on-call staff, as required, and fixed training facilities, although these are normally rented only for the duration of the training. Irregular satellite sites have no fixed training room and no regular on-call staff. Training may take place only from time to time (sometimes only once) – usually because there is a considerable CR caseload ready to travel to Canada on a one-time or irregular basis. In 2004, for example, one-time sessions took place in Tajikistan and Kyrgyzstan.

2.  Evaluation Framework

This chapter outlines the evaluation issues and questions that were addressed in the COA evaluation and describe the methodologies that were employed to assess these issues and questions.

2.1  Evaluation Issues and Indicators

Appendix A contains a summary of the issues and questions addressed in the evaluation and the key methodologies employed. In the course of the development of the Evaluation Framework, CIC Settlement Division officials indicated that the evaluation should place a good deal of emphasis on the performance of the delivery agent in delivering the COA. Consequently, the evaluation questions and methodologies reflect this emphasis.

It was apparent that certain questions relating to the capacity of the IOM to meet program targets, its capacity to respond to shifts in the movement of CRs and immigrants, and the accessibility of the COA to priority groups are highly interrelated. Accordingly, questions from the original evaluation framework have been regrouped and consolidated under the following issue areas.

Planning and Design

Policy Direction

• Has a clear policy and planning framework been established to guide design and delivery?
• Does the delivery agent have a clear understanding of objectives, priorities, policy guidelines and standards, and his or her role and responsibilities?

Delivery Agency Capacity

• Does the delivery agent have the capacity to deliver training at targeted locations and to priority groups? How many immigrants have received training compared to targets?
• Does the initiative have the capacity to respond effectively to shifts in immigration and refugee trends? How effectively has the COA adjusted to shifts in immigrant and refugee trends?
• How accessible is training to immigrant groups who could benefit most from it?

Logistical, Managerial and Technical Support

• Is the process for identifying and referring immigrants to orientation training effective?
• Is efficient and effective use being made of facilities and equipment?
• Are appropriate ancillary services provided at program delivery sites?

Training Development and Delivery

• Do the course content and materials support the learning objectives?
• How effective is the facilitation approach to learning?
• Are facilitators adequately trained?

Monitoring and Evaluation

• Does the COA obtain participant feedback on course content and delivery? What use is made of participant feedback to improve program design or delivery?
• What aspects of program delivery are monitored and reported on by the delivery agent? How is this information used to improve planning, design and delivery?

Results

• To what extent has the COA contributed to improving participants’ knowledge of Canada or settlement issues, programs and services?
• To what extent has integration into Canadian society been facilitated for immigrants who participated in the COA?

2.2  Evaluation Methodologies and Data Sources

A variety of data collection and research methodologies were used for the evaluation of the COA. A consultant with expertise in adult education assisted in the development of the data collection instruments used in the evaluation and in the conduct of in-Canada interviews. The key methodologies were the following.

2.2.1  Review of Documents and Data

Appendix B contains a detailed list of the documents and data reviewed. Key documents reviewed in the course of the evaluation included CIC policy, planning and financial documents and data; CIC/IOM contribution agreements; IOM financial and statistical data and reports on the COA.

2.2.2  Interviews and Focus Groups

Appendix C contains a detailed list of the individuals interviewed. These included officials from CIC HQ and regional offices; officials at CIC missions abroad; IOM officials; in-Canada settlement service providers; and officials of the United States Overseas Program (USCO), an orientation program offered by the U.S. government.

The evaluators conducted five focus groups with COA participants at the four sites visited. In Cairo, the participants were CRs; in Nairobi, a mixture of FC and CRs; and in Islamabad, one group consisted of CRs and another, of SWs. In Manila, the focus group members were LICs.

2.2.3  Site Visits

Four COA main sites were visited—Cairo, Nairobi, Islamabad and Manila—and a satellite site in Peshawar, Pakistan. At the sites, in addition to the focus groups and staff interviews, we:

• reviewed the COA facilities and equipment;
• reviewed course instructional and supplementary materials; and
• observed the delivery of the COA in actual classes and assessed against adult education criteria.

3.  Evaluation Findings

In this chapter, we present the evaluation findings, organized according to the five main evaluation issues identified in Chapter Two:

• Planning and Design
• Policy Direction
• Delivery agency capacity
• Logistical, Managerial and Technical Support
• Training Development and Delivery
• Monitoring and Evaluation
• Results

3.1 Planning and Design

Five evaluation questions were identified under this evaluation area. The first two questions relate to the policy direction provided by CIC to the IOM and the next three have to do with the capacity of the IOM to be an effective delivery agent for the program. Thus, the discussion is organized under these two groups of questions. Overall conclusions are presented following the discussion.

3.1.1  Policy Direction

It is important for the success of any government initiative that those involved in managing and delivering it have a common understanding of the overall objectives of the initiative. This includes the intended priority target group or groups; delivery policies or standards; and the roles and responsibilities of management and the delivery agent. A common understanding of these is especially important when the initiative is delivered by an external delivery agency, as is the case with the COA.

The findings in these areas are organized under the following themes.

3.1.1.1  Objectives

The research, including interviews with the CIC Settlement Division and the IOM COA Project Manager in Manila, and a review of CIC and IOM documents and correspondence indicated a common understanding of the main objectives of the COA. The key document outlining this common understanding is the contribution agreement between the IOM and CIC. This document describes the COA objective as “to facilitate immigrant adaptation and eventual integration, through the provision of orientation sessions abroad.” Both CIC and IOM officials with whom this issue was raised felt that CIC provided clear direction regarding the objectives of the COA.

3.1.1.2  Roles and Responsibilities

The contribution agreement also clearly defines the roles and responsibilities of the IOM in the delivery of the COA. Services to be provided include delivery and setup of the COA and related services, an effective referral process, management, record keeping, child minding and logistical support. Both parties to the agreement demonstrated a clear understanding of their respective roles and responsibilities.

While all the CIC mission officials interviewed were generally knowledgeable and supportive of the COA; officials at two missions were more familiar with the details of the initiative than others and worked more closely with IOM officials. All of those interviewed understood, in general terms, the role of the missions in referring potential participants to the program. It became clear from interviews, however, that there is no systematic process in place whereby CIC HQ ensures that mission officials in regions where the COA is delivered are thoroughly familiar with the COA and with their responsibilities in relation to it. One manifestation of this was the question raised by one mission official as to whether it was appropriate for CIC missions to be informing immigrants about the COA.

This lack of systematic direction from CIC HQ has had an impact on the effectiveness of the referral process at some sites. It may have also affected the ability of the IOM to adjust its attendance targets for some sites when the number of immigration cases processed at referring CIC missions differs from start-of-year targets. Both these issues are discussed in more detail later, in the sections on the referral and targeting processes, respectively.

IOM officials indicated that some missions work closely with the IOM to ensure that immigrants other than refugees are informed about the program. A few (notably Moscow) also assist the IOM in course logistics in locations with no permanent site. However, some missions are much less conscientious about ensuring that the COA information is included with visas. The IOM COA Project Coordinator indicated that if CIC mission officials choose not to cooperate with the IOM, the latter cannot effectively deliver the program in that region. This has been a problem once or twice in the past number of years.

Overall, while those involved in the COA are generally clear about their roles and responsibilities, CIC mission officials do not always appear to have a clear understanding of the COA and how they can help ensure its success. This could be addressed through closer communication between the IOM, the CIC Settlement Division at HQ and CIC’s International Region, to ensure that missions are fully cognizant and supportive of the initiative.

3.1.1.3  Targets and Budgets

The sites at which the COA will be delivered and the attendance targets for the COA are referenced in the contribution agreement between CIC and the IOM. These targets are negotiated between CIC and the IOM, based on the proposal submitted by the IOM for the coming year.

Although the overall process of setting attendance targets and budgets is not an issue, the IOM has experienced difficulties in achieving targets at some sites in recent years. This problem is discussed in more detail in a later section of this report, addressing the issue of the IOM’s capacity to provide training at targeted locations and to priority groups.

From the point of view of policy direction, CIC has responded to this issue by including more stringent reporting requirements on the IOM so that slippages in targets can be identified earlier. This would enable unused funds to be allocated to other sites or allow a reduction of contribution agreement funds, which would then be allocated to other settlement programs.

3.1.1.4  Priority Groups

CIC has indicated to the IOM that CRs are to be given priority under the COA, then SWs and, finally, FCs. In addition, the IOM is to encourage participation in the COA on the part of women and children. Although no formal source for this policy was found in CIC’s policy documents, this order of priority is articulated in the information sheet about the COA on the CIC website. As well, both IOM and CIC officials interviewed indicated that the above reflects the COA policy.

CIC has provided clear direction, in general terms, to the IOM with regard to priority groups. However, there is no consensus among CIC and IOM officials and among in-Canada settlement service providers as to whether CRs should always be the priority group. Among those interviewed, views varied as to which immigrant category benefits the most from this sort of training. Some indicated that CRs are the group in greatest need of orientation training. Others indicated that CRs have access to a full year of settlement assistance, while SWs have limited support once they arrive in Canada.

One reason for the diversity of views is the lack of systematic studies of the specific needs of immigrants by immigration category or other characteristic, such as literacy, education, English or French language skills, cultural background and access to other sources of information.

Given this lack of systematic information, it is difficult to determine whether the COA is being targeted at the appropriate groups. Based on observations of COA sessions with CRs, SWs and LICs, it appears that CRs have the greatest need for predeparture orientation training in that they tend to be less educated, have poorer English language skills, and possess less knowledge of many aspects of Canadian society than do SWs or LICs. By contrast, the SWs in a focus group in Pakistan were, for the most part, extremely well educated and computer-savvy, had good English-language skills and were already quite knowledgeable about western society, if not specifically Canada.

CIC needs to decide whether to set priorities based on immigration classification (i.e., CRs) or need. If they choose the latter, then the program needs to take a more systematic approach to identifying regions and cultures of high need. IOM officials are very experienced in the field and are quite knowledgeable about the needs of CRs and others in different regions. Nevertheless, it would be very helpful in making decisions on the targeting of the COA if a more rigorous basis could be found for determining the needs of different groups, or of immigration categories within specific regions. Approaches to establishing this basis could include:

• much more extensive communication between IOM officials involved with the COA, with CIC HQ and regional offices, and with in-Canada settlement service providers. This could be in the form of participation in conferences and workshops; and e-mail and telephone contacts;
• the conduct of research to assess the needs of groups, based on the country and culture of origin, literacy, education and other qualitative data, and communication of the results of this research to the IOM.

Of course, CIC may wish to pursue the current policy of giving CRs priority for reasons other than just need. In this context, it is noteworthy that the IOM is considering offering, quite independently of government sponsorship, an orientation course to SWs and other non-CR immigration categories for individuals emigrating to Canada and other western countries. These courses would be optional and provided on a cost-recovery basis. If the IOM goes ahead with this concept, then the COA could likely focus almost exclusively on CRs, except at sites where the IOM-sponsored program is not available for other categories. As well, CIC might have to reach agreement with the IOM not to compete with the Canadian program at sites where SWs or other categories are necessary to provide a “critical mass” for the COA at sites that would otherwise not be viable.

Key Conclusions

Based on the above, the overall conclusions with regard to program policy direction are as follows:

• In general, CIC has established a clear policy and planning framework for the COA, and the IOM understands CIC’s policies and expectations.
• Not all CIC missions abroad are equally aware of the objectives and benefits of the COA, and not all are equally clear about their roles and responsibilities for referring individuals to the COA.
• There is a lack of systematic knowledge of the orientation needs of immigrants and CRs from different countries and cultural groups.

3.1.2 Delivery Agency Capacity

Three main evaluation questions were identified under this issue, as discussed in Chapter Two. These questions are addressed under the following themes.

3.1.2.1 Reach

The IOM is an international agency with offices in over 200 countries, including virtually all countries that are sources of migrants for Canada. It provides a wide range of service to migrants around the world on a cost-recovery basis on behalf of a large number of member states. As an international agency that is viewed as non-political, the IOM has a high degree of credibility with both member and non-member states.

The IOM, in addition to providing orientation courses for Canada, arranges transportation to Canada for CRs and, in a few countries, for FCs. It also provides immigration medical examination services on Canada’s behalf in certain countries. Because of these latter two roles, the IOM has access to contact information on CRs in every country where COA is provided. This provides the IOM with a unique advantage in terms of informing CRs about COA and encouraging them to participate in the program. There is currently no other international organization that has the combination of the IOM’s reach, in terms of potential sites, and its advantage in terms of access to CR contact information.

Under current Canadian privacy legislation, CIC missions abroad cannot provide contact information on immigrants to external organizations. For this reason, the IOM cannot obtain contact information on FC or SW category migrants unless it is also providing transportation assistance to them. Thus, if CIC were to contract with another international organization for the provision of orientation training abroad, it would have to address this issue.

Currently, the IOM is also delivering orientation training for the U.S., Australia, Norway and Finland. This provides another advantage to the IOM in terms of its reach. There are a number of sites where the Canadian and other programs share facilities and equipment. This has enabled the IOM to offer COA sites where, if only the COA was being offered, the site would not be viable. At other sites, it has enabled the IOM to provide the COA at a lower cost than would otherwise be the case.

3.1.2.2  Targets

As noted earlier, the sites at which the COA will be delivered and the attendance targets for the COA are referenced in the contribution agreement between CIC and the IOM. These targets are negotiated between CIC and the IOM, based on the proposal submitted by the IOM for the coming year. Attendance targets traditionally have been set at between 9,000 and 12,000. The IOM establishes its targets by obtaining estimates from CIC missions of the number of CRs and other immigrants they expect to process in the coming year. In some years, sites may be added or dropped, depending on the expected volume of CRs and other immigration categories. In 2004–2005, sites in Syria, Jordan and Vietnam were closed.

Table 3.1 shows the attendance targets and actual number of COA participants for the last three years. As can be seen from this table, while the program failed to meet its target in 2002–2003, it exceeded its target in 2003–2004 by a substantial amount and will likely not quite reach its original target for fiscal year 2004–2005. However, during this period, the targets themselves have been decreasing, in response to fewer than expected referrals to the COA by CIC missions.

A review of COA site data on participation vs. targets revealed that the major factors in the shifts in volume in the last couple of years were:

• a dramatic increase in the number of LICs attending the COA in Manila. This group’s participation increased from 728 in 2002–2003 to 4,044 in 2003–2004 and 3,505 in the period April to December, 2004;
• closing of the sites in Syria and Jordan;
• a significant fall-off in volumes in Egypt from 2003–2004 to 2004–2005;
• peaking of volumes in 2003–2004 in Iran and Kenya, followed by significant decreases in volumes; and
• a decrease in Vietnam from 2003–2004 to 2004–2005.

According to IOM officials, the failure to achieve program targets in two of the last three years reflects the fact that the number of CRs referred to the COA by CIC missions is less than the estimated numbers originally provided by the missions to the IOM for the purposes of setting the budget and attendance targets. IOM officials did not know whether this reflected a reduction in the number of CRs processed by missions in countries with COA sites or whether, due to changes in mission staff or for some other reason, the referrals are not being made. Because one COA site draws participants from more than one referring mission, and because of variances between the visa processing dates established by the missions and the dates of participation in the COA, it is difficult to verify which of these has been the primary factor. However, based on the available data, it would appear that decrease in the number of cases processed at the missions, compared with targets, has been the main factor.

While CRs represent only 31% of the IOM caseload, sites with a large proportion of CRs (Pakistan, Iran, Egypt, Kenya, Syria, Jordan) account for a significant proportion of the variance between actual participants and targets in recent years.

This situation likely will be improved through the more stringent obligations, agreed upon by IOM in the most recent contribution agreement, to report earlier on projected shortfalls in targets.

The CIC Settlement Division should also obtain regular information on the number of CRs and other immigration cases processed compared to start-of-year levels plans for CIC missions that refer individuals to the COA. This information could be used to validate the IOM’s monitoring reports.

3.1.2.3  Priority Groups

Convention Refugees

While both CIC and IOM officials agree that CRs are a priority under the COA, the percentage of participants who are CRs has been declining in recent years and the percentage who are Independent Class has been increasing, as illustrated in Table 3.2. The percentage of participants who are FC has remained fairly stable.

IOM staff indicated that it is extremely difficult to give CRs priority on a global basis for several reasons. Chief among these is that CR populations tend to be small and scattered so that it is difficult to achieve a critical mass at some sites. According to the IOM COA Project Manager, it takes a critical mass of about 1,000 participants per year to make a site viable, even when facilities are shared with other countries, as is the case with most COA sites. Consequently, the IOM has closed several sites in the last few years, notably Syria and Jordan, where the clientele, although relatively small, was virtually 100% CRs. During this same period, the IOM Project Manager relocated from Pakistan to Manila, which is the IOM administrative headquarters, and has provided orientation to increasing numbers of LICs. According to IOM officials, these are not a high-need group.

The problem at sites with low volumes of potential participants is that low volumes of CRs make it difficult to justify a COA presence. When there are only 200 to 300 CRs approved over the course of a year, it is very difficult to be able to put together a full class of participants at any given point in time.

Some sites could be viable if a greater number of SWs or FC immigrants participated in classes to supplement the CR caseload. However, if sufficient numbers of these are to be recruited, it will be necessary to find a way to address the privacy issue so that CIC missions can provide contact information to the IOM.

A further problem at low volume sites, according to IOM officials, is that the cost per participant increases significantly due to the inefficient use of facilities, equipment and staff. With some exceptions, the larger sites tend to have lower per capita costs (see Table 3.3).

The raw per capita costs are not very useful as they allocate all the management costs of the global COA Project Manager and the former Regional Coordinator, based in Damascus, to the Manila and Damascus sites, respectively. Strictly speaking, these costs should be allocated across all sites on a proportionate basis.

The adjusted per capita costs reflect only the direct costs at each site and exclude these management costs. Consequently, while they do not represent the full allocated costs of the COA at each site, they are more accurate with respect to the relative costs at each site.

In reviewing the program data and budgets for the COA sites visited, it would appear that the following are the key factors that explain the uneven correlation between the volume of training and the raw per capita costs per site:

• Some sites provide more days of training per person, on average, than others. Egypt and Kenya, for example, provide 2.7 and 2.4 days, respectively, while Pakistan, despite a significant CR caseload, provides only one day of training per person;
• Salary costs are significantly higher at some sites. In Kenya, for example, salary costs in 2002–2003 were about $67,000 while in Pakistan, they amounted to only $22,500. This reflects the much higher proportion of salaried employees in Kenya versus on-call facilitators in Pakistan, as well as higher salary scales in Kenya;
• Travel costs at some sites, notably Kenya, were much higher than at other sites. This reflects the extensive training at satellite sites by Kenya-based staff.

In conclusion, while there is some merit to the argument that per capita costs increase significantly at low volume sites, there are other variables that affect costs, some of which could contribute to offsetting cost increases resulting from low volumes.

One factor that appears to contribute to lower costs is the use of on-call facilitators, who are paid by the hour or by the session and only for the hours they are needed. CIC and the IOM may wish to look into the feasibility of using on-call facilitators in mobile teams, based in neighbouring or nearby sites, to provide training at some low volume sites. This might necessitate close cooperation with CIC missions, in an attempt to “cluster” the issuance of a sufficient number of visas within a specific time period to make trips worthwhile.

If CRs continue to be a priority, CIC and the IOM may have to accept higher per capita costs at some sites and, by implication, lower program targets, in order to focus training on sites and groups that are considered high need. At the moment, almost one-third of the COA caseload is LICs in the Philippines. IOM officials do not consider these a high-need group.

Women

As reflected in the IOM program proposal in 2003–2004, the IOM and CIC have agreed in the last couple of years to make the participation of women in the COA a priority and to encourage children to participate where family groups can attend sessions together.

We were unable to obtain data on the participation of children in the COA. However, Chart 3.1 shows the rate of participation by women in the last few years.

As this chart illustrates, participation rates by women had been decreasing – from just over 50% of participants in 2000–2001 to 48% in 2002–2003. However, in 2003–2004, this rate increased to 54%, indicating that the IOM has been successful in encouraging more women to participate in the COA

Chart 3.1

3.1.2.4  Response to Shifts In Convention Refugee and Immigrant Trends

Because of its wide reach and the way it operates, the IOM can respond quite quickly to shifts in the source countries of CRs and migrants. The IOM operates in over 200 countries around the world, including virtually all countries that are sources or are likely to be sources of CRs or immigrants to Canada. In addition, because most IOM staff members are contract employees, they can be released with very limited notice if required. Many IOM staff members are highly mobile and tend to accept assignments at different sites around the world of their own volition.

Further, because the COA often shares facilities and equipment at sites with other orientation programs, it is not always necessary to deal with the disposal of unneeded offices or equipment when sites are closed.

In the case of the COA, recent adjustments have included:

1. closing sites in Albania, Yugoslavia, Bosnia and Croatia between 2001 and 2003 in response to reduced caseloads;
2. moving the Regional Coordinator in the Baltic to Damascus in 2003;
3. closing offices in Syria and Jordan last year in response to reductions in caseloads;
4. closing the site in Vietnam this year; and
5. responding to ad hoc requests from CIC for one-time training sessions in Kyrgyzstan (March 2004) and Tajikistan (December 2004).

Overall, the IOM has adjusted fairly well, from one year to the next, to shifts in CRs. As noted in the section on program targets, however, they have not been as effective in responding to lower than expected volumes occurring within a single year. This has led to a failure to meet program targets in a couple of years.

Within CIC there is no formal system in place for ensuring that the IOM gets regular updates on changes in expected caseloads at the CIC missions that refer people to the COA. Consequently, the IOM has to rely on informal working relationships with CIC mission staff to obtain this information. While this approach has worked well at some sites, it has not worked well at others. CIC should consider a more formal system through which the IOM receives regular and timely updates on variances in caseloads.

Key Conclusions

The issues of program targets, priority groups and capacity to respond to shifts are highly inter-related, as is that of access of priority groups to training. The key conclusions on these issues are:

• The IOM has placed a great deal of emphasis on reaching attendance targets and doing so within budget. This has resulted in an overemphasis on these at the expense of ensuring that CRs – the current priority for the COA – are well served by the initiative.
• While the IOM has responded quite well to year-over-year shifts in immigration movements, it has not responded as well to within-year changes, in part because it lacks consistent access to timely information on changes in caseloads from start-of-year estimates at participating CIC missions.
• The process for monitoring the COA’s progress against attendance targets, while recently made more stringent by CIC, could be further improved.
• The IOM has been successful, at least in the short term, in increasing the participation rates of women in the COA.
• Canadian privacy legislation has been interpreted in a way that prevents CIC from providing the IOM with contact information for potential participants, unless the IOM is arranging medicals or transportation to Canada for these individuals. This has had a negative impact on the participation rates of those in the skilled worker and family classes.

3.2  Logistical, Managerial and Technical Support

This includes the process for identifying and referring potential participants to the COA; the adequacy of facilities and equipment; and the appropriateness of ancillary services, such as child minding, offered at the various sites.

3.2.1  Referral Process

The process for referring individuals to the COA varies to some extent, depending on the immigration category and the site. For refugees at virtually all sites, the IOM receives the travel documents (visas) for refugees whose transportation to Canada is being assisted by the IOM. When they receive the travel documents, the contact information is automatically provided to the COA staff at the appropriate site. In many refugee cases (and some family reunification cases), the IOM conducts medical examinations of the individuals on behalf of Canada and can take the opportunity to invite them to participate in the COA.

With respect to most immigrants and temporary workers (SWs, FCs and LICs), the IOM relies on the CIC missions processing cases in the region served by the COA site to include COA information flyers in the letter accompanying the visa. These individuals contact the COA to express interest in participating.

For refugees, the referral process works quite well, according to IOM officials and CIC mission officials. Because the IOM is able to directly contact refugees, virtually 100% of the refugee populations served by COA sites attend the COA.

With respect to other types of migrants, interviewees indicated that at some sites, CIC mission staff do not, or do not on a consistent basis, distribute the COA information flyer. Another problem with respect to these categories is the fact that Canadian privacy legislation prevents CIC from providing the IOM with contact information for migrants unless the IOM needs that information to carry out transportation or medical examination services on behalf of the Canadian government. As a result, participation rates for non-CRs are much lower than for CRs.

CIC could help to ensure that all eligible individuals, including CRs, in areas with access to the COA are made aware of the course by developing and implementing a process, in conjunction with the International Region, to ensure that all immigration officers at sites that refer clients to the COA are fully aware of the course and their responsibilities in the referral process.

3.2.2 Facilities and Equipment

Four COA sites were visited in the course of the evaluation. The findings and conclusions in this section are based on observations at these sites, interviews with COA facilitators and other IOM officials, and focus groups conducted with COA participants at the four sites.

All the sites visited had television monitors and videocassette players, which are required to view most of the audiovisual materials that support the course content. As well, similar equipment is available at most satellite sites, including refugee camps, according to IOM site officials interviewed. At most COA sites, facilities and equipment are shared with other orientation programs. In Cairo, the COA is located at a separate office from the IOM, and facilities are shared among Canadian, U.S., Australian and Norwegian programs. The equipment is adequate, although there will be a need to upgrade to DVD format in the near future as new material will most likely be available in this format.

In general, the COA focus groups had little to say about the equipment used for the course. At one site, a few participants commented that the COA should be providing a “multimedia” presentation. However, others disagreed, saying there is no need for TV in an interactive session.

Participants were generally positive about the facilities as well, with the exception of the focus group in Peshawar, which consisted of skilled workers. A number of participants in this group were critical of the room, indicating that the air conditioning was quite noisy and the room was not comfortable.

Among IOM site officials interviewed, the majority view was that equipment and facilities were adequate. However, several mentioned that classroom capacity is often met or exceeded during sessions. Other concerns raised included background noise from the air conditioning, the stability of the electrical supply, VHS tapes that stop in the middle and the high temperature in classrooms at times.

Based on classroom observations, the issue with respect to equipment is not the equipment itself but how effectively it is used. There is a plethora of audiovisual materials at most sites. However, as COA courses currently average less than two days, they can be used only sparingly. Although some of these audiovisual materials are dated, most could be used effectively to reinforce concepts taught in the course, especially for groups for whom these concepts are new and very different from concepts of their own culture.

Both facilitators and focus group participants were of the view that COA facilities are accessible to participants. Most are located in central areas with good public transportation, and most serve individuals within the urban area. However, participants do sometimes come from neighbouring or even distant locations to participate. In most of the countries visited, the COA is offered in different parts of the country. The exception is in the Philippines, where the course is offered only in Manila.

The classroom facilities used for the COA range from inadequate to quite adequate in terms of the physical layout, size, noise levels, climate and other factors. A summary table, describing and assessing the classroom facilities for the sites where the COA is currently offered, is provided in Appendix D.

Most of the classroom facilities being used for the COA are adequate and some are quite good. The main problem with the facilities is that the IOM has been stretching the classrooms beyond their capacity in terms of numbers. The IOM estimates of classroom capacity (see Appendix D) overestimate the capacity of these rooms. Based on the site visits, the numbers that can be accommodated comfortably and that are compatible with a student-centred, facilitated approach to learning are lower than the numbers cited by the IOM. In addition, it appears that the COA exceeds the maximum class size capacity.

CIC needs to work with the IOM to establish guidelines for average class sizes. While it may be necessary on occasion to fill a classroom beyond its capacity for logistical reasons, this should not be the norm.

3.2.3 Ancillary Services

The contribution agreement between CIC and the IOM requires, as one of the IOM’s performance objectives, that it provide logistical, managerial and technical support for the COA, including the provision of snacks and child minding. The agreement is not specific as to whether these are to be offered on a universal basis or whether the IOM is to exercise its judgment with respect to the provision of these ancillary services.

Based on interviews with IOM officials during the site visits and on a review of the IOM, the following are the current practices at COA sites with respect to ancillary services.

Snacks

Currently, the IOM either provides snacks or a daily food allowance to all refugees at sites where the COA is offered to refugees. In addition, the IOM provides snacks for breaks at other sites where the COA is offered. The cost of this service is quite minimal. According to IOM budget documents, in 2003–2004 the total costs for food and child minding combined at all sites amounted to less than $25,000.

The focus group participants had no comments on the provision of snacks or meal allowances.

Child Minding

Of the four sites visited, only one—Cairo—provided child-minding services. The IOM is now planning to offer child minding in Islamabad. There are no child-minding services in Manila or Nairobi, although child-minding services are provided at satellite sites in refugee camps (Kakuma and Dadaab) because of the large number of children at the camps.

IOM officials were generally of the view that either child-minding facilities are not needed or they don’t have the budget to provide them. At some sites, for example Manila, participants are believed to have large extended families that can assist with child care for short periods.

Most of the COA participants in focus groups had little or nothing to say on this issue. At two sites, however (Kenya and Pakistan), a small number (2 or 3) of focus group participants indicated that child-minding services would have been appreciated. At one site, two participants left at noon and missed part of the afternoon session because they went home to check on their children.

Transportation

The IOM provides a small subsidy to refugees, averaging about $3 a day per participant, to help cover the costs of local transportation. Of the four sites visited, only two—Cairo and Kenya—provide this subsidy. Manila trains LICs, SWs and FC immigrants who, it is assumed, can afford to pay the local travel costs. Although Pakistan’s clientele is mostly SWs at present, they do train some refugees. It is not clear why the IOM does not reimburse local travel costs to refugees in Pakistan.

Among the focus group participants, transportation was mentioned as an issue only in Islamabad. One individual at this focus group indicated that assistance with transportation costs would have made it easier to attend. Others in the group nodded agreement but had no verbal comments.

Key Conclusions

Our main conclusions with respect to the above issues are:

• The process for referring individuals to the COA works very well for CRs, for the most part. However, it is much less effective for other categories because the IOM cannot contact potential participants in these categories to encourage them to participate in the COA.
• While most CIC missions cooperate with and support the IOM, some have been more consistent in ensuring that all non-refugees who are potential participants in the IOM are provided with information about the program.
• The equipment at COA sites meets the needs of the COA. However, not all classroom facilities were adequate from a learning perspective. The average COA class size approaches or exceeds the classroom capacity at some sites. This detracts from the comfort of participants and, more importantly, from the learning process.
• The IOM’s overall approach to providing ancillary services, such as reimbursement of local travel costs, child minding and snacks, is based on an assessment of need. However, this approach is not consistent at all sites.

3.3 Training Development and Delivery

Training development and delivery is the issue of whether the delivery agent is providing training that reflects best practices and that supports the learning objectives of the course. The findings are organized under the following headings.

• Course Content and Materials
• Facilitation Approach and Facilitator Training

3.3.1 Course Content and Materials

3.3.1.1 Course Content

The COA course guide was originally developed in 1998 by the current COA Project Manager. The course guide contains the core content of the COA and is available on the CIC website as the “Facilitator’s Guide to Pre-Departure Orientation Modules for Immigrants.” This is the document that facilitators use as the foundation document for delivering the course.

The COA course curriculum was developed by professionals with significant experience in providing orientation to immigrants and CRs under the Language Instruction for Newcomers to Canada Overseas program, the precursor to the COA. This curriculum is adapted to the perceived needs of the group that is receiving the orientation. This is more a matter of emphasis, time allotted, specific content and level of detail allocated to individual topics than actual differences in the topics covered.

There should be a high degree of comparability among different orientation programs with respect to the topics addressed, even though these courses are offered by different countries. Table 3.4 provides a comparative overview of the topics addressed by the COA course; the “First Steps” Program, an orientation package available to government-assisted refugees after they arrive in Ontario, Canada; and “USCO,” the cultural orientation program offered abroad by the U.S. government.

* Topic covered or partially covered under other topics.

As Table 3.4 illustrates, there is a great deal of commonality among the three programs. The First Steps program does not address the topic of transit as it is an in-Canada program. As well, neither First Steps nor the USCO includes modules on all the topics for which there are modules in COA. Nevertheless, in these cases, they do address many of the sub-topics under other headings.

This high degree of commonality among the orientation material is indicative of the relevance of the COA curriculum to the needs of immigrants and CRs. This is not surprising.

The overall relevance of the course content to the course objectives was confirmed by the vast majority of those interviewed with regard to this issue, including COA course facilitators and coordinators; the IOM global project coordinator; CIC officials at Canadian missions abroad; CIC officials within the Settlement Division and at regional offices in Canada; and officials of in-Canada settlement service providers familiar with the COA. Many of these interviewees did offer additional suggestions and observations related to course content.

In addition, immigrants and CRs who participated in the focus groups were also highly positive, for the most part, about the overall course content. CRs, especially, tended to be very enthusiastic about the topics covered and indicated that the course provided them with a great deal of information of which they were not previously aware. The SWs, while also positive, were more critical of aspects of the course and had many more comments and suggestions for improving the relevance of the course for individuals in their immigration category.

Although the suggestions and comments were wide ranging, a number of common themes emerged from the interviews, the focus groups, the classroom observations and the review of course materials. The following section presents a synthesis of key observations and findings relating to these themes.

Course Content: General Observations

There were several topics that were repeatedly identified as important or relevant. These were:

• employment, especially the process of looking for work; transferability of professional and other credentials; regional unemployment rates (especially by occupation); and having reasonable expectations regarding employability, especially in their profession;
• education, including the public and private school systems; availability of student loans; and transfer of academic credits;
• health, in particular the Interim Federal Health Program; provincial public health insurance; and private health insurance;
• banking and finance, including the use of ATMs and transferring funds to Canada; and
• geography and regions, especially such things as regional climates, regional economics and employment, and rural living and amenities.

Based on classroom observations and a review of the course materials, all of these topics are covered in the course. However, some of the sub-topics may not be covered in sufficient detail, or given sufficient emphasis. Several individuals representing in-Canada service providers indicated that immigrants and CRs tend to immigrate to Canada’s large cities, and that they are under misperceptions regarding the availability of education, health and other services and employment in other, smaller centres. In the views of these individuals, the COA should provide information on the prospects and benefits of emigrating to other regions or to smaller urban or rural areas.

Course Content: Customization

It is clear that there can be a good deal of variance in the nature of the information requirements of various groups, based on their immigration category (e.g., SWs vs.) CRs), on their culture or country of origin, and on educational and other factors. The COA coordinators and facilitators recognize this and adapt the content of the course and the time allocated to specific topics to better meet the perceived needs of participants. By and large, they appear to have been relatively successful in this regard. Nevertheless, based on interviews, focus groups and observations, the course, as currently structured, better meets the needs of CRs and FC immigrants than of SWs, who appear to have different and more selective needs than the other two groups. The following sections provide key observations on the needs of SWs and CRs, respectively.

Skilled Workers

The one topic that is of very little use to SWs is that of transit to Canada, as most have previously travelled by air and many have previously travelled to Europe or North America. In the focus group with SWs, a couple of individuals also indicated that climate and weather were not all that important as most were already familiar with Canadian or North American climate. Nor do SWs (at least those in the focus groups) need basic information about banking and finance or the use of technology (e.g., Internet, ATMs) to conduct banking transactions. They are already very familiar with these services.

On the other hand, SWs are also interested in more detailed or different information on certain topics, compared to CRs. These needs and interests include the following.

Employment

SWs are most in need of detailed information on employment rates in different regions and even cities of Canada for persons in their profession or occupation. As many are professionals, they are also very interested in the potential for, and the requirements of, transferability of their professional credentials. Many do not understand how it is that they are accepted for emigration to Canada based on their educational and professional qualifications and are then not allowed to pursue their profession when they arrive in Canada.

Education

For many SWs, information about education in Canada is a high priority, for both themselves and their children. They are interested in both the public and private school systems and obtaining regionally specific information about the quality of education in these systems. They are also very interested in post-secondary education, including admission requirements and the availability of financial assistance, such as student loans.

Cost of Living

SWs tend to be sophisticated in financial matters and need little or no basic information on this topic. However, it would appear that some are not familiar with the cost of living in Canada and, consequently, have unrealistic expectations regarding their ability to live off of their savings in Canada in the event that they are unable to find employment. In this regard, they need more detailed information on such things as the cost of housing and food, the purchase and maintenance of vehicles, and other factors affecting the cost of living in Canada.

Housing

A related subject that focus group participants indicated was important is housing. SWs want to know how they can arrange for initial accommodation; how to buy or rent a house; the requirements for obtaining a mortgage; and how to determine the suitability of neighbourhoods.

Other areas of interest or importance to SWs identified in the interviews or focus groups included:

• information on driving licences and car insurance;
• information on other urban centres besides Toronto;
• settlement assistance available to SWs in the city or region that is their destination.

Obviously, the COA is not in a position to provide information on all of these topics or at the level of detail desired by SWs, nor is this necessary. Most SWs are proficient in the use of computers and accessing the Internet. In many of these areas, the program could advise participants where they can go to obtain detailed or regionally specific information.

Convention Refugees

The information needs of CRs can also vary widely, depending on whether they have exposure to modern urban environments, their education, their age, the extent of the differences between their culture and North American culture, the education and health systems in their country of origin and their familiarity with modern technology, especially such things as computers, telecommunications tools and the Internet.

Nevertheless, there are a number of migration-related topics that are of importance to most CRs. Most of these topics are already covered by the COA. However, the following areas need to be covered in more depth or given more emphasis.

Transit to Canada

Many CRs have never travelled by airplane and most have never travelled very far from their country of origin. Depending on where they are from, they may lack understanding of such things as baggage limits, no smoking rules on airplanes and the use of Western-style washroom facilities.

Cultural Adaptation/Culture Shock

Depending on their cultural and educational backgrounds, CRs may need significantly more information about cultural differences, rights and obligations and related topics than other groups. These same factors may make CRs more susceptible to culture shock than either SWs, who tend to be much more familiar with the challenges of life in Western societies, or FC immigrants who have support from their sponsors. Because some of the concepts covered under these topics are so different from the sociocultural concepts of some in these categories, they are not always easily understood and may need to be reinforced more than once. Based on classroom observations, the COA does address these issues in its sessions with CRs. However, facilitators are not always able to devote sufficient time to these topics.

After-Arrival Settlement Assistance

According to several of those interviewed, CRs need to be better informed about the programs and services available to them after they arrive in Canada. One program specifically mentioned was the Interim Federal Health Program. As well, several of the in-Canada interviewees mentioned the need to ensure that CRs understood the roles and limitations of in-Canada settlement service agencies. In the views of these interviewees, some individuals have unrealistic expectations of these agencies with respect to such things as finding employment and housing.

3.3.1.2 Course Materials

The main course materials, common to most or all sites, are the following.

Manuals and Other Written Materials

The core course manuals are:

Facilitator’s Guide to Pre-Departure Orientation Modules

All the facilitators at the sites visited have copies of this guide. The guide has not been formally updated since it was first developed.

Welcome to Canada

This is a manual developed by the IOM for students which covers, in an abbreviated fashion, the key learning points under the entire COA topic. The manual is available in English, French, Arabic, Afghani and Irani.

In addition to these core course materials, individual sites make use of a wide variety of common and site-specific written material related to individual topics, including:

• A Newcomer’s Introduction to Canada (booklet): This was developed by CIC to provide immigrants and CRs with information on Canada. It was used by the COA as the student manual for some years, but has been replaced at most sites by the “Welcome to Canada” booklet, as it does not address all of the COA topics and is not quite as up to date;
• magazines, journals and newspapers;
• handout exercises;
• information sheets (e.g., 6 pages of website references at one site); and
• copies of Canadian documents and forms (i.e., customs forms, examples of drivers’ licences, SIN cards, health cards, etc.)

Videos

The primary video used at COA sites is “Canada Day-to-Day” which covers a number of topics that are addressed in the course. It is available in English, French and Arabic. In addition to these, sites make use of a variety of other audiovisual materials, including:

• “Are You Ready for Work in Canada”;
• “Working in Education”;
• videotapes on health care;
• “Over Canada” (videotapes about cities across Canada).

A wide range of other material is available at all of the sites. The COA Project Manager researches Canadian websites and other sources for new and updated information on a regular basis and forwards this information to the individual sites. As well, many of the facilitators search for supplementary material and handouts that will contribute to the learning experience and that will provide the most up-to-date information.

The COA sites do not lack for educational material that supports the course content. On the contrary, a problem at many course sessions is finding time to present audiovisual information. Rather, the key issues with respect to the materials are the following.

Currency of Materials

According to the COA Project Coordinator, the Facilitator’s Guide has not been updated in a number of years, primarily for budgetary reasons. While topics are updated regularly, based on ongoing research by COA staff, these updates are in the form of supplemental materials rather than being incorporated into the manual itself.

As well, “Canada Day-to-Day,” the primary audio-visual presentation used at the COA sites, is approximately ten years old and is badly out of date, in terms of both content and “look and feel.”

The CIC Settlement Division is moving to award a contract for the filming of a new video to replace “Canada Day-to-Day.” This project could be paralleled with an initiative to update the COA Facilitator’s Guide and COA staff could have significant input into the content of both of these.

Support from Canada

At present, COA staff receive very little support from Canadian sources in their efforts to keep up to date. CIC should take steps to encourage a closer relationship and better information exchange between the COA and in-Canada agencies, as has been discussed earlier in this report. This would help to ensure that the COA is providing up-to-date, accurate information.

CIC should also work with the COA to develop a more formal process for regular review and updating of core course materials.

Equipping Participants with Materials

In interviews with USCO officials, they indicated that every CR going to the U.S., whether or not they participate in the USCO, receives a copy of the course participant manual titled “Welcome to the U.S.” As well, the USCO has translated this document into numerous languages so that CRs from many parts of the world can understand it.

At one time, the COA did provide the participant manual to all participants. However, this was cut back at most sites due to budgetary concerns. Only recently have some sites begun providing participants with individual copies of “Welcome to Canada” again. Furthermore, while the manual is available in a few languages, it is not available in some languages that are widely spoken among CRs at COA sites (for example, in Al-Hariq or Nairobi).

A suggestion was made that each COA participant receive a take-home booklet and a checklist of pre-departure and post-departure items of importance. Based on classroom observations, this is particularly germane for CRs, many of whom have difficulty absorbing all the information provided during the course.

Investment in Materials

The main reason cited by IOM officials for not updating course materials, for cutting back on the distribution of participant manuals and for the limited translations of “Welcome to Canada” and other materials was concern over budget reductions. IOM officials have felt that CIC expected them to reach attendance goals despite budget reductions in recent years. Consequently, they were reluctant to put forth proposals for expenditures for development or updating of course materials. CIC officials indicated that they would be supportive of such proposals but that the IOM had not put forth funding requests for this purpose.

Key Conclusions

Based on the various lines of enquiry used to address the issue of content and material, the following are the key conclusions regarding this issue.

• The content of the COA is highly appropriate with respect to CRs. However, the course is not as well matched to the needs of SWs in terms of content, sub-topic focus or level of detail on some topics.
• The core Facilitator’s Guide is out of date, necessitating the facilitators to maintain large quantities of updated material as a supplement to the manual and creating the potential for inaccurate information to be provided.
• The main COA video is out of date with respect to both the accuracy of some of the content, and the look and feel. However, CIC is in the process of updating this videotape.
• The IOM has little support from in-Canada organizations to assist it in keeping course materials up to date, nor is there in place a process for regular review and updating of core course materials.
• The COA has, until recently, been severely limiting distribution of the COA participant’s manual. This could potentially have negative impacts on the benefits of the program, especially for CRs.
• The COA participant’s manual has been translated into only a few other languages, which impairs its usefulness, especially for CRs.

3.3.2 Facilitation Approach and Facilitator Training

The effectiveness of the facilitation approach to learning can be affected by numerous factors. With respect to the COA, four potentially significant factors were identified:

• commitment to facilitated learning;
• facilitator training;
• class size; and
• course length.

Our findings are based primarily on interviews with COA coordinators and facilitators and the COA Project Manager; focus groups with COA participants; classroom observations; and adult education theory.

3.3.2.1 Commitment to Facilitated Learning

The IOM utilizes a facilitation approach to learning in the delivery of the COA. Facilitation is an approach to learning that encourages the active participation of participants in the learning process, in contrast to more traditional models, such as lecture-based approaches, in which participants play a more passive role.

Most of the participants in the COA are adults. Current theory in adult education indicates that adults learn much more effectively in learning environments in which they can play an active role in the learning process and can learn through direct experience.

According to Carl Rogers, “learning is facilitated when: (1) the student participates completely in the learning process and has control over its nature and direction, (2) it is primarily based upon direct confrontation with practical, social, personal or research problems, and (3) self-evaluation is the principal method of assessing progress or success. Rogers also emphasizes the importance of learning to learn and an openness to change.

[T]he role of the teacher is to facilitate such learning. This includes: (1) setting a positive climate for learning, (2) clarifying the purposes of the learner(s), (3) organizing and making available learning resources, (4) balancing intellectual and emotional components of learning, and (5) sharing feelings and thoughts with learners but not dominating.” [ note 5 ]

The COA staff members interviewed were all highly positive regarding the facilitated approach to learning. Several of them pointed out that the interactive approach has been adopted by the Australians in their program and, to a lesser extent, by the USCO. Comments on the facilitated approach included the following:

• “The lecture method makes them go to sleep.”
• “The course reinforces the notion that everyone has something to contribute.”
• “People’s interest is maintained; they understand the material better.”
• “Canada’s approach is good and has been imitated by other countries.”

Despite the strongly positive tenor, these individuals also pointed out that there are challenges to using the facilitated approach. A major challenge is that, in some cultures, individuals are taught in school systems that take a more traditional, lecture-based approach. Consequently, students expect to be passive and are reluctant to be vocal and active during the course.

The level of student interest also affects the success of the facilitated approach. According to some IOM officials, the level of interest can vary, particularly among FC immigrants and LICs. In both cases, it is thought that the fact that these individuals know they have relatives in Canada who can support and assist them reduces the level of interest in the COA on the part of some.

In the focus groups, COA participants were asked if they found the course interesting and if they were encouraged to interact and engage in dialogue with one another. There was a widespread consensus among the focus group participants that the sessions were interesting and that people were encouraged to ask questions of the facilitator and each other.

It was evident from classroom observations that SWs tend to be much more vocal and to ask many more questions than do other categories of immigrants. SWs will also challenge the facilitator if they feel they are not getting correct or sufficient information and will seek information on topics not addressed in the course.

Other groups tended to be more passive in their learning approach during the sessions. CRs, in particular, tended to listen but not to ask many questions. They often had to be strongly encouraged by the facilitator to ask questions, to discuss issues, or to tell the class what they knew about a subject. As noted above, this may reflect cultural differences in learning, as well as lower levels of education. The SWs whose sessions were observed were all highly educated and fairly knowledgeable about some topics. They also had clear ideas about what they wanted to know and, thus, were less reluctant to ask questions in these areas.

During the focus groups, the SWs tended to be more willing to share their views on the COA and to voice comments, positive or negative, about the course. It was much more difficult to draw out individuals in other categories, especially CRs, and the latter group was especially reluctant to say anything negative about the course. It may be, as well, that during the COA session and the focus groups, CRs were intimidated by the presence of a Canadian government official, whereas the SWs were not. More generally, CRs may be reluctant to voice questions or comments that could be interpreted as critical, for fear of jeopardizing their status. In any case, while these individual and cultural differences pose a challenge to facilitators, they are not insurmountable if the facilitator is well trained.

In summary, the COA staff is strongly committed to the facilitation approach. CRs and, in some cases, individuals in other immigration categories can especially benefit from the facilitated approach when they are dealing with new concepts. They need to be able to ask questions, to discuss issues and to have opportunities to have the concepts explained more than once, as well as to have them reinforced through exercises or through the use of audiovisual material.

3.3.2.2 Facilitator Training

The COA has not defined formal requirements for facilitators. However, based on interviews with the CIC Settlement Division HQ officials and IOM officials, there is an expectation that facilitators will be sufficiently educated to be able to communicate well in English or French and, at some sites, in other languages spoken by CRs or immigrants. As well, facilitators are expected to have knowledge of Canada and to have lived in Canada.

COA facilitators were asked how their backgrounds prepared them for their role as facilitators. Of the eight individuals interviewed, all but one had post-secondary education, the single exception having had a secondary education. All were fluent in English and several spoke other languages used at their sites, such as Arabic, Afghani or Urdu. Of the eight facilitators interviewed, two were native-born Canadians who had been living outside Canada for a number of years. Both of these individuals return to Canada periodically. Four individuals had lived in Canada for extended periods (eight to 15 years) as immigrants or CRs. Of the other two, both located in Pakistan, one, who is also the coordinator in Pakistan, has lived in Europe and in Australia and has spent only three weeks in Canada. The other facilitator interviewed is an Afghani CR who has never been to Canada.

The backgrounds of the COA facilitators, for the most part, provided them with the right kind of knowledge of Canada to enable them to understand key course concepts and to convey accurate information to COA participants. The exception was Pakistan, where the coordinator or facilitator, while knowledgeable about Western culture, had very limited direct knowledge of Canada, and the facilitator interviewed had never lived outside Afghanistan or Pakistan.

In some locations – and Pakistan is one – the COA faces tremendous challenges in recruiting facilitators and coordinators. For one thing, the working environment can be very difficult. In the case of Pakistan, large numbers of expatriate Canadians have left the country in recent years, due to the political situation. Also, at a number of sites, in particular those with large numbers of CRs, courses must often be delivered in the mother tongues of participants as they don’t speak English or French. At the same time, much of the course material is available only in English or French. Thus, facilitators and coordinators must be able to speak one of these languages. Finding effective facilitators who speak local languages and who are also fluent in English or French, and who have lived in, or at least visited, Canada at length can be extremely difficult in some countries.

The COA will likely continue to have difficulty finding facilitators with ideal backgrounds at some sites. Finding individuals with experience living in Canada will likely be the main challenge. The USCO has dealt with this situation by providing opportunities for its course facilitators to accompany flights of CRs to the U.S. and to visit in the U.S. for a period. This has the advantages of providing CRs who require assistance with a guide on their initial travel to the U.S. and gives the facilitators the opportunity to experience life in the U.S. for themselves.

All COA facilitators have been trained using the same approach, according to the COA Project Manager, the site coordinators and the facilitators themselves. This training essentially comprised a period of observation of experienced facilitators, combined with study of the course materials. Subsequently, facilitators practised facilitating sessions for periods of several weeks to two months, during which they were observed and provided with feedback on both content and their style of interaction with participants. Again, based on classroom observations and interviews, the facilitators appeared to fully understand the facilitated approach to learning.

3.3.2.3 Class Size

Class size is an important concern for facilitated learning. Because active participation, in the form of discussion, questions and exercises, is central to this approach to learning, class sizes cannot be too large. Otherwise, it is difficult for the facilitator to ensure that everyone in the class has an equal chance to participate.

On-site observations demonstrated that large class sizes, at times, detracted from the effectiveness of the facilitated approach. The maximum class sizes that can be accommodated at the five sites visited (Cairo, Nairobi, Islamabad, Peshawar and Manila) ranged from 20 to 25 in Cairo to 40 to 60 in Manila. [note 6] At three of the sites, the number of individuals participating met or exceeded these maximums. As a consequence, the classrooms were overcrowded and excessively warm and stuffy (although in one case, faulty air conditioning contributed to this). More importantly, once the class size moves beyond 20–25 participants, it is extremely difficult for the facilitator to give everyone a chance to participate. In one session, there were 43 participants and in Manila, over 50. Classes this large seriously undermine the facilitated approach.

It may be that larger classes and a less interactive approach are feasible at some sites, in particular those where sociocultural differences are not as pronounced and where participants speak English and are more familiar with Canada. At such sites, participants may simply need information as opposed to explanations of concepts and phenomena outside their range of experience. Consequently, more of an informational style, as opposed to an interactive style, may suffice.

3.3.2.4 Course Length

Another factor that limits the ability of facilitators to adopt a facilitated approach is the length of the course. Most sessions currently being delivered are of one or two days’ duration. Given this, and given the amount of material that must be covered, facilitators have to control the pace at which the class covers topics and, consequently, have to limit questions, discussions and exercises.

Classroom observations support the view that the length of the course is a significant barrier to the effective use of a facilitated learning approach and prevents facilitators from covering topics at the appropriate depth. Site visits allowed for direct observation of five sessions, involving four different facilitators. At three of the sessions, the participants were mostly or all CRs, one session included only SWs, and one only LICs. At most of these sessions, facilitators made efforts to encourage interaction and participation and, at some sites, made use of learning exercises. However, facilitators were under significant time pressure to cover all the relevant material. Consequently, at many points, questions or discussions were limited and few or no exercises used. As a result, at times, the sessions reflected a lecture-based approach rather than a facilitated approach.

The contribution agreement with the IOM contains provisions to the effect that the IOM must provide courses of one, three or five days’ duration. While the agreement does not say so explicitly, CIC and IOM officials both indicated that the policy was to provide five days for CRs; three days for FCs and one day for SWs. However, according to IOM officials, they have been forced to reduce the course for CRs to three days and, in some regions, to one day, primarily for budgetary reasons and, in a few cases, because of the difficulties faced by participants in attending a five-day course. (These same factors contribute to the large COA classes.)

Interviews with U.S. program officials indicated that they have a flexible approach to setting the course length for their orientation program, which is offered only to CRs. Depending on circumstances, the course can range from several hours to 30 (five days at six hours a day). The primary reasons for reducing the length of the course, in the case of the USCO, are logistical. At times, the program has only a last-minute opportunity to provide orientation. In the case of the Canadian program, it is clear that there are circumstances where it is difficult to provide a three- or five-day course because of the lack of time between the time the IOM obtains information about potential participants and their scheduled departure date for Canada. Nevertheless, while one day appears to be sufficient for SWs in some countries, it is insufficient for most CR and FC individuals. IOM officials have noted some obstacles to increasing the length of the course. In the case of CRs, it can present an obstacle with respect to their employment, transportation or child care.

Key Conclusions

• The facilitation approach to learning is known to be a highly effective approach to adult education and, as such, should be the preferred approach to learning of the COA.
• While COA staff are strongly committed to the facilitation approach, cultural differences among participants present a challenge to gaining active participation. As well, CRs may be reluctant to be vocal or questioning, out of concern for jeopardizing their status.
• Most COA facilitators have appropriate backgrounds for their roles, although a few lack direct knowledge of Canada.
• COA facilitators all have appropriate training to enable them to implement a facilitated approach to learning.
• The COA classes are too large and are an impediment to the facilitated approach to learning.
• The reduction of the COA course to one or two days’ duration at virtually all sites seriously undermines the facilitated approach and the capacity of facilitators to cover the material at an appropriate level, especially in the case of CRs.

3.4 Monitoring and Evaluation

Our findings in relation to these questions are organized under two headings: participant feedback, and monitoring and reporting on program delivery.

3.4.1 Participant Feedback

The COA seeks participant feedback in two ways:

• through the “Participant Evaluation I” form distributed at the end of the COA sessions; and
• through the “Participant Evaluation II” form, also provided to participants at the end of the sessions and which they are asked to fill out three months after their arrival in Canada.

In the course of interviews with U.S. officials regarding the USCO, the officials indicated that some sites made use of site-specific feedback questionnaires, but no general participant evaluation questionnaire was used by the U.S. program.

IOM officials responsible for delivering the Australian orientation program provided us with a copy of the end-of-course questionnaire used by this program.

Table 3.5 compares the questions used in the two COA evaluation questionnaires and those used by the Australian program. As this table shows, the two COA questionnaires are virtually identical. The Australian questionnaire contains several questions that differ from the COA questionnaires.

Based on interviews and an independent assessment, the Evaluation I and Evaluation II questionnaires, in their present form and as currently used, are of limited value to the COA.

Of the four main sites visited, only two currently ask participants to complete the Evaluation I questionnaire. COA staff at the sites where it is not distributed indicated that it is due to one or more of the following reasons:

• many participants are illiterate and have difficulty understanding the questions;
• many participants do not speak English and the questionnaire has been translated into only a few languages; and
• the responses are virtually 100% positive and, therefore, provide little useful information for improving the sessions.

At several sites, facilitators do seek oral feedback from participants, having them answer the questions:

• “What do I know?”
• “What do I need to know?”
• “What have I learned?”

Most of those interviewed who commented were also skeptical regarding the Evaluation II questionnaire. One site no longer gives these out. While the others do, they indicated that the responses are mailed to CIC and IOM staff members never see the results of any analysis.

In fact, officials of the CIC Settlement Division at HQ indicated that response rates for the Evaluation II questionnaires were extremely low and that those that are sent in are highly positive and provide little useful commentary that could be used to improve the COA. Furthermore, because of the limited resources CIC has to allocate to the COA, they have been unable, until recently, to conduct any analysis of the Evaluation II questionnaire. At the time of the interviews, the IOM had not received any feedback based on the Evaluation II questionnaire.

The CIC Settlement Division recently conducted a summary analysis of a sample of 200 Evaluation I questionnaires provided between 2001 and 2003. The results of that analysis are included in the discussion of the impacts of the COA. In general, the responses to questions were highly positive. However, 18% of respondents indicated that the sessions were not long enough. In addition, respondents provided several specific suggestions to improve the COA.

The Evaluation I and II questionnaires ask virtually the same questions. These questions are highly general and are of little value in assessing what was learned that was useful or important, or in assessing the impact of the COA on the experience of CRs and immigrants once they arrive in Canada. This high level of generality and the similarity in content reflect a lack of clearly defined objectives for the two questionnaires. Overall, the Evaluation I and II questionnaires are of limited value to the COA. Appendix E provides a discussion of potential objectives for in-course and post-course participant feedback, and potential improvements to the content of questionnaires and the processes for seeking feedback.

3.4.2 Monitoring and Reporting on Program Delivery

The IOM reports on the COA to the CIC Settlement Division every three months and monthly for the last three months of the year. In addition, they provide a year-end report covering the entire fiscal year. The quarterly reports provide essentially the same information, including statistical data on:

• the number of individuals who were referred to, and attended, COA sessions at each site;
• the demographic profile of participants at each site, based on gender, immigration category, level of education, session hours per person per day, total session days per person and total session hours per person;
• the number of adults and children, respectively;
• the number of participants trained in each official language and in their mother tongue;
• the number of children for whom child-minding services were provided;
• the cost by site, per participant day and per participant hour; and
• other data as specified by CIC.

The reports also provide narrative information on successes, obstacles and opportunities encountered during the reporting period. The reports are also supposed to contain accounting statements. However, the examples provided (which included most reports for the last three years) did not contain this information. The IOM does submit invoices to CIC on a quarterly basis that would provide this information.

The annual reports are similar to the quarterly reports but provide summary statistical and financial data for the year and a narrative report on the program as a whole.

The interviews with the CIC Settlement Division revealed that, in the past, while CIC staff reviewed these monthly, quarterly and annual reports, they made little direct use of them. More recently, CIC officials have been monitoring the financial performance of the COA more closely as the IOM had lapsed funds unexpectedly several years in a row. In response, CIC has incorporated clauses into the most recent agreement that strengthen the Department’s ability to decommit expected slippages in expenditures from the budget at an earlier date.

CIC also makes use of other tools for managing and monitoring the COA, including:

• IOM annual proposals for the upcoming fiscal year;
• a review of invoices sent by the IOM;
• informal e-mail contacts from previous COA participants; and
• communications with the International Region (which receives site-specific information on the COA from CIC mission staff).

CIC has also established an internal COA Working Group consisting of representatives from the Integration Branch, the Refugee Branch, the Selection Branch and the International Region to assist in the selection of COA sites and to look at ways of increasing participation rates in the COA.

According to several of the CIC officials interviewed, the lack of adequate resources within the Settlement Division to monitor the report (currently CIC allocates one-third of an FTE to the COA within the Settlement Division) is a bigger constraint on effective management of the program than the reporting regime itself.

Another problem has to do with the timeliness of reporting by the IOM, according to these same officials. In the last few years, the IOM has been late in submitting reports and this has created difficulties for CIC in terms of budget slippages and meeting their own obligations for reporting on the program. Based on interviews with the IOM, it would appear that these delays in submitting reports have resulted partly from delays in receiving data and narrative reports from individual sites and partly from the fact that resources to manage and report on the COA are quite stretched within the IOM.

While the monitoring and reporting regime is adequate, there are a number of ways in which it could be made both more effective and more efficient, reducing the work involved in both preparing and reviewing reports. Chief among these are the following.

Content of Reports

Given the limited resources within the Settlement Division to monitor the COA, there is a limit to the scope and level of detail that can be expected. The information contained in the reports provided to CIC should focus on:

• information required for program planning;
• financial management of the program, especially with regard to actual vs.) planned expenditures, estimated and actual unit costs at individual sites and for the program as a whole;
• performance monitoring with respect to program priorities; progress against program targets for participation and related measures; and
• to a limited extent, reporting on specific challenges or issues.

For the purposes of meeting these requirements, CIC could consider the following planning and reporting regime for the COA.

COA Annual Plan or Proposal

This document would be developed and prepared by the IOM for CIC’s consideration well in advance of the upcoming fiscal year. It could:

• identify sites at which the IOM proposes to conduct training and a rationale for these in terms of CIC priorities;
• provide estimates of the number of persons, by immigration category, that will be trained at each site, by quarter;
• indicate the number of participant days of training to be provided at each site;
• provide estimates of the cost per participant and total costs at each site; and
• provide a plan for investments in updating course materials; for recruitment, and the professional development of facilitators; and for facilities and equipment.

Quarterly and Annual Reports

These reports should focus primarily on:

• the number and demographic characteristics of people trained at each site, compared to estimates contained in the IOM annual plan;
• the variance of actual expenditures by site and in total, compared to the annual plan;
• factors explaining financial or numerical variances from estimates;
• summaries of the results of any surveys of participants completed during the period; and
• significant issues that have arisen.

CIC should discuss with the IOM the way in which it wants financial and related information reported and the appropriate level of detail, so as to allow for better oversight of the management of the program. For example, CIC officials have indicated that they do not have sufficiently detailed information on how facilitators’ time is allocated, nor are they easily able to determine the cost factors (e.g., travel, salaries) that affect the differences in the costs per participant at each site.

More detailed information, such as the time spent by facilitators on class preparation and travel versus class time, or the total costs incurred for travel, salaries and other costs at individual sites would facilitate decisions on the cost effectiveness of training at sites and would contribute to better management of the program by CIC. However, in examining this issue, it would be important to consider the demands that would be placed on the IOM’s resources by additional or more detailed reporting, as well as the capacity of CIC to make effective use of this information given the limited resources allocated to the COA.

The annual report should provide a summary of this same information for the year.

Frequency of Reports

Currently, the CIC requires quarterly reports from the IOM and monthly reports during the last three months. The primary reason for the increased reporting requirement during the last quarter appears to be CIC’s concerns regarding potential budget slippages by the IOM. However, this monthly reporting requirement places a significant workload on the IOM.

If the reporting requirement suggested above for the quarterly reports are met in a timely fashion, the IOM and CIC should both have a good basis for estimating any budgetary shortfalls or potential lapses by the end of the third quarter, and there should be no need for monthly reports in the fourth quarter.

Production of Statistical Data

The IOM makes use of a spreadsheet approach to compiling quarterly and annual statistical data. The IOM COA headquarters in Manila relies on the individual sites to provide this data in a timely fashion. The individual sites provide the source data to COA HQ, where they are aggregated for all sites. The IOM should look into the feasibility of moving to a distributed database approach where data on participants are entered into a common program database when individuals are registered for the COA and again when their participation is confirmed. The compilation of data and the production of reports would then be done automatically for individual sites and for the program as a whole. This would likely improve both the accuracy of the data and the timeliness of reports. If estimates of participation rates, developed for program planning purposes, were also entered on this database, the comparison of actual vs.) estimated volumes would be greatly facilitated as well.

On-Site Visits

No CIC program officer responsible for managing the COA has visited a COA since its inception, although other CIC personnel have visited COA sites. If CIC is to take a more active role in both monitoring the IOM in its delivery of the COA and providing more support to the IOM with regard to its need for up-to-date information, materials and learning tools, it is important that CIC officials meet with the COA Project Manager and visit IOM sites on a regular basis.

These visits would provide opportunities to observe actual sessions, become familiar with the local challenges faced by COA staff, evaluate facilities, equipment and materials, and discuss issues with the Project Manager and site facilitators.

In addition, the IOM Project Manager should be encouraged to make periodic visits to COA sites for the purposes of recruiting, training and evaluating staff; ensure a consistently high standard in course delivery; evaluate facilities, equipment and materials; maintain close working relationships with CIC mission staff; and provide quality control over the delivery of the program.

Key Conclusions

• The Evaluation I (end-of-course) and Evaluation II (after arrival in Canada) questionnaires are not useful tools, in their current forms, for obtaining participant feedback on the COA for the following reasons:
• the content of the COA questionnaires does not appear to be driven by clear objectives for their use;
• the questions are too general and they are virtually identical on both questionnaires.
• COA staff at some sites are skeptical of the value of the Evaluation I questionnaire and have concerns regarding the difficulties participants with poor literacy skills have in completing it. Consequently, they do not distribute them.
• Only a very small percentage of course participants complete and mail in the Evaluation II questionnaire, which greatly diminishes its value as an unbiased reflection of the population of participants.
• Monitoring and reporting of the delivery of the COA, while adequate, is not ideal. Reporting is not clearly focused on providing CIC with data and information useful for financial and performance management or for planning.
• The IOM has had difficulty, in recent years, in providing quarterly and annual reports in a timely fashion.
• CIC officials do not visit COA sites periodically as part of their monitoring of the COA, and CIC has not suggested site visits by the IOM Project Manager. This lack of site visits is an impediment to the effective management and monitoring of the COA.

3.5 Results

The findings in relation to this question are focused on the contribution of the COA to:

• improved participant knowledge; and
• facilitated integration.

3.5.1 Improved Participant Knowledge

A necessary condition of the COA having a positive impact on the settlement experience of those who arrive in Canada is that the course provide participants with accurate, timely information that is helpful to them after their arrival.

All the stakeholders interviewed were asked whether participants improved their knowledge of Canada in general, settlement issues and settlement programs and services. CIC mission officials were the least informed about this: only at one site were these officials able to answer this question. Officials at that site believed participants did improve their knowledge in these areas. These officials commented that participants get less upset regarding travel, have better knowledge of women’s and children’s rights, and have more realistic expectations.

COA facilitators and coordinators and the COA Project Director were virtually unanimous in the view that participants do improve their knowledge in areas essential to their survival. A number of facilitators indicated that the most significant benefit of the course was that participants had much more realistic expectations, especially with regard to critical issues such as the recognition of educational or professional credentials, finding a job and the difficulties of resettling in a new country.

CIC HQ (Settlement Division) officials, while they believed that participants improved their knowledge, indicated that they had only the Evaluation II surveys and other anecdotal evidence on which to base their opinions.

Representatives of in-Canada service providers were of mixed opinion on this. While all felt that the program helped prepare participants to some degree, two interviewees indicated that the benefits were limited because the expectations of COA participants, like those of many other migrants, were too high.

CIC regional office officials and officials of in-Canada settlement service providers were asked a more detailed series of questions regarding the impact of the COA on participants, including whether the course provided them with appropriate information about Canada and settlement services, and the difficulties they would face after their arrival; whether it gave them realistic expectations about Canada; and whether it made them aware of the need to bring educational and other documents. Only one of the four CIC regional officials and four of the 12 service provider officials responded to these questions.

Most of these interviewees felt that the program helped and was successful to some degree in providing migrants with appropriate information and knowledge. However, several felt that the expectations of COA participants were too high, and that many still didn’t understand the difficulties they would face finding a job and some didn’t know what winter would be like. One interviewee felt that the program was of greater value to those settling in large urban centres than those settling in rural areas. While most indicated that COA participants were aware of the importance of bringing documents proving their educational or professional credentials, several indicated that other immigrants, who had not taken COA, were also aware of this.

Based on the sample (N=200) of Evaluation II questionnaires from the period 2001 to 2003 analysed by CIC, COA participants themselves are strongly of the view that the COA helped them to settle in Canada. Seventy-one percent of these individuals indicated that the sessions helped them prepare for, and know what to expect in, Canada in terms of culture, laws, rights and the way of life. Some of the comments of these participants, as to how the COA helped them, are informative:

In summary, most of those interviewed see value in the COA and believe that it is imparting to immigrants and refugees the information and knowledge they require to prepare for life in Canada. Individuals who actually assist migrants after their arrival in Canada, while still positive, tend to be less enthusiastic about the benefits of the program. Based on the small sample of participants who completed and returned the Evaluation II questionnaire, the participants themselves appear to be of the view that the COA has provided them with useful information that they applied once they arrived in Canada. It is important to note, however, that because of the small sample and its non-random nature, the responses of these individuals do not provide a reliable basis for generalizing about all participants.

3.5.2 Facilitated Integration

The ultimate measure of the value of the COA is whether it contributes to enhanced integration into Canadian society for individuals who participate in the program. Enhanced integration would be reflected in a more rapid acquisition of housing, employment and education, and access to health and social services; fewer psychological or emotional problems related to the resettlement process; and reduced dependency on settlement services.

We asked the stakeholders whether the COA facilitated integration into Canadian society. CIC mission officials were unable to comment on this. Officials at CIC HQ felt that the program did help individuals integrate, but indicated that their views were based on anecdotal information and the comments provided in the Evaluation II survey.

COA facilitators and coordinators, on the other hand, tended to be strongly of the view that the COA does facilitate the integration of participants. One individual who had been a refugee himself indicated that he wasted his first two years in Canada because he had not been adequately prepared. Another stated that she could not imagine how the COA could not help them integrate as it provided information on all the important topics. Many of the facilitators and site coordinators also receive a large number of e-mails from people in Canada who have taken the COA and who want to express their appreciation for how it helped them.

The sole CIC regional office official who answered this question wrote that presumably, the COA did facilitate integration. Representatives of in-Canada settlement service providers who, aside from migrants themselves, are in the best position to answer this question, were more mixed in their views. One individual felt that participants settled more easily and tended to be more open and less isolated than non-participants. Another was of the view that the program offered limited help, while a third indicated that it gave them some perspective on what to expect. The fourth individual was unable to provide an opinion on this question.

Based again on the analysis of the Evaluation II questionnaire, individuals who participated in the COA indicated that they felt it had helped them to integrate: 98% of the individuals whose survey questionnaires were analysed (N=200) indicated that they thought the COA helped them to settle in, and 80% said they would recommend the course to other immigrants, indicating a strongly positive view of the course.

While these lines of enquiry provide some support for the view that the COA facilitates integration into Canadian society, they are not sufficiently reliable to draw firm conclusions on the issue. To properly evaluate the impact of the COA on integration into Canadian society, it is essential to have objective data relating to valid indicators of integration, such as those discussed at the beginning of this section. Ideally, to measure the incremental impact of the program, we would need comparative data on indicators of integration for both COA participants and non-participants.

In preparation for a future evaluation and, possibly, to provide periodic feedback to guide adjustments to the content and other aspects of the COA, CIC should explore one or more of the following options for obtaining these data.

After-Arrival Survey

A methodology for evaluating impacts – and a methodology that would also provide useful information for program monitoring and program modifications – is to conduct periodic surveys of recently arrived refugees and immigrants to Canada. Ideally, the survey would be based on a randomized, experimental design and would sample both past COA participants and non-participants. The survey could investigate the perceived usefulness of the program as a whole and of individual topics, the accuracy of the information provided, areas where information should have been provided but was not, and whether participation facilitated integration.

There would likely be a number of significant challenges to implementing this methodology, including the difficulty of locating immigrants and refugees after they have arrived in Canada, language and literacy issues, and costs. Nevertheless, it would provide a basis for inferences regarding the impact of the program.

After-Arrival Focus Groups

Focus groups of recent arrivals, both COA participants and non-participants, would be less costly than a survey approach and could be carried out in the mother tongue of the participants, minimizing language issues. Focus groups also allow for more in-depth exploration of topics and, therefore, can provide richer information than surveys.

However, information gained from focus groups is not based on a randomized sample. Consequently, it does not provide a reliable basis for generalizations about the populations represented in the focus groups.

Data on Use of Settlement Services

CIC is in the process of strengthening its capability to monitor the performance of its settlement service providers in Canada. To this end, it has established the Immigration Contribution Accountability Measurement System, which will provide a basis for the collection and compilation of performance and accountability information by these service-providing agencies. It would be highly valuable if the data collection systems could include data that would enable comparisons between COA and non-COA participants regarding such things as:

• rate of use of these services;
• duration of use of services;
• nature of specific services used (e.g., informational, counselling, direct assistance, etc.).

These data, if collected accurately and consistently, would provide valid and reliable information on several important indicators of integration.

Key Conclusions

• Based on limited evidence from stakeholder interviews and analysis of a small sample of the COA participant after-arrival questionnaire, the COA does appear to improve participants’ knowledge of topics that are important to the settlement experience.
• The lines of enquiry adopted in the evaluation did not provide a basis for firm conclusions on whether or not the COA facilitates the integration of participants into Canadian society.

Management Response