Evaluation of Government Assisted Refugees (GAR) and Resettlement Assistance Program (RAP)

Appendix B: Evaluation methodology

The current evaluation utilized a hybrid model to conduct research activities. Both CIC and the Consultant actively participated in all phases of the evaluation. Using the hybrid model a joint evaluation was conducted of the GAR and RAP programs. By examining both programs simultaneously the evaluation investigated the refugee experience from the stage of selection and processing overseas to the settlement stage in Canada.

In keeping with Treasury Board requirements the evaluation assessed the:

1. Relevance of the GAR and RAP programs in terms of continued need, federal role and alignment with government objectives and priorities.
2. Performance of both programs in terms of effectiveness, efficiency and economy.

The Evaluation focused on the previous five fiscal years: 2005/06 to 2009/10 and examined outcomes for GARs from the landing years 2005 to 2009. A series of questions related to design and delivery for both programs were utilized for the evaluation.

Relevance

• Is there a continued need to provide protection to refugees?
• Is there a continued need for RAP?
• Are RAP and the GAR program consistent with departmental, government-wide and international protection priorities and commitments?
• Are RAP and the GAR program consistent with federal roles and responsibilities?

Design and delivery

• Are GAR selection, matching and processing efficient and effective?
• Is RAP appropriate and sufficient for the needs of the GAR population arriving in Canada?
• Is resettlement policy and program development for GARs evidence-based, consultative and responsive to the diverse needs of refugees and communities?

Performance (effectiveness)

• Are the immediate and essential needs of RAP recipients met through RAP?
• Do GARs have the necessary knowledge, skills and means to live safely and independently?
• Are they linked to services they need to address issues as they emerge?
• Do GARs obtain and benefit from CIC settlement services? If not, why?
• To what extent does CIC influence international protection policies through resettlement?
• Does CIC’s resettlement program leverage benefits for both selected refugees and those not resettled?

Performance (efficiency and economy)

• Are there alternative RAP design and delivery options that would better facilitate the achievement of improved outcomes for GARs?
• Are there approaches to GAR selection and processing that could lead to a more coordinated and efficient process?

Multiple lines of evidence were collected during the evaluation. Data was collected and analyzed from a variety of primary and secondary data sources. The multiple lines of evidence were triangulated during reporting.

Primary data sources

The primary data sources used for the evaluation included key informant interviews, focus groups, case studies, and surveys. Each of the data collection methods are described in more detail below:

Interviews with key informants. The key informant interviews were designed to address the evaluation questions related to relevance, program design and delivery, effectiveness, and performance in terms of efficiency and economy. These interviews were completed with stakeholders that have a larger view of refugee issues and the role that the GAR and RAP programs play to address them. As indicated in the table below, 98 interviews were completed with 197 representatives of CIC National Headquarters; CIC Regional Program Advisors; local CIC offices; executive directors of SPOs that provide RAP; RAP managers and counsellors; stakeholders within local community services; international and national stakeholders, and provincial representatives.

Eight (8) interview guides were developed to gather the perceptions, opinions, knowledge and experience of these various stakeholder groups. The Consultant and the CIC evaluation team worked together to determine the appropriate interviewees and to recruit these individuals for interviews. The approved interview guides were distributed to each stakeholder upon the confirmation of the interview time to assist in their preparation. The Consultant completed these sessions either in English or French, dependent on the preference of the individual being interviewed. All interviews were administered by telephone.

Case studies. The case studies consisted of field visits to four (4) Canadian Visa Offices Abroad (CVOA): Bogota, Colombia; Singapore, Singapore; Nairobi, Kenya; and Damascus, Syria. These visits were designed to provide a better understanding of GAR selection and processing, and were selected to provide a perspective of different types of refugees and processing models used across CIC (source country vs. convention, individual vs. group processing), as well as refugee settings (sites that worked with camp-based vs. urban refugees). Furthermore, the sites were selected to provide a good representation of the different world areas and thus of the various refugee populations. Finally, the international sites were selected based on the high number of refugees associated with these locations. International case studies consisted of interviews with Visa Officers and CVOA staff, local UNHCR and IOM staff (the Consultant developed two (2) separate interviews guides, one for each stakeholder group); review of program and output documentation; and where possible, visits were conducted to refugee camps or settlements. Furthermore, case studies were completed within ten (10) Canadian communities: Vancouver, Edmonton, Lethbridge, Saskatoon, Winnipeg, Kitchener, Toronto, Moncton, Halifax and St. John’s. As part of these visits, interviews were conducted with SPO directors and staff; local CIC officers in all locations except Toronto, and with local stakeholders/partners of SPOs. In addition, the Consultant completed program documentation review, toured local SPOs facilities, and, where possible, observed service delivery/intake interviews. These sites were selected to reflect each of the regions across Canada, to provide a representative overview of large urban and smaller regional sites, as well as different target ranges for refugees in each city. The sites were also selected to reflect the different SPO sizes, the different models of working with clients, the different modes for temporary accommodation, the array of community services available, and the range of refugee groups that each site worked with.

SPOs representatives’ surveys. Surveys were conducted with 500 participants of GAR and 20 representatives of the SPOs. For the SPO survey, the Consultant designed the survey instrument that assessed the extent to which SPO programs operate as intended and contribute to achieving desired outcomes. Although the survey was available in multiple modes (telephone, online, hardcopy), it was initially distributed to each SPO in physical hardcopy format. This action intended to maximize input from SPO staff members by allowing SPOs to solicit input from a broad range of staff, while still providing only one completed survey submission. Prior to full survey administration, a communiqué was sent to each SPO by the CIC evaluation team detailing the evaluation and requesting participation. The Consultant further completed invitational phone calls to recruit SPO members.

The survey with SPOs commenced on September 29, 2010, and continued until October 29, 2010. The Consultant sent out a reminder email on October 22, 2010 to all participants who had not completed the survey at that point. Of the 26 SPOs invited to participate in the survey (i.e. all SPOs that provided services between 2005 and 2009), 20 provided a response, resulting in an 80% response rate.

GAR participants’ surveys. For the survey of GAR participants, the Consultant developed a survey instrument designed to collect data on the GARs’ perceptions of Canadian refugee process and the supports provided. To address potential language issues, the survey was available in both official languages, English and French, as well as translated into the top six (6) languages spoken by GARs who landed in Canada between 2005 and 2009: Spanish; Burmese; Arabic; Dari; Farsi; and Somali. Given the vulnerable nature of this population, recruitment and participation in the survey occurred through a multi-stage process.

The survey targeted GARs who arrived in Canada between 2005 and 2009, who were not destined to Quebec, and who were between 18 years old and 65 years old when they landed. There were 15,334 GARs who met these criteria. However, the decision was made to target only one person per case, as many of the survey questions were designed to address experiences of the household. So, of the 15,334 GARs who fell in the population targeted for the survey, CIC randomly sampled one individual per case, which resulted in the mailing of 9703 consent form letters to GARs.

Prior to being contacted by the Consultant, the GARs received a communiqué from CIC. This communiqué provided them with information about the evaluation and solicited their involvement in the project. The letter was pre-populated with their basic personal information (i.e., name, address) and included a unique identifier. This identifier linked the potential respondent to the CIC databases, with an intent that it would help reduce the survey demands made on GARs. Through the use of the unique identifier it was possible for CIC to extract demographics and background information, and some GAR outcomes, from their files rather than asking that information on the survey. The GARs were asked to review and correct any wrong information on the communiqué and to consent to participate in the evaluation. By completing and returning the enclosed form to CIC, they authorized the release of their information to the Consultant for further involvement in the evaluation.

In addition, the Consultant created communication materials, such as posters and frequently asked questions (FAQ) brochures that were distributed to the SPOs to encourage GARs to complete and return the consent form in a timely manner. The Consultant worked with the SPO representatives, to ensure that staff were adequately aware of the evaluation and its purpose and able to assist GARs in the completion of consent forms.

These consent forms were returned to the CIC via a postage-paid envelope. In turn, CIC forwarded the envelopes to the Consultant on weekly basis. By processing the consent forms, the Consultant generated a database of 1,234 potential participants, which was used to recruit GARs for further research activities (i.e., survey).

GARs who had consented to being involved in the evaluation were contacted by the Consultant through mail. A cover letter, prepared on CIC letterhead, and a hardcopy of the survey were sent to the address provided by GARs. The cover letter introduced the Consultant and explained the purpose of the survey and the various options the individual had for completing the questionnaire (i.e. hardcopy, online, telephone). The cover letter also informed the GARs that they would receive a telephone call from the Consultant to discuss their participation in the survey. These measures were taken to enhance the perceived legitimacy of the survey and increase the participant response rate.

The GAR survey administration began on November 9, 2010, and concluded on November 25, 2010, reaching the intended target of 500 completions with 501 survey responses, representing a gross response rate of 41%. As highlighted in Table B-2, the profile of GARs who participated in the GAR survey generally approximates the total profile of the sample of GARs who were selected in each case for the mail-out of informed consent. However, some differences can be noted in terms of their key characteristics. A larger share of survey respondents were male, between 25 and 64 years old, had university education, had knowledge of the official languages, and were from countries such as Afghanistan, Iraq, Colombia and Iran.

*It is expected that the sample selected for the mail-out is somewhat different from the overall GAR population, as only one person per case was selected.  Thus the marital status and gender composition are variables that are the most affected by the sampling method. The sample represents the population of cases.

Focus groups. Seventeen (17) focus groups were completed by the Consultant. These included one (1) group with RAP Working Group; three (3) groups with representatives from SPOs; and thirteen (13) groups with GAR participants. A focus group guide was developed to reflect the issues/concerns and/or experiences of each group.

A preliminary focus group was conducted in Winnipeg with the RAP Working Group. This group contained representatives from CIC National Headquarters, Operations Management and Coordination and Resettlement Services; regional program advisors; and a SPO representative from each of the four regions (i.e., Atlantic, Ontario, Prairies and British Columbia). The focus group allowed participants to provide input into the methodological approach of the evaluation, including the potential challenges and solutions, honing the terms of reference, and detailing the context in which RAP is provided.

The focus groups with GAR participants were conducted in Vancouver (2); Edmonton (1); Lethbridge (2); Saskatoon (1); Winnipeg (1); Kitchener (1); Toronto (1); Moncton (2); Halifax (1); and St. John’s (1). All groups were completed in English, except for one Moncton session, which was conducted in French. These focus groups were broken down to address each of the key themes explored by the evaluation. As such, GAR participants, in addition to reflecting the community setting in which they had been resettled, were also selected based on gender, age group, the refugee type and the processing method used by CIC to select them. Participants were recruited with help from SPOs. Each participant was paid $25.00 to help defray the cost of attending these sessions (i.e. travel, parking, child care expenses). Groups were held in appropriate facilities with refreshments provided. In total, 107 participants attended GAR focus groups.

The focus groups with representatives from SPOs were conducted via tele-conference. For one (1) of the groups, the Consultant worked with the CIC evaluation team to secure CIC facilities through major Canadian centres with the video-conference capabilities. This allowed several SPO representatives to attend the session in a centrally located venue.

In addition, the Consultant completed two (2) groups using a Skype video-conferencing application. These sessions were held with SPO representatives in locations that were outside the proximity of CIC local offices with formal video-conferencing capabilities. In preparation for these groups, the Consultant communicated with the potential participants to determine if they had the required system (PC) capabilities and hardware (i.e., webcam with a microphone). The Consultant mailed out webcams to those participants not equipped with the essential equipment. Each participant also received a confirmation letter prior to the commencement of the session, and an instruction/help manual to download and install the Skype video-conferencing application and webcam, if required. In addition, the Consultant facilitated the use of a tele-conference line for those participants that did not have access to speakers, or in case a technical issue with the application occurred.

The participants of all three SPO focus groups were provided with a $25.00 Tim Horton’s gift card and invited to purchase refreshments for their enjoyment during the session.

Secondary data sources

The secondary data sources used for the evaluation included the analysis of administrative data and document/literature review. Each of the secondary data collection methods are described in more detail below.

Document/literature review. A document review was undertaken to enhance the understanding of the context, activities, objectives, and mandates of the GAR and RAP programs. The review included:

• Legislative documents and Government of Canada policy documents;
• CIC and other government department documents related to priorities and commitments;
• Documentation on refugee needs;
• Resettlement program documents such as CVOA directives, policies, priorities (including selection approach protocols), briefing notes, financial reports, statistical reports, research documents, partnership agreements, and reports for the UNHCR, operational manuals, etc.;
• Contribution agreements, SPO reports and other related documents, including operational profiles, needs assessments, process and procedure documents, annual reports, products from their research (including analysis of pre/post assessments), and special projects and documentation related to their service delivery approach;
• Relevant stakeholder reports, including UNHCR reports and statistics and the Agenda for Protection; and
• Relevant Conventions, Declarations, Agreements, etc.

The results of the document review were used in the development of research instruments such as key informant interview guides, survey questionnaires, focus group guides and case study protocols.

The literature review was completed to help place the results of the evaluation within the context of global efforts to address the refugee issues. The literature review examined existing research on government-assisted refugee outcomes; best practices in resettlement program design and delivery from Québec and other countries; and GAR selection and processing in other countries. In particular, the review focused on the link between GAR and RAP programming, including the impact that GAR selection and processing have on the provision of RAP services and the role that RAP (and other settlement programs) have on GAR outcomes.

Analysis of administrative data. The evaluation included an analysis of the Field Operations Support System (FOSS), the immigration-Contributions Accountability Measurement System (iCAMS), the longitudinal Immigration Database (IMDB), and the database for the Immigration Loans Program. As FOSS is the main Immigration database for CIC, it contains information related to temporary and permanent residents who have entered Canada. Demographic data from FOSS (such as immigration category, date of birth, gender, country of birth, etc) was used to draw a profile of GARs who landed during the period reviewed by the evaluation. iCAMS captures detailed information on the resettlement assistance services provided to clients by service providers. Demographic data from FOSS were also downloaded into iCAMS and linked to each client receiving resettlement and settlement services. iCAMS data allowed evaluators to draw a profile of clients served and services received under RAP. The IMDB maintains linked immigration records from FOSS and tax files from the Canada Revenue Agency for landed immigrants in Canada since 1980 who have filed at least one tax return. This database is managed by Statistics Canada on behalf of a federal-provincial consortium led by CIC and provides information on the economic performance of landed immigrants in order to help understand the impact of Canada’s Immigration Program. IMDB data was used to look at the economic integration of GARs as well as their mobility across Canada

The analysis of these databases were conducted to help assess a number of evaluation issues related to the programs’ impacts such as: the effectiveness of the selection process (i.e. is GAR selection and processing efficient and effective); the appropriateness and adequacy of the RAP for the needs of the GAR population arriving in Canada; and whether the GARs obtain, and benefit from, CIC settlement services.

Methodology strengths and limitations

Strengths. The key strength of the evaluation approach included the collaborative working relationship between the Consultant and Citizenship and Immigration Canada on the evaluation of the Government Assisted Refugees (GAR) and the Resettlement Assistance Programs (RAP). This evaluation used a hybrid model and conducted international and domestic case studies to follow the experiences of GARs from selection and processing abroad to resettlement and integration in Canada. Under a hybrid model, both the Client and Consultant supplied evaluators as part of the team. The joint efforts of CIC and the Consultant were essential to engage stakeholder participation as well as understand the context of the programs to carry out the case studies. The case studies allowed the evaluators to observe the issues impacting the programs and the population served firsthand. In addition, the evaluation relied on different methodologies, both quantitative and qualitative, which were triangulated to ensure robust findings.

Challenges and limitations. One of the limitations of the evaluation methodology included a self-selection bias in terms of GARs participation in the survey and focus groups. Although every attempt was made to ensure that all GARs had an opportunity to participate in the survey, it is unclear as to whether GARs who self-selected to participate would have any inherent bias as compared to GARs who did not participate. Similarly, it should be noted that the evaluation team visited four (4) international CVOAs and that the results of the processing model used in CVOAs is based on the results/findings associated with, in most cases, the one CVOA visited. It should be noted, however, that the CVOAs visited were the ones that accounted for more than 80% of GARs processed in 2009.

• Previous
• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9 (current)
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• Next