Evaluation of the Host Program

2. Evaluation methodology

The evaluation used multiple lines of evidence including qualitative and quantitative methods. The following section describes these primary and secondary data sources as well as the strengths and limitations of the study. A description of the detailed methodology used in this evaluation is included in Appendix A.

2.1. Primary data sources

2.1.1. Interviews

Thirty-two interviews were conducted with three groups of Host Program stakeholders to collect information on all issues addressed by the evaluation. The groups of interviewees included CIC management (n=12), other CIC staff (n=15), and Provincial/Territorial representatives (n=5). The regional distribution of each group is shown in Table 2-1.

Table 2-1: Regional distribution of interview participants
  Location Total
NHQ ON ATL AB YU BC MB
CIC Directors & Managers 7 2 2 1 - - - 12
CIC Program Officer & Local Managers 4 7 2 1 1 - - 15
Provincial/ Territorial Representatives - 1 1 1 - 1 1 5

2.1.2. Surveys

In total, 336 questionnaires were completed by various Host Program stakeholders (Table 2-2). Four unique surveys were administered to various stakeholders including SPO managers, SPO staff delivering the Host Program, clients, and volunteers.

Table 2-2: Number of stakeholders surveyed
  Host Program Stakeholders' Groups Total
SPO Managers/Directors SPO Staff Clients Volunteers
Number of Respondants 32 44 92 168 336

2.1.3. Focus groups

Separate focus groups were conducted with CIC officers, SPO representatives, and a combination of Host volunteers and clients. In total, 18 focus groups involving 183 different Host participants/stakeholders were conducted in six cities across Canada. One focus group with SPO was conducted in French. The purpose of the CIC and SPO staff focus groups was to collect data on Program need, delivery and management. Focus groups with clients and volunteers on their experience participating in the Program. The breakdown of participants is shown in Table 2-3.

*NOTE: Focus groups of CIC officers and SPOs included those that delivery ISAP A and/or Host.

Table 2-3: Host focus groups by location and type of participant*
Location Number of focus groups Program Participants SPO Representatives
CIC Officers Host Volunteers Host Clients
Toronto 3 9 3 6 16
Hamilton 2 6 13 11 n/a
Mississauga 3 8 4 6 15
Halifax 3 3 9 3 8
EdmontonFootnote 3 4 4 7 6 18
Calgary 3 3 4 5 16
Total 18 33 40 37 73

2.2. Secondary data sources

2.2.1. Document review

A detailed document review was undertaken with a particular emphasis on program relevance. Three major areas were covered under the relevance section: the continuing need for the Host Program; consistency with the priorities of the Government of Canada and CIC; and federal roles and responsibilities in funding and developing such programs. The document review also provided information relating to particular issues of Program performance, design and delivery. A wide variety of federal, departmental and Program documents were reviewed as well as SPO materials including websites, training and promotional materials, Program statistics, guidelines, tools and intake/needs assessment forms. See Appendix B for a complete list of documents reviewed.

2.2.2. Literature review

The literature review included on-line and library materials and archives. The literature review provided contextual information regarding the need for Host, the role of networking in immigrant settlement, the relationship with other settlement programs, and a comparison of similar programs in other jurisdictions. A list of references can be found in Appendix C.

2.2.3. Administrative data analysis

The iCAMS and the Integrated Financial and Material System also known as SAP, were reviewed and analyzed to prepare a profile of the service providers, program services, and clients. iCAMS is an Internet-based system through which SPOs provide CIC with information about their services and clients. iCAMS has collected information on Host since April 2004Footnote 4. SAP is a financial data system that tracks all funds committed and spent by CIC and serves as a central repository of financial information for all contribution agreements (CAs)Footnote 5.

For the Host Program, iCAMS collects information on the number of clients matched for individual/ family-based activities, the number of clients participating in ongoing activities as well as one-time group activities. iCAMS is augmented by information from CIC’s Field Operations Support System (FOSS), which allows a comprehensive analysis of the profile of immigrants participating in the Program by their individual characteristics (i.e., gender, country of origin, age, etc.). iCAMS also collects information on the number of volunteers, the number of orientation sessions and the number of volunteers trained. Other services captured in iCAMS are the SPO promotion/recruitment sessions. Information not captured in iCAMS includes: the number of volunteers who are involved in the group activities, the time spent by volunteers with clients (during individual matches or group activities), or the type of group activities.

2.3. Evaluation limitations and strengths

The methodology used in this evaluation had the following limitations:

Lack of access to extensive reliable secondary data - Data on the numbers and characteristics of clients served may be incomplete. A comparison of the data in iCAMS and SAP databases demonstrates that a significant percentage of SPOs are not reporting in iCAMS. According to SAP, between 2004/05 and 2007/08 contribution agreements were signed with 58 SPOs to deliver the Host Program; however, only 45 SPOs (78%) reported data into iCAMS. Similarly, the SPO focus groups not only found that some SPOs are not reporting data in iCAMS, but also that some were unsure regarding whether they were doing so correctly. In addition, some clients will not be reported in iCAMS because they are unwilling or unable to provide their Permanent Resident card number. This further constrains the use of iCAMS data to analyse the reach of the Program, to conduct year-to-year comparisons, and to conduct adequate cost-effectiveness analyses. As a result of these constraints, any conclusions based upon iCAMS data presented in this report should be considered with caution.

Representativeness of data collected – As discussed previously, information on the entire client and volunteer populations was not available through the data reported in iCAMS. It was therefore neither possible to obtain a random sample nor to compare characteristics of the survey respondents to that of their respective populations, to determine if they were statistically representative. As a consequence, the results from the client and volunteer surveys can only be used as an indication of the perception of those two groups, and cannot be interpreted as being representative of the entire populations.

Lack of information on client outcomes – While CIC collects financial information through SAP and output data through iCAMS, it does not have a systematic approach for collecting client outcome information. CIC is currently working on addressing this shortcoming, although a new methodology for collecting data on outcomes will not be available for another two years. Thus, the evaluation relied significantly on surveys and focus groups to obtain information on client outcomes.

  • The evaluation used several methods to enrich the data collection and increase confidence in the overall results. The strengths of the evaluation methodology include:
  • The use of multiple lines of evidence including qualitative and quantitative data allowed for the triangulation of findings;
  • Multiple stakeholders were consulted, including five distinct groups of stakeholders (i.e., clients, volunteers, SPOs, CIC representatives, and provincial/territorial representatives) which increased the reliability of data;
    Regional representation was obtained through the surveys, interviews and focus groups;
  • The client survey was available in 9 different languages and offered through three different modes of communication (online, on paper, by phone) to increase the number and range of clients willing to participate; and
  • Over one-half of the SPOs (55%)Footnote 6 involved in delivering the Host Program participated in the evaluation.

Page details

Date modified: