Evaluation of the Immigrant Settlement and Adaptation Program (ISAP)

Appendix A: Methodology

The following lines of evidence were employed for the evaluation of ISAP:

• data and document review;
• interviews with 21 CIC representatives;
• interviews with nine SPOs and two stakeholders;
• six focus groups with 54 ISAP clients; and
• a telephone survey of 84 SPOs.

Data and Document Review

Where key informants supplied Goss Gilroy Inc. (GGI) with examples of materials or reports, these were reviewed in the context of developing a better sense of how the program was delivered in that region/community. websites were also reviewed, including settlement.org and integration-net.

GGI also received some high-level data from the LSIC and IMDB and this was used to develop an understanding of the context for the programs.

Interviews with CIC NHQ, Regional and Local Offices

The research team conducted interviews with 24 individuals (often in a group interview setting). Of those, six individuals were interviews at NHQ and 15 were interviewed from regional and local offices. Interviews were conducted in-person if the respondent was located in the National Capital Region or in a city visited as part of the focus groups (where possible). Otherwise, interviews were conducted over the telephone. Interviewees were invited to participate in their preferred official language.

Interviews with Service Providers and Stakeholder Organizations

GGI conducted a total of 10 service provider interviews and two stakeholder interviews. Interviews with service providers differed from those conducted as part of the survey in that this line of evidence probed more deeply into issues of program relevance, design, overall success and alternatives.

As for the interviews with CIC, interviews conducted in Ottawa and in the sites visited for the focus groups were conducted in person (where possible). The others were conducted by telephone. Respondents were invited to participate in the interview in the official language of their choice.

Focus Groups with Clients

A total of six (6) focus groups were undertaken with program clients. Focus groups were held in service provider locations, with the exception of one of the Ottawa group conducted in the GGI boardroom. Focus groups were carried out as follows:

• 1 in Ottawa;
• 3 in Toronto;
• 1 in Moncton (in French); and,
• 1 in Edmonton.

Up to two service providers were asked to assist in the recruitment of each focus group (each was paid $75 for each group of five participants they recruited). They were asked to recruit a mix of clients who:

• are no longer accessing ISAP services, as well as some clients who continue to receive services;
• are men and women;
• are of different age groups;
• are single individuals and immigrants with families;
• are of different ethnic/racial backgrounds; and
• are government-assisted refugees (GARs) (just one or two per group).

In order to get a different perspective on the services offered, some focus groups had participants who spoke some language other than English or French. In those cases, interpreters were available for each non-English or non-French speaker at the group.

In all, 10 participants were recruited for each group with the intention of having eight show up for the discussion. Overall, groups were well attended. Participants were offered an honorarium of $50 to offset any expenses they may have incurred to participate (e.g., child minding, transportation, parking).

GGI prepared a focus group moderator’s guide. This guide is semi-structured, allowing the moderator to pose questions that flow from one issue to the next. The moderator made efforts to involve all participants in the discussion. The other evaluation team member attending the discussion was responsible for taking notes. Due to the fact that many newcomers are sensitive about the confidentiality of the information being provided, no audio or video tapes were produced.

Survey of Service Providers

A telephone survey of all service providers delivering ISAP was conducted to assess the ongoing relevance, success, design and delivery of the programs. All current service providers (i.e., those who have signed a contribution agreement for the 2004–05 fiscal year) were contacted. GGI managed to complete 84 of the 120 names provided by CIC (representing a response rate of 70 percent).

Prior to the fieldwork commencing, a notification letter was sent to all potential respondents in the sample to outline the purposes of the study, the specific survey, and the time requirements.

Potential respondents were contacted by telephone to ascertain whether they were willing to participate in a phone interview. If they were willing to participate, the respondent was asked to complete the interview over the telephone at that time. Should this time not be convenient, an appointment was arranged when the GGI interviewer could call them back to complete the interview.

All potential respondents were called up to five times before being considered a non-response. Service providers were called on different days and at different times during the business day (in their time zone), to ensure that respondents were not called back at the same time each day. All phone interviews were conducted using Computer Assisted Telephone Interviewing (CATI) systems. Interviews were conducted in the preferred official language of the respondent. Interviews lasted between 40 and 50 minutes, on average for the ISAP component of the questionnaire.

Case Studies of Innovative Practices

Three case studies of innovative practices in ISAP delivery were conducted as follows:

• School Support Program (Atlantic) – a school-based outreach program that locates a settlement work in schools in selected schools and communities. The worker provides support and resources to students and their families to assist in the settlement process;
• Newcomer Information Centres (NIC) (Ontario) – addresses the needs of self-directed, skilled workers in the Toronto area with more advanced technological and language skills; and,
• Job Search Workshops (JSW) (Ontario) – to provide newcomers with the skills and resources to facilitate the job-search process and is similar to other employment programs offered through ISAP employment counselling.

For each of the case studies, a document review and up to two interviews were conducted. For each, the factors that make the project/case an innovative practice and how it might be applicable to a wider context and in different jurisdictions were explored. A short (three to four page) case study write-up was prepared, including conclusions regarding wider applicability of the case (see Appendix C).

<< contents | previous | next >>