2. Methodology

2.1 Approach, design and methods

The terms of reference for this evaluation, including the methodology and the evaluation matrix, were approved by the CIC Departmental Evaluation Committee prior to the start of this evaluation.

The level of effort and lines of evidence used in this evaluation were calibrated in order to take into account the fact that the element being evaluated is a grant. A grant is “a transfer payment subject to pre-established eligibility and other entitlement criteria” and “is not subject to being accounted for by a recipient. The recipient may be required to report on results achieved”^{Footnote 7}. In this instance, the recipient, the government of Quebec, is not required to report on results achieved. As a result, the amount of information available for this evaluation was limited, and the evaluation was calibrated accordingly.

A goal-based approach was used for this evaluation. It addresses the relevance and performance of the grant in relation to its key goal/objective which was to provide reasonable compensation to the province for the withdrawal of services by the federal government, as long as the services, when considered in their entirety, correspond to the services offered by Canada in the rest of the country and the services provided by Quebec are offered without discrimination to any permanent resident of Quebec.

For the most part, a non-experimental design was used, relying heavily on a review of program documents and interviews to answer the evaluation questions. That said, a quasi-experimental design underpins the longitudinal data analysis used in the development of the economic profile of immigrants to Quebec.

Five main lines of evidence were used to collect data and information: a document review, interviews with key informants, an analysis of administrative data, an analysis of longitudinal data, and an assessment of financial information. The methods used and the amount of information collected were subject to certain limitations which are described later in this section.

Data collection for this evaluation was conducted between June and December 2011.

2.1.1 Document review

The document review represented the main line of evidence for this evaluation. The review of internal documents provided information on the origins and objectives of the grant. Documents regarding the settlement and integration services provided in Quebec were identified primarily through a web-based search. This search also identified relevant research reports and papers related to federal-provincial relations and the provision of settlement services. Annex II provides a list of the main documents reviewed.

2.1.2 Interviews

Interviews were conducted in order to obtain information on the relevance of the grant, particularly with respect to the context under which the grant was first created and its current linkages to CIC priorities, the mechanisms in place to monitor the grant, and the funding formula used to calculate the amount of the grant.

Key informants were identified in consultation with CIC’s International and Intergovernmental Relations and Integration Branches and were selected based on their knowledge of the grant and their involvement in its management. The Evaluation Division developed a series of questions to guide the interviews, based on the evaluation matrix. A list of interviewees and the list of questions are provided in Annex III.

A total of 13 individuals were interviewed either by telephone or in person (see Table 2.1). Most interviews were conducted with current or former CIC staff in the IIR and Integration Branches at National Headquarters (NHQ) or in the Quebec Region. An interview was also conducted with a representative of the service provider community in Quebec. Interviews with provincial representatives and with other stakeholders in Quebec were not undertaken, as explained in the limitations section of this report. Interviews lasted between 20 minutes and 90 minutes.

2.1.3 Analysis of administrative data

Data from CIC’s Field Operations Support System (FOSS) were used to create a profile of Quebec permanent residents. Given that the grant to Quebec was implemented in 1991, the profile covers the last 20 years. For this profile, all permanent residents that declared Quebec as their intended destination, whether or not they were selected by the province^{Footnote 8}, were included in the profile, as the provision of settlement services is not limited to Quebec selected immigrants but to all immigrants residing in Quebec. For comparison purposes, the analysis also provides information on permanent residents arriving in the rest of Canada.

2.1.4 Analysis of longitudinal data

The source of information for the analysis of longitudinal data was the Longitudinal Immigration Database (IMDB). The IMDB is a database that is managed by Statistics Canada on behalf of a federal-provincial consortium led by CIC. This database links records from CIC’s FOSS to Canada Revenue Agency (CRA) tax files (mainly from the T1 personal tax return). It is a tax filer database that contains information on all landed immigrants in Canada since 1980 who have filed at least one tax return. At the time of the evaluation, the IMDB contained information up to and including the 2008 taxation year.

For the purposes of this evaluation, a specific sub-set of the population included in the IMDB, namely immigrants destined to Quebec^{Footnote 9}, was used to provide an economic profile of permanent residents in Quebec.

2.1.5 Assessment of financial information

Two types of financial information were gathered: information on the amount of the grant disbursed in each year since 1991, the first year of the Accord, and information on the administrative cost to CIC of managing the grant. For comparison purposes, information was also collected on funding for settlement and integration services in other provinces and the associated administrative costs to CIC.

2.2 Limitations

Prior to beginning this evaluation, a letter was sent to provincial representatives to inform them of the evaluation, and ask them whether they would be willing to share relevant documents and collaborate with CIC. The province declined to participate.

Under the Accord, the province of Quebec has the sole authority for the administration and delivery of settlement, resettlement and integration services for clients in that province. It is not required to report to the Federal government on the use of the funds. Furthermore, as the Federal government withdrew from the services to be provided by Quebec, CIC has limited interaction with service provider organizations in the province.

As a result, only publicly available documents were used to inform this evaluation and no interviews were conducted with representatives of the provincial government. To mitigate the lack of direct access to stakeholder groups and newcomers, an interview was conducted with a representative of an organization that represents over 100 service provider organizations in the province of Quebec.

In addition, while the MICC publishes an annual report which it submits to its National Assembly, this report does not provide a breakdown of how the grant money is used within the department or by other provincial government departments. As a result, it is not possible to link the funds to specific programming offered to immigrants in the province.

Finally, while the IMDB allowed for an analysis of the economic outcomes of immigrants destined to Quebec, it is challenging to determine to what extent these outcomes are due to the settlement services funded by the province through the grant to Quebec. As it has been impossible to identify those who benefited from settlement services from those who did not among individuals found in the IMDB, attribution to the different programs offered to immigrants is not possible.

• Previous
• 1
• 2
• 3
• 4
• 5
• 6(current)
• 7
• 8
• 9
• Next