ARCHIVED – Summative Evaluation of the Metropolis Project Phase II: Knowledge Transfer Activities and Impacts

2. Evaluation Methodology

2.1 Evaluation Issues and Questions

The evaluation matrix was developed to frame the evaluation questions, performance measures and methods (see Appendix A). The list below outlines the evaluation questions that guided the evaluation. The findings reported in section 4 draw upon the full range of methods.

Evaluation Questions

  1. Have the Metropolis Centres successfully integrated key policy issues identified by federal funding partners into their research plans?
  2. Do the Centres, Secretariat and Federal Consortium operate as effective knowledge brokers in the knowledge transfer process?
  3. Do federal policy-makers access Metropolis research?
  4. Do departments support the access and use research in policy-making?
  5. Are Metropolis research products relevant to government policy makers?
  6. Have the Metropolis Centres conducted macro (including pan-Canadian), comparative and longitudinal studies to support policy development?
  7. Has knowledge transferred from Metropolis informed and influenced government policy development?

2.2 Methodology

The evaluation methodology employed multiple lines of evidence as a means to enhance the reliability and validity of the information and data collected. The methodologies included:

  • A literature review and synthesis
  • A document review
  • Interviews
  • Case studies (which included interviews, document reviews, and focus groups)
  • A review of administrative data
  • A survey of potential Metropolis research users

2.2.1 Literature Review

A systematic search of a broad range of KT and public policy peer reviewed literature was performed using best practices in bibliometric research methods. The results of this literature review were summarized and synthesized in a paper by H.D. Dickinson (with P. Graham), 2009. Knowledge Transfer & Public Policy: A Literature Review and Synthesis. Appendix A in that paper summarizes the bibliometric research methods used for the literature review.

2.2.2 Document Reviews

Documents reviewed for the evaluation include background documents, such as the Metropolis Phase II memorandum of understanding, program documents such as the Results Based Management Accountability Framework (RMAF), previous review and evaluation reports, and newsletters and other Metropolis products. The document review was conducted using a customized template to extract relevant information and organize it according to indicators and evaluation questions. Appendix B contains a list of the documents reviewed.

2.2.3 Stakeholder Interviews

A total of thirteen interviews were conducted with representatives of the federal funding partner organizations. These included two from each of CICNote 8 ] and CMHC, four from both HRSDC and PCH, and one from SSHRC. Interviewees included Directors General, Directors, Managers, Senior Policy Analysts or Officers, and Senior Researchers. Interviewees were selected in consultation with the Metropolis Evaluation Advisory Committee and CIC evaluation team. See Appendix C for list of interviewees. It should also be noted that, as part of the case studies described below, other interviews were also done with CIC Secretariat staff and representatives of federal funding partners.

All interviewees received the interview guide in advance of their interviews which were conducted either in-person or by telephone. A template, organized according to evaluation indicators and questions, was used to summarize the interview data which were then coded and analyzed. See Appendix D for the interview guide.

2.2.4 Case studies

Case studies consisting of interviews, focus groups, and Centre specific document reviews, were conducted with the Metropolis Secretariat and the four original Centres of Excellence – the Centre de recherche interuniversitaire de Montréal sur l’immigration, l’intégration et la dynamique urbaine (IM); the Joint Centre of Excellence for Research on Immigration and Settlement (CERIS); the Prairie Centre of Excellence for Research on Immigration and Integration (PCERII); and the Vancouver Centre of Excellence for Research on Immigration and Integration in the Metropolis (RIIM).

Focus group participants and other research users interviewed as part of the case studies were identified by the Metropolis Centres or Secretariat, and consisted of people who were reportedly users of Metropolis research products or participants in a variety of other research activities of the Centres.

Two sets of interviews were done. The first was with Directors and Domain Leaders at each of the Centres. The second set of eight interviews was representatives from some of the federal funding partner organizations – including six CIC representatives, one HRSDC representative and one representative from the Department of Justice. The interviewees occupied a range of positions including Directors, Directors General, an Associate Assistant Deputy Minister, senior research officers and a policy officer/analyst.

A focus group was conducted involving federal government representatives who had been identified as users of Metropolis research products.

Finally, documents, such as activity reports, research products, and annual reports, from each of the Centres were reviewed.

2.2.5 Administrative data

Administrative data for the five-year period covered by this evaluation that was reviewed included Centre activity reports, budget information and other data received from the Metropolis Secretariat, such as website statistics and mailing list information.

Activity reports for each Centre and the Secretariat were originally developed by the CIC evaluation division based on information available on the Metropolis website. These activity reports were then provided to each Centre and Secretariat for verification. Although each of the four Centres and the Secretariat verified and updated the activity reports, additional information requested for the reports, but unavailable from the websites, was provided by only two of the four Centres.

2.2.6 Survey

A survey was developed and distributed to 1678 federal government employees of the four federal funding partners targeted for this evaluation. Note that although SSHRC representatives were interviewed for this evaluation, they were not part of the survey or case study methodologies, since SSHRC employees are not part of the target audience for Metropolis research and activities.

Members of the Evaluation Advisory Committee were asked to provide a list of all their employees during the Metropolis Phase II time period who potentially needed Metropolis-type research to conduct their work. CIC, PCH and CMHC provided a list of potential Metropolis users who worked in their department/ organization during that time. However, because of a re-organization within the department that occurred in December 2003, HRSDC could not identify potential research users from the Phase II timeframe. Instead, HRSDC provided a list of current potential users of Metropolis. In order to minimize the impact caused by this, the survey included two filter questions to identify respondents who should not have been included in the lists (i.e., those who did not work in the federal government during Phase II, and those who did not need Metropolis-type research for their work).

A total of 552 responses were received to the survey, which represents a 33% response rate. Of those, 278 respondents qualified as federal government employees who needed Metropolis-types of research products during Phase II. These 278 responses are the main focus in the analysis of potential user perceptions and assessments of Metropolis Project research [ Note 9 ].

It is noteworthy to mention that, due to non-response on some questions, the figures do not always add up to the initial numbers of respondents who answered the survey or who were identified as potential users. Throughout the report, even though the analysis will be mainly targeting the 278 potential Metropolis users identified in the survey, the numbers on which the counts will be reported (valid answers) may differ slightly from the initial count due to non response on each specific survey question.

The proportion of potential respondents provided by each of the four participating departments changed from the original survey list, to the group of respondents, to the group who made it through the filter questions (Table 2).

Table 2: Repartition of Survey Respondents from Participating Federal Funding Partner Organizations

Department

Total

Survey Respondents

Respondents – Potential Users

HRSDC 53.9%
(n=904)
41.8%
(n=205)
38.7%
(n=106)
CIC 30.2%
(n=507)
39.8%
(n=195)
43.4%
(n=119)
PCH 5.7%
(n=96)
8.0%
(n=39)
10.6%
(n=29)
CMHC 3.0%
(n=50)
2.4%
(n=12)
1.5%
(n=4)
Other 7.2%
(n=121)
8.0%
(n=39)
5.8%
(n=16)
Total 100.0%
(n=1678)
100.0%
(n=490)
100.0%
(n=274)

The proportion of survey respondents from HRSDC (41.8%, n=490) was relatively under-represented compared to their proportion of the original survey population (53.9%, n=1678). This may be due, in part, to the broader solicitation for interviewees in this organization.

Profile of Potential Users (survey respondents):

As mentioned previously, 278 survey respondents qualified as potential users of Metropolis-types of research products. During Phase II, over ninety percent of these (out of a total of 274 respondents who provided a valid answer for this question) worked for three of the federal funding partner organizations - CIC (43%), HRSDC (39%) and PCH (11%). About 65% of the 273 respondents who provided their location were in the National Capital Region. About 8% of the 274 respondents who gave information on their position identified themselves as being “senior management” and 12% as “other management” (see Table 3). Most of the respondents fit into the two categories of “program/project advisor, analyst or officer” or “policy advisor, analyst or officer” (61%).

Table 3: Percentage of Survey Respondents by Position

Position %
Senior Management 8.4%
Other management 11.7%
Program / Project Advisor, Analyst or Officer 37.6%
Policy Advisor, Analyst or Officer 23.7%
Research Advisor, Analyst or Officer 17.5%
Other 1.1%
Total 100.0%

Most of the 274 potential users worked either in policy development (29%), research (24%), or program design or delivery (25%). Another 16% worked in operational or client services. The remaining 6% of respondents worked in communications, evaluation, IT/Systems, or “other”.

About half the respondents (47%) reported being either ‘quite familiar’ or ‘very familiar’ with Metropolis, while 33% reported being ‘not very familiar’ and 19% ‘not at all familiar’.

The survey results will be presented for all potential users throughout the report. The approach was adopted as knowledge transfer activities are intended to reach all potential users and not only a sub-group of people who are highly familiar with the Project. However, further analyses of the survey results based on user familiarity with Metropolis were conducted as respondents perceptions and opinions may be modulated by their familiarity with Metropolis. Appendix F presents these results.

2.3 Limitations of Methodology

This section outlines limitations to the methods and data used. Generally, the use of multiple independent lines of evidence helps to triangulate findings and increase confidence in the overall results.

2.3.1 Representativeness of data collected

A list of interviewees and case study participants was developed by the Advisory Committee and the Metropolis Centres. The selection of participants was not representative and it is not possible, therefore, to make generalizations about the program based only on these findings. Similarly, federal funding partner representatives on the Evaluation Advisory Committee identified potential survey recipients. The resulting lists of recipients may not have included all possible users of Metropolis-type research within those organizations. A further limitation was the fact that the survey recipient list included only those potential users identified by the four major funding partners involved in the evaluation. Other departments may also make use of Metropolis-type research. The overall effect of these limitations is to present a conservative picture of users and uses of Metropolis research.

Case study users who were interviewed and included in focus groups were identified by the Centres. This introduces a potential bias since Centres may have identified those who are more involved in their Centres, or who generally are happier with their Metropolis experiences. It should be pointed out, though, that comments received during the case studies were balanced. In addition, focus groups conducted across the four Centres and the Secretariat sometimes drew the same participants.

2.3.2 Timing

The evaluation covered the period from 2002 to 2007, but the data was collected in 2008. Therefore, stakeholders were asked for views and opinions approximately one year after the end of Phase II. This could affect the accuracy of recollections.

2.3.3 Administrative Data

Outputs of the Metropolis Centres and Secretariat are not systematically reported. Quantitative information on outputs for the evaluation was derived from annual reports and other activity reports the Centres were asked to complete. Annual reports are not produced in a standard format, therefore, the form and content of activities and products reported varies. Activity reports for the Centres were originally developed by the CIC evaluation division and provided to the Centres for updates or corrections. Not all Centres provided all the requested information. Specifically, not all of the reports contained information relating to the proportion of Metropolis research that was comparative, pan-Canadian, or longitudinal. Nor did all of the reports include information about the percentage of the research projects that were funded by Metropolis. Only two of the four Centres provided this information.

___________

8. Several other CIC senior managers were interviewed for the cases studies.

9. The survey was aiming at the population of potential Metropolis users rather than at a sample. Therefore, no confidence interval is reported since all potential Metropolis users that worked for the federal government at the time of Phase II were targeted be the survey. Likewise, for the same reason no significance test will be provided for the analyses that are presented in this report.

Page details

Date modified: