ARCHIVED – Summative Evaluation of the Metropolis Project Phase II: Knowledge Transfer Activities and Impacts

4. Findings

The findings are organized into three main sections that correspond to three essential dimensions of KT – research processes, products and application (impact). The research questions list in section 2.1 framed the findings.

In section 4.1, several aspects of Metropolis research processes are examined. First, findings are presented related to participants’ perceptions of, and satisfaction with, the processes used by the Centres and federal funding partners to integrate research priorities into Centres’ research agendas. Second, the processes used by Centres to convey these priorities to researchers are examined and assessed. Third, satisfaction of federal funding partners with these processes is reported. This is followed in a fourth sub-section by a summary of federal funding partners’ satisfaction with the results of these processes. Fifth, the issue of knowledge brokering is reported on, including the degree to which Metropolis Centres, federal funding partners and the Metropolis Secretariat were seen to have acted as knowledge brokers during Phase II. Finally, the degree of federal funding partners support for, access to, and use of, research products is summarized.

Section 4.2 reports federal funding partner research users’ general satisfaction with Metropolis research products and other related activities like conferences and workshops. This section also presents more specific information on user satisfaction with the degree to which the Centres produced longitudinal, comparative and pan-Canadian research as per the Phase II MOU.

Section 4.3 reports the results of the inquiry into whether Metropolis research products informed and influenced government policy and program development.

4.1 KT processes involving Metropolis researchers and federal research users

The literature shows that collaboration between researchers and research users is important for effective KT. The clear communication of research user needs and priorities to researchers is preliminary to the transfer and use of research knowledge by policy makers. In this section we look at whether, in general, Metropolis Centres successfully integrated into their research plans key policy issues identified by federal funding partners (see Question 1, section 2.1). We also report on the knowledge brokering role of Metropolis in the KT process (see Question 2, section 2.1). The next topic is an examination of whether federal government policy makers access Metropolis research (see Question 3, section 2.1). Finally we look at the extent to which research users employed by the federal funding partners access, use and receive employer support for using research (see Question 4, section 2.1).

The emphasis in this section is on the nature and effectiveness of researcher-research user relationships within the context of Metropolis. The necessity for this as a determinant of effective KT is an overarching message from the KT literature.

4.1.1 What are the Centre processes for soliciting research priorities from federal partners & what are the federal partner processes for conveying their research needs to Centres?

In this section the two interrelated questions above are explored along with the issue of whether the Centres successfully integrated federal funding partners’ research priorities into their research plans.

Conclusion: With regard to these questions responses were mixed. Although all Centres identified processes used to solicit policy needs from departments, and to relay these priorities to researchers, there was little ongoing and formal soliciting of policy research needs from the federal funding partners. Most interviewees noted that their input into the research agenda depended upon their informal conversations and inter-personal relationships with researchers.

Context: The Metropolis evaluation conducted in 2000 mentioned that “improvements are necessary to make the Centres’ research activities more relevant to the federal partners. The Centres, with the Secretariat’s assistance, must better integrate into their research programs key policy issues that the federal funding partners have identified.” [ Note 11 ]

The Centres and the federal funding partners share responsibility for ensuring that policy research needs are effectively communicated and acted upon. In order to assess the extent to which the Metropolis Centres have successfully integrated into their research plans key policy issues identified by federal funding partners, this evaluation examined three interrelated processes necessary for this to occur. First, the processes used by Centres to solicit input from federal funding partners on key policy issues. Second, the processes used by federal funding partners to inform Centres of their key policy research needs. Third, the Centres’ processes to convey these policy needs to the research community.

Perhaps the most formal means of conveying policy research priorities to the Centres was through a memorandum of understanding (MOU) signed prior to the start of Phase II. Federal funding partners expended considerable effort to develop a list of research priorities as part of the Phase II MOU. The result was eleven major policy issues that the federal funding partners regarded as strategic priorities [ Note 12 ]. According to the MOU, Centres were “required to devote 50 percent of their core grant to studying these topics.” [ Note 13 ] However, 100% of the total budget had to be spent on work within the domain structure, which had been determined by the federal departments at the launch of the project. The requirement to spend 50% of the budget on more focused questions within those domains was a means to refine the articulation of the federal priorities, as identified in the previous evaluation.

Findings: Processes for receiving input from federal funding partners regarding their policy research priorities varied by Centre, although certain elements were common to all. Among these common processes were the inclusion of federal representatives on governance committees, annual research retreats, informal discussions, and discussions at Metropolis events.

Survey respondents were asked to identify, from a list of possible processes, those they had used to convey research priorities to Metropolis. Table 4 shows the percentage of survey respondents (n=274) who reported participating in a variety of processes as a means of communicating their research priorities to Metropolis Centres.

The two processes most frequently reported were discussions with individual researchers (31%) and Centres’ workshops (22%). Table 4 clearly shows that most survey respondents were not involved in all these processes. This is expected, however, because most of those who would make use of Metropolis research products or participate in research related activities would not be involved in communicating research priorities to Metropolis. Only a limited number of individuals in each department would be involved in providing research priorities to Metropolis.

Table 4: Percentage of potential users from federal funding partners that participated in various processes to convey research priorities to Metropolis Centres

4.1.2 What processes are used by Centres to convey federal funding partners’ research priorities to researchers?

Conclusion: Several processes used to convey federal funding partners’ research priorities to researchers were identified.

Findings: Interviews with Centre directors and domain leaders revealed several processes used to convey research needs and priorities to the research community, including the following:

• The Call For Proposals (CFP),
• Through the networks developed and/or domain (node) meetings,
• The annual research retreats,
• The Centres’ websites, and
• Through other Metropolis activities or events, such as conferences and workshops.

Call for Proposals (CFP) process: Priority themes, from the Phase II MOU, or as identified at the Centres’ annual research retreats, were included in Centres’ CFPs to researchers. As well, one of the criteria for assessing research proposals was the degree to which they addressed one or more of the priority areas. For some Centres, the proposals were required to demonstrate that the research would involve consultations with policy-making bodies, and also to show how the research was expected to influence policy. For many Centres, but not all, federal representatives participated on the grant review committees, or on the committees that approved the Centre’s funded research. Most Centres required 50% of funded projects to align with research priorities. One Centre, however, required all research projects to align with identified research priorities. In this respect it also should be noted that interviewees indicated that the eleven priority areas included in the MOU were very broad and that “almost anything could fit” into them.

Networks / Node meetings: Domain leaders conveyed research priorities to researchers though domain meetings and other modes of communication. Reportedly, however, some domains were better at this than others. As well, in some cases, domain leaders would provide guidance to researchers on how to prepare proposals in response to the CFP. This guidance would include encouragement and suggestions on how to address the research priorities.

Researchers participated in Metropolis events: Through their participation in Centres’ annual research retreats researchers were exposed to discussions relating to research priorities and there they could hear the priorities articulated by the federal funding partners in attendance. As well, when attending other events such as conferences and workshops, informal discussions took place involving researchers and departmental representatives. According to Centres’ directors and domain leaders, researchers could become aware of departmental research priorities through these informal discussions.

Through the Centres’ websites: Centres identify the research priorities on their individual websites, mainly with regards to their CFP processes. Therefore, researchers who visit the website, and in particular those who are interested in responding to the CFP, would have become aware of the research priorities on these websites.

4.1.3 Are Federal funding partners’ satisfied with the processes?

Conclusion: The greater the degree of involvement of research users the greater the satisfaction with the processes used for communicating policy needs and priorities to Metropolis researchers.

Findings: Out of 161 survey respondents who provided a valid answer [ Note 14 ], 43% were satisfied with the processes for identifying and communicating policy information needs and priorities to Metropolis while 27% were neutral, 10% were dissatisfied, and 20% did not know. When including only those who had indicated they had been involved in one of the previously mentioned processes for identifying and communicating research needs and priorities (n=105), the satisfaction rate increased to 61%. The percentage of these individuals who were dissatisfied was 11%, with 22% providing a neutral response and 6% indicating they did not know.

With regard to specific processes, most survey respondents found them to be useful, with the most useful being discussing priorities directly with researchers and the least useful being through a Board of Directors (see Table 5).

Table 5: Federal funding partners’ assessments of the usefulness of existing processes for communicating research priorities to Metropolis researchers*

* Useful includes “quite useful” and “very useful” responses. Not useful includes “not at all useful” and “not very useful” responses.

4.1.4 Are federal funding partners satisfied with the outcomes?

In the previous section we examined research users’ satisfaction with the processes for including federal funding partner research priorities on Metropolis Centre research agenda. In this section we look at federal research users’ satisfaction with the extent to which their policy research priorities were integrated into Centre research plans and programs.

Conclusion: Federal funding partner responses were mixed in this regard, some expressing positive views about the extent to which their needs and priorities were integrated, while others expressed the opposite.

Findings: Among the 214 valid responses, one quarter of survey respondents indicated that their needs and priorities were ‘often’ or ‘always’ integrated into Centres’ research plans, another 24% said ‘sometimes’, 7% said ‘never’ or ‘rarely’, and 44% didn’t know.

The views of focus group participants were equally mixed. Most focus group participants noted that the extent to which they were able to provide input into establishing the Metropolis research agenda depended upon informal conversations, or the relationships they had developed with their contacts at the Secretariat or the Centres. It was also stated, however, that it was impossible to know whether the input they provided through research retreats or otherwise, was incorporated into the research conducted since there was no feedback loop. Some focus group participants indicated there was little real soliciting of research priorities, whereas others indicated their frequent interactions with the Secretariat did provide adequate opportunities to put forward research priorities. They also noted there was no initiative from Metropolis to evaluate the extent to which Metropolis research aligned with the stated priorities of federal funding partners [ Note 15 ].

Both federal funding partners and case study research users who were interviewed also had mixed assessments of the effectiveness of Phase II processes for soliciting and incorporating key policy needs into Metropolis research plans. Just over half (11/21) of those interviewed, for example, indicated existing processes were ‘effective’ or ‘somewhat effective. Six of 21 said they were ‘not very effective’, and 4 of 21 did not answer the question.

Despite dissatisfaction with the fact that Metropolis provided little in the way of ongoing, formal opportunities for federal funding partners to communicate their research priorities, there was broad-based agreement that the networking opportunities offered by Metropolis activities, and the access to a large pool of researchers with expertise on immigration and diversity issues were significant benefits of the Metropolis project. Among the 268 valid answers, over four-fifths of survey respondents (82.5%) indicated it was ‘quite important’ (41.4%) or ‘very important’ (41.0%) for them to have access to a network of people working on population migration, settlement and integration of immigrants and/or cultural diversity issues. This was echoed by federal funding partners interviewed, almost all of whom said Metropolis was important to their department (12/13), and almost half of whom noted the networking opportunities that Metropolis facilitates (6/13).

This is exactly what one would expect based on the KT literature. The establishment of these relationships and networks has been found to be among the best means to ensure effective KT. Unfortunately the opportunities for this during Phase II appear to have been mainly informal. This means the opportunity to take more advantage of the KT potential inherent in collaborative relationships between the whole range of researchers and research users was not fully taken advantage of during Phase II by either Metropolis researchers or federal funding partners. Contributing to these missed opportunities is the fact that, according to Secretariat staff, the engagement of federal funding departments varied, and changed over time for a number of reasons, including career mobility.

The commonly found tension between researcher driven and research user driven research emerged in the interviews, focus groups and case studies. Some academics associated with Metropolis, in response to expressions of dissatisfaction with the fit between research user priorities and the actual research that was funded, asserted that research proposals and, therefore, the relevance of the research products produced, depended on the interests of individual researchers. Whether or not research was conducted on a particular topic (i.e., policy-relevant research) was entirely dependent on the quality of the proposals received by Centres in response to their CFP. Centres can only fund research for which they receive proposals. Therefore, if researchers do not submit proposals to conduct research that aligns with the priorities of the federal funding partners, there is little, if anything, the Centres can do.

This, of course, is a contestable claim. Research funding agencies have many procedures to ensure funded research fits with the strategic priorities of research users. This is widely known, but not always approved of by academic researchers, among others. Good examples of research agencies that have developed procedures to help ensure that funded research is congruent with the priorities and needs of research users exist in the Canadian context. The Canadian Health Services Research Foundation (CHSRF) and some of the programs of the Canadian Institutes for Health Research (CIHR) are among the national leaders in this regard.

SSHRC also has research programs that do this. The Community-University Research Alliance (CURA) program provides one excellent example. The findings of the current evaluation confirm that Metropolis has a mixed record of matching the research done to research user priorities. It is equally clear, however, that at least some federal funding partners feel that within the context of Metropolis a new balance is desirable between researcher instigated and research user instigated research.

As has been seen, the Phase II MOU specified an equal balance between these two types of research. One Centre, as stated previously, reported it had shifted the balance entirely to the side of research that corresponded to federal funding partner priorities. Representatives of other Centres seemed to suggest that the balance was more in the direction of investigator driven research because of the Centres’ complete dependence on researchers submitting fundable proposals that may or may not match research user priorities. Achieving a general agreement on this among all Metropolis principles and stakeholders seem important for Phase III and beyond.

4.1.5 Knowledge Brokering – building and sustaining collaborations between researchers and research users

The importance of interpersonal relationships and networking opportunities as a determinant of effective knowledge transfer is well known. Given the nature of Metropolis it was expected that these types of relationships would develop as a way to facilitate the production of policy relevant research and the transfer of various research products to users. As has been seen, this has occurred, at least to some degree.

It is one thing to expect collaborative and cooperative working relationships to develop between researchers and policy-makers. It is another thing to make the establishment of these relationships a priority. In fact, without an explicit KT plan, the systematic establishment and maintenance of such relationships seems likely to remain underdeveloped.

As has been seen, establishing collaborative relationships between Metropolis researchers and federal research users has been left to motivated researchers and policy-makers to do themselves. The results of this informal ad hoc approach, although often good, are unreliable and unstable. A mutual lack of knowledge about the work environments and workplace priorities of collaborators, for example, often leads to wrong assumptions and misunderstandings about what will be produced and when it will be completed. This, in turn, can lead to strains in, and dissatisfaction with, the relationship.

A lack of time and institutionalized incentives to learn about each others’ needs, interests and patterns of work on the part of both researchers and research users further impedes individuals’ capacities to forge and to sustain effective collaborations. For example, the products of policy relevant research and knowledge transfer activities generally are not recognized or rewarded by universities for tenure and promotion decisions. Frequently the results of academic research are not immediately relevant to policy-makers because they address academic debates, not policy problems. This impedes the KT process and exacerbates the apparent inability of academic researchers and policy-makers to effectively co-ordinate their actions.

To bridge this gap, efforts have been made to develop positions and roles, the occupants of which are responsible for linking the individuals and the activities of researchers and research users. The title of this type of bridging position varies but among the most common are knowledge linkers, boundary spanners, and, the one in most common use in Canada at the present, knowledge brokers.

What are knowledge brokers and what do they do? Like every question in this area there is not a single answer. In fact, knowledge brokers can perform at least eight functions or roles [ Note 16 ]. Two of these relate more or less directly to knowledge application in the context of service delivery. Three relate to providing access to research products. Three others are directly concerned with linking together for knowledge exchange purposes researchers and research users. The issue of knowledge brokering is more fully discussed in the literature review commissioned as part of this evaluation.

Conclusion: In reference to the question – Did the Centres, Secretariat, and Federal Consortium act as effective knowledge brokers in the KT process? - knowledge brokering was done informally by the Centres, the Secretariat, and the departmental representatives of the federal funding partners. This is despite the fact that during Phase II of Metropolis there was no formally defined role for knowledge brokers in the knowledge transfer process. In fact, explicit reference to KT was not included in the Phase II MOU, although the intention to enable KT was clearly present. Based on the interviews and case studies, however, it is evident that all parties were generally unclear about whose role it was to be the knowledge brokers for Metropolis (i.e. the Centres, the Secretariat, or representatives from the federal funding partners).

Context: Recognizing the importance of the engagement of the producers (Metropolis Centres and Metropolis Secretariat) and the users (federal consortium members), it was not the purpose of this study to evaluate the funding consortium’s engagement throughout the knowledge production phase (planning of the research, methodology discussion, participation to discussions on preliminary findings, etc.). However, their level of engagement may have an impact on their use and perceived usefulness of the products.

Findings: The Phase II MOU does describe the roles and responsibilities of Domain Leaders, which includes such items as “serving as a point of contact between other Centres or stakeholders and researchers in the domain”, and “responding to inquiries for information from the federal partners and other stakeholders about the domain”. Likewise, certain elements of the roles and responsibilities of the Metropolis Secretariat and members of the Federal Consortium touch upon knowledge brokering.

In contrast, the Phase III MOU includes much more explicit references to knowledge brokering functions. For example, the Centre Directors are described as being “the principal liaisons between their Centres, the federal funders, other orders of government and local stakeholders”, and they are expected to “take a coordinating role in communications and knowledge transfer/mobilization at their Centre”. As well, federal partners are said to be “jointly responsible (with the Secretariat) for transferring/mobilizing knowledge within their organization”. Finally, Phase III also includes the creation of the Priority Leader position. The Priority Leaders are “tasked with ensuring knowledge transfer/mobilization to the federal partners on policy priorities”, which also includes “[making] the research results accessible to federal policy-makers in a manner most conducive to enhancing their utilization”. In Phase III, each federal partner produces a knowledge transfer plan, as do the Secretariat and the Priority Leaders for the priorities.

As determined through the case studies and interviews, none of the Centres nor the Secretariat, or any of the federal funders, had explicit knowledge transfer strategies in place during Phase II. In light of this, interviewees and survey respondents commented that government staff were not always aware of Metropolis research. Centre staff stated that they did not always know which policy-makers to target. Note that these were recurring comments that were not given in response to a particular question.

Despite the fact that there was no defined role for knowledge brokers in Phase II, survey respondents who had an opinion were somewhat positive about the effectiveness of knowledge brokering within Metropolis. Table 6 shows that 32% of survey respondents felt that the Metropolis Centres ‘often’ or ‘always’ operated as effective knowledge brokers, and another 15% reported they ‘sometimes’ operated as effective knowledge brokers. Twelve percent (12%) said ‘never’ or ‘rarely’; (it is important to note that a high percentage of respondents – 41% – did not know). Similarly, 37% of survey respondents felt that the Secretariat ‘often’ or ‘always’ operated as effective knowledge brokers, with another 9% saying ‘sometimes’. In this case, 12% said ‘never’ or ‘rarely, and 42% stated they did not know.

Table 6: Extent to which the Metropolis Centres and Secretariat operated as effective knowledge brokers in the knowledge transfer process

For both questions above, research staff (n=37) were generally more positive. Forty-six percent felt that the Metropolis Centres ‘often’ or ‘always’ operated as effective knowledge brokers, and 43% noted this about the Secretariat. Only 3% expressed that they believed the Secretariat ‘never or rarely’ operated as effective knowledge brokers, while this figure rose to 8% for the centres. This is noteworthy because throughout the evaluation it became evident that departmental research groups were those most often involved in the knowledge transfer processes, particularly processes to transmit research priorities to Metropolis Centres and to disseminate Metropolis research throughout their departments.

In addition, for both questions above, senior management (n=21) were slightly more polarized. That is, 43% of senior managers felt that the Metropolis centres ‘often’ or ‘always’ operated as effective knowledge brokers, and 24% said ‘never’ or ‘rarely’. Forty-eight percent (48%) of senior managers felt that the Secretariat ‘often’ or ‘always’ operated as an effective knowledge broker24% said ‘never’ or ‘rarely’.

Overall, the evaluation found that during Phase II, knowledge brokering was done informally by the Centres, the Secretariat, and the departmental representatives of the federal funding partners, rather than through formal processes. Based on the interviews and case studies, there is evidence that the success of knowledge brokering and the knowledge transfer process in general depended upon informal conversations, or on the relationships and networks that had developed between researchers and research users. It also depended, to a great extent, on the level of engagement of federal funding departments, which varied and changed over time.

Federal funders and case study participants both noted, however, a number of improvements in Phase III:

• More concrete and rigorous mechanisms / more formality in the establishment of research direction and policy questions;
• The creation of the priority leader positions; and
• Greater communication and opportunities to allow the partners to provide more feedback/input on research plans; more direct involvement between government and academics.

Very few additional suggestions were made for improvements to the process, although the literature provides considerable guidance on a variety of knowledge brokering functions that may be relevant to Metropolis.

4.1.6 Do federal funding partners access and use research products for policy making?

Conclusion: There is evidence that federal policy-makers are accessing Metropolis research.

Findings: Results of the survey of federal employees, who were selected because they were considered to be potential users of Metropolis research, indicate that federal policy-makers do access Metropolis research.

Different Metropolis activities, such as national or international conferences, centre retreats or workshops, Metropolis Conversations or Metropolis Presents, as well as different research products, which include, for example, research reports or publications (including working papers), policy syntheses, magazines, journals or websites, were identified. Those two categories are referred as Metropolis outputs in the following section.

When asked specifically about the series of Metropolis activities and research products listed as examples above, 75% out of 273 respondents reported having used at least one Metropolis output, 48% participated in at least one activity and 67% used at least one of the research products.

This percentage was higher when limiting respondents to those who work in policy development and/or advice or in research. Out of a total of 79 respondents working in policy development and/or advice, 87% used at least one Metropolis output (66% of them participated in at least one Metropolis activity and 82% used at least one Metropolis research product). Of the 66 respondents who worked in research, 83% used at least one Metropolis output (53% participated in at least one activity and 77% used at least one research product). In opposition, respondents performing other types of work, such as program design and delivery (69% used at least one Metropolis output, 40% at least one activity and 59% at least one research product, out of a total of 68 respondents) reported lower participation/use of Metropolis products.

It was higher for those respondents who work in the NCR, as 71% used at least one Metropolis output (51% participated in at least one activity and 70% used at least one research product, out of a total of 182 respondents), as opposed to 71% for other regions (42% participated in at least one activity and 62% used at least one research product, out of a total of 91 respondents).

4.1.7 Do federal funding partners support access to, and use of, research products for policy making?

Effective transfer and use of research knowledge, of course, requires supportive cultures and processes in the user organizations.

Conclusion: Notwithstanding the results presented in the preceding section, the evaluation found that access to and use of Metropolis research may be limited by potential user capacity and institutionalized support.

Findings: In general, survey respondents do not spend a great deal of time reading or reviewing research materials or attending research dissemination activities. In fact, 72% of the 185 survey respondents who provided a valid answer spent 10 or fewer hours per month on these types of activities. It is important to note, however, that those respondents whose work involved research or policy development and/or advice generally spent more time on these types of activities, whereas those whose work involved program design generally spent less time (see Figure 2).

Figure 2: Time spent per month by survey respondents reading research products or attending research dissemination activities relating to immigration, settlement and or diversity

Of the time that was spent reading or reviewing research products and materials or attending research dissemination activities related to immigration, settlement or diversity, survey respondents spent a relatively high proportion of their time on materials and activities produced by Metropolis. On average for the 167 respondents who answered this question, 46% of the time survey respondents spent reading or reviewing research materials or attending research dissemination activities was spent on materials and activities produced by Metropolis (the median was 40% of time). In fact, 17% of survey respondents reported that 100% of the time they spent reading or reviewing research materials or attending research dissemination activities was devoted to Metropolis products and activities, and another 15% of survey respondents answered 50% of the time.

Among the 276 valid responses, close to a third (30%) of survey respondents, however, felt that the overall amount of time they spent reading research materials or attending research dissemination events was not adequate for their job, and another 20% did not know. When asked why they did not spend more time on these activities, the majority (68%) of the 137 respondents who provided an answer reported that it was because they did not have the time (it is important to note that this response was not particularly attributable to one department or type of work). In addition, another 23% of respondents indicated that they did not spend more time on these activities because the use of research and participation in research activities was not a priority for their manager or department (again, this response was not particularly attributable to one department or type of work).

While the representatives of federal funding departments who were interviewed generally indicated that their department did in fact view access to and use of research in policy-making as a priority (this was stated by 11 of the 13 interviewees, while the remaining two interviewees answered ‘somewhat’), only a little over half (7 of 13 interviewees) said that their department allocated time for identifying, reading and incorporating research into policy-making efforts. Three of the remaining interviews (which, incidentally, all represented different federal departments) felt that their department did not allocate time for these activities, and a fourth interviewee said ‘not formally’. The two remaining interviewees were unsure or did not respond to this question.

4.2 Policy maker satisfaction with Metropolis research products & outputs

In this section attention is turned from issues associated with research user participation in Metropolis processes to the question of the quality and utility of Metropolis research products to federal government policy makers. We report on the perceived relevance of Metropolis research products to federal government policy makers (see Question 5, section 2.1). Following this, the extent to which Metropolis researchers are perceived by research users to have successfully responded to the identified priority for longitudinal, comparative and pan-Canadian research is discussed (see Question 6, section 2.1).

4.2.1 Is Metropolis producing research products that are relevant to government policy makers?

Conclusion: Survey respondents indicated variability in their responses to relevance issues. Respondents reported that:

• they used Metropolis research products (up to 61% depending on the product), or participated in Metropolis research activities (up to 43% depending on the activity);
• they found these outputs useful (between 53% and 90% depending on the output);
• they were satisfied with the timeliness (58%) and quality (70%) of the products; and
• they were satisfied with the relevance of Metropolis products (70%).

As well, most interviewees and case study participants indicated that Metropolis research was relevant to government policy makers. However, views were mixed when participants were asked to compare the relevance of Metropolis products to that of research produced by other sources, such as Statistics Canada, academic researchers, or others within their own department.

Findings: The proportions of all 275 survey respondents who indicated they used, or participated in, Metropolis outputs is presented in Table 7. The most used products were:

• Research reports or publications (61%)
• Websites (58%)
• Policy syntheses (46%)
• Our Diverse Cities magazine (46%)
• Canadian Diversity magazine (45%)
• Conferences (43%)

Use was greatest for ‘researchers’ or those involved in ‘policy development and/or advice’. For these groups (n=145), reports and publications were used most and retreats were used the least. It should be noted though that participation in retreats was by invitation only, therefore it is not surprising that a large proportion of those surveyed had not participated in them.

Table 7: Use of Metropolis outputs

Generally, survey respondents who used Metropolis products/outputs found them to be useful. The majority rated all outputs as either ‘very useful’ or ‘quite useful’. The most useful outputs were reports and publications with 90% of those who reported using them finding them to be useful. Retreats were reported to be the least useful, 47% of respondents reported finding them not useful (See Table 8).

Table 8: Usefulness of Metropolis outputs*

* Not useful figures include ‘not at all useful’ and ‘not very useful’ responses. Useful includes ‘quite useful’ and ‘very useful’.

The majority of survey respondents were satisfied with the timeliness (58%) and overall quality (70%) of Metropolis products [ Note 17 ]. Two percent of survey respondents reported being dissatisfied with the timeliness of Metropolis products, 21% were neutral and 19% didn’t know. Five percent stated they were dissatisfied with the overall quality of the products, 12% were neutral and 13% didn’t know. HRSDC respondents were slightly more satisfied than other respondents. Seventy-one percent of HRSDC respondents were satisfied with timeliness, and 74% were satisfied with overall quality. The same trend was noted for respondents whose work involved research or policy development/advice. Sixty-eight percent of this population were satisfied with timeliness, and 79% were satisfied with overall quality (See Table 9).

Table 9: Satisfaction with timeliness and overall quality of Metropolis products*

* ‘Satisfied’ figures include ‘satisfied’ and ‘very satisfied’ responses. ‘Not satisfied’ includes ‘dissatisfied’ and ‘very dissatisfied’.

Interviewees, survey respondents and case study participants indicated that during Phase II Metropolis research was relevant to government policy makers. Stakeholders generally agreed that Metropolis research was relevant to policy making, largely because it contributed to a growing body of research knowledge that policy makers can refer to and draw upon. This is consistent with the KT literature that indicates that single studies are generally not a reliable foundation upon which to base policy. Rather, a substantial body of research knowledge is a more valid foundation upon which to build policy.

Of the 207 survey respondents who provided an opinion, 70% indicated they were ‘satisfied’ or ‘very satisfied’ with the relevance of Metropolis products (14% were neutral, and 5% were ‘dissatisfied’ or ‘very dissatisfied’). Those who worked in research (n=53), and those located in the NCR (n=141), were more satisfied with 77% of those who worked in research and 73% of those located in the NCR indicating they were ‘satisfied’ or ‘very satisfied’ (see Table 10).

Table 10: Satisfaction with relevance of Metropolis research*

* ‘Satisfied’ figures include ‘satisfied’ and ‘very satisfied’ responses. ‘Not satisfied’ includes ‘dissatisfied’ and very dissatisfied’.

Federal funding partners who were interviewed also agreed that Metropolis research products were relevant to government (10/13). One person indicated Metropolis products were not relevant to government. When asked what they were using Metropolis research for, case study users most often indicated they used the research to build or enrich their knowledge base.

However, more than 1/3 of interviewees (5/13) and case study users (3/8) expressed concerns about the policy link of Metropolis research. Some of these stakeholders suggested that the policy link of Metropolis research should be more clearly identified in dissemination materials. A weak link to policy priorities was the most common reason cited (5/21) for Metropolis research being less relevant than research from other sources.

Funding partners’ views were mixed when asked to compare the relevance of Metropolis products to that of research produced by other sources. All federal funding partners and case study users interviewed reported that policy development within their departments was informed by research from other sources (e.g., Statistics Canada, academic researchers [ Note 18 ], others within their own department). When asked to compare the relevance of Metropolis research products to those from other sources, views were mixed. Four of 13 interviewees stated Metropolis research was ‘about the same’ as research from other sources; four said it is ‘worse’, three were unable to say, one said ‘better’ and one said ‘it depends’.

Interviewees noted that researchers and policy makers operate in very different realms. According to interviewees, researchers want to look at very specific questions, and they do not really want government intervention. These interviewees, however, also noted that researchers do not necessarily understand policy making and what is needed.

4.2.2 Have the Metropolis Centres conducted macro (incl. pan-Canadian), comparative and longitudinal studies to support policy development?

During Phase II federal funding partners identified macro (including pan-Canadian), comparative and longitudinal policy relevant research as a specific priority. In this section we look at whether Metropolis conducted these types of studies to support policy development.

Conclusion: Centres did conduct macro, comparative and longitudinal studies despite the fact that, according to the Centres, increased funding for Phase II, which was (in part) to allow for these types of studies, did not fully materialize. Case studies interviewees and focus group participants noted some potential barriers to these types of studies. However, the majority of survey respondents indicated they were satisfied with the extent to which the Centres conducted these types of studies.

Context: The previous evaluation of Metropolis conducted in 2000 recognized the need “for more projects of a macro, comparative, pan-canadian nature as well as for longitudinal studies to assist federal policy development.” [ Note 19 ] The Metropolis RMAF produced in July 2005, includes a list of issues and questions that were identified by program stakeholders as the highest priorities for the summative evaluation of the Metropolis project. [ Note 20 ] Included in the RMAF is the question “To what extent have the Metropolis Centres conducted macro, comparative, and longitudinal studies to strategically feed policy development?” As well, the mid-term review of Metropolis, produced by SSHRC in April 2006, included the recommendation that “the centres should strengthen comparative research and transnational research ties and collaboration.” [ Note 21 ]

Funding to the Metropolis Project was to increase by 25% for Phase II, amongst other things, to allow for increased studies of these types. However, this funding increase did not fully materialize. The overall funding for Metropolis decreased by approximately 4% down from $8,230,900 to $7,881,059 from Phase I to Phase II. It is difficult to compare the total investments for both phases since Phase I lasted 6 years, one year longer than Phase II, which was 5 years. Also, some federal partners left the Metropolis Project between the two phases which resulted in loss of funding.

Increased funding was received from some partner organizations, but not all. As well, two organizations that provided funding during Phase I, withdrew from Metropolis for Phase II. As a result, the 25% increase to funding was not achieved. This increase was required to make many sections of the Phase II memorandum of understanding operative (including the increase in pan-Canadian, comparative and longitudinal studies).

Findings: Centres did conduct macro, comparative and longitudinal studies, despite the reported lack of increased funding. Interviewees for all Centres indicated they produced macro, comparative and longitudinal research studies - particularly in the economics domains - during Phase II. Only two Centres (IM and RIIM), however, provided quantitative information relating to these types of studies. In total, these two Centres produced 114 Metropolis-funded research studies in the form of Capsules de Recherches at IM, and Working Papers’ at RIIM. [ Note 22 ] Of these studies, 34% represented pan-Canadian research, 29% were comparative studies, and 11% were longitudinal. Note that these categories are not mutually exclusive and some studies were, for example, both comparative and longitudinal.

On the other side of the equation, survey recipients were asked to indicate how much of the Metropolis research that they used was based on macro (including pan-Canadian), comparative or longitudinal studies. Of the 272 survey respondents who provided a valid answer to this question, just under half (47%) indicated that ‘some’ of the Metropolis research they used was based on studies of these types. Almost another quarter of respondents said ‘most’ (21%) or ‘all’ (2%). Ten percent indicated ‘none’. The remainder (20%) said they didn’t know.

Case study interviewees and focus group participants noted some potential barriers to conducting these types of studies. One of these barriers is a lack of funding for large studies. Macro, comparative and longitudinal studies tend to be quite expensive. Typically, however, the level of funding provided to researchers through Metropolis is relatively small, and generally not sufficient for these types of studies. Structural barriers also were identified by Metropolis stakeholders. For example, Metropolis funding is generally available to researchers for one year only. Longitudinal studies take longer than one year, unless existing data sets are used. As well, the Metropolis project is administered through Centres that tend to prefer to fund their ‘own’ researchers, rather than researchers from across the country, which could be necessary for pan-Canadian studies. Finally, Metropolis Centres can only fund research for which they have received proposals. During Phase II, there were no financial or other types of incentives in place to encourage researchers to propose and undertake these larger, more complex, types of studies.

Just over half of survey respondents indicated they were satisfied with the extent to which the Centres conducted these types of studies. Survey respondents indicated that these types of studies are ‘quite important’ or ‘very important’ for their work. Comparative studies were slightly more important (79% indicated “quite important” or “very important”) than pan-Canadian (73%) and longitudinal (74%). Only 10% of survey respondents indicated comparative studies were ‘not very important’ or ‘not at all important’. Fourteen percent indicated this in relation to pan-Canadian studies, and 12% for longitudinal studies [ Note 23 ]. Those who work in research or in policy development/advice, and those located in the NCR, were more likely than those in other groups to view these studies as ‘quite important’ or ‘very important’.

Table 11: Importance of pan-Canadian, comparative and longitudinal studies*

* Important includes those who indicated the type of research was ‘quite important’ or ‘very important’. ‘Not important includes those who indicated the type of research was ‘not at all important’ or ‘not very important’. Percentages do not add to 100% because some respondents answered ‘don’t know’.

Of the 206 survey respondents who had an opinion, just over half (53%) were satisfied with the extent to which the Centres conducted these types of studies. Six percent were dissatisfied, 22% were neutral, and 19% did not know. Those located in the NCR were somewhat more satisfied (59.4% vs. 41.2% for those in the regions).

4.3 Impact of Metropolis Research on Policy Making

The ultimate purpose for this summative review of Metropolis is to help advance understanding of how and to what extent its research processes and products had a positive impact on policy making and program delivery during Phase II and to use this knowledge as a basis to enhance the policy impact of Metropolis research in Phase III.

Government policies and programs are among the main ways that research knowledge is translated into beneficial social, economic, environmental and or cultural outcomes. Ensuring the timely transfer of relevant research knowledge to the right people in the best format to inform or influence government policy development is the goal of KT in general and, inter alia, of Metropolis in particular. Thus, in this section, responses to the question - Has knowledge transferred from Metropolis informed and/or influenced government policy development? – are reported (see Question 7, section 2.1).

Conclusion: There is evidence that research knowledge transferred from Metropolis has informed government policy development to some extent. The degree to which it has influenced policy development is less certain.

Findings: There is evidence that research knowledge transferred from Metropolis to some extent has informed government policy development. For example, 43% of the 278 survey respondents indicated that they had used Metropolis research to inform policy discussions, whereas 57% had not. However, the group that is most likely to use Metropolis products to inform policy discussions is the one whose work involves policy development and/or advice. When limiting the analysis to the 79 respondents in this group, the percentage of those who have used Metropolis research to inform policy discussions increases to 79%.

For their part, representatives of the federal funding partners and other Metropolis research users who were interviewed as part of the case studies were split when asked whether they believe Metropolis research had informed policy-making, with approximately half saying that it had, and the other half saying that it had not.

While a direct link between research and policy was not always easy to identify, some stakeholders provided examples of Metropolis research informing policy development. One interviewee noted that “it is not one piece of research that changes the world, but if all the research points to the same conclusions, then you have major impact”. Despite difficulties in attributing research impact on specific policy outcomes, a number of examples were identified. These examples, drawn primarily from the interviews, user focus groups and case studies, include (but were not limited to):

• The creation of the foreign credential referral office at CIC (Metropolis research made government aware of the need, and what evidence-informed public policy).
• Metropolis research helped PCH achieve departmental approval of $56M for the Action Plan Against Racism
• CMHC’s Affordable Housing Initiative was informed by Metropolis research. This has generated policy ideas and has led to programs.
• At HRSDC, Metropolis research has informed Employment Equity policy (HRSDC was considering modifying the Employment Equity Act, and based on consultations and research done by Metropolis, they decided not to change it).

When asked which types of Metropolis research products and activities had been most useful in supporting policy and program development, federal funder interviewees and case study research users identified a broad range of products. Those that were mentioned most frequently included conferences and symposia (identified by 8 of the 21 interviewees), workshops (identified by 6 of the 21 interviewees), magazines (identified by 5 of the 21 interviewees), websites (identified by 4 of the 21 interviewees), Metropolis Conversations (identified by 3 of the 21 interviewees), working papers (identified by 3 of the 21 interviewees), ‘research capsules’ (identified by 2 of the 21 interviewees), research papers and/or reports (identified by 2 of the 21 interviewees), and journal publications (identified by 2 of the 21 interviewees). Survey respondents reported somewhat different assessments of the usefulness of the research products (see Table 8).

Although knowledge transferred from Metropolis appears to have informed government policy development, the degree to which it directly influenced policy development is less certain. It was often quite difficult for interviewees and case study participants to identify specific policy documents that had been influenced by Metropolis research. However, a number of interviewees and case study participants did attempt to explain why such specific policy documents could not be easily identified. The main reasons that were provided were that government policy papers generally do not cite sources of information, and that there is a limit to how much influence one research product can have on policy (rather, policy is informed by many sources, over many years).

___________

11. Evaluation of the Metropolis Joint Initiative Program, Fall 2000, CIC and SSHRC, pg 9.

12. For the list of the 11 policy priorities, see Appendix G.

13. Memorandum of Understanding between the Social Sciences and Humanities Research Council (SSHRC) and the Department of Citizenship and Immigration (CIC), Annexes, pp. 9-24, Annex H.

14. In analyzing survey findings, ‘not applicable’ responses were removed from the analysis prior to calculating percentages for each question. Individuals who answered ‘not applicable’ could have done so for a variety of reasons, including not being familiar enough with Metropolis or the specific aspect being questioned. ‘Don’t know’ responses were retained and included in the analysis. For this particular question (i.e., how satisfied were survey respondents with the processes for identifying and communicating policy information needs and priorities to Metropolis), 273 individuals responded to the survey question, however 112 chose “not applicable”. Percentages are therefore based on n=161.

15. The report on the mid-term review by SSHRC did include some information on policy-relevance of Centres’ research, although it did not specifically examine the link between research conducted by the Centres and the priority areas identified by funding partners.

16. R. G. Havelock, 1986. “Linkage: Key to Understanding the Knowledge System.” In Knowledge Generation, Exchange and Utilization, edited by G.M. Beal, W. Dissanayake and S. Konoshima, 211-43. Boulder, CO: Westview Press.

17. There were 178 valid responses given to the question on timeliness and 202 on overall quality).

18. It is possible that academic research referenced by interviewees was also research produced by Metropolis researchers, for example though academic publications that did not reference the Metropolis Project.

19. Evaluation of the Metropolis Joint Initiative Program, Fall 2000, CIC and SSHRC, pg 9.

20. Results-Based Management and Accountability Framework (RMAF), Version 4: July 26, 2005, SSHRC, pp. 25-26.

21. Some longitudinal, comparative or pan-Canadian research conducted by IM may result in products other than Research Capsules, however, quantitative information was provided in the activity report for Research Capsules only.

22. Some longitudinal, comparative or pan-Canadian research conducted by IM may result in products other than Research Capsules, however, quantitative information was provided in the activity report for Research Capsules only.

23. There were respectively 219, 220 and 217 valid responses that were given to the question on the perceived importance of comparative, pan-canadian and longitudinal studies.

• Previous
• 1
• 2
• 3
• 4
• 5
• 6
• 7 (current)
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• Next