Formative Evaluation of Canada’s Citizenship Week

1.0  Background

Study Methodology
Goals of Canada’s Citizenship Week
CCW Budget
Citizenship Week
Target Audiences
Programming Processes
Intended Results

Canada’s Citizenship Week (CCW) was created as an opportunity to promote the intrinsic value of Canadian citizenship, to support a sense of belonging among all Canadians, and to recognize the contribution that immigration makes to Canada’s growth as a nation.

From 1993 to 1999, Citizenship and Immigration Canada (CIC) partnered with the Department of Canadian Heritage and the Heritage Foundation to deliver Citizenship and Heritage Week in February. That week highlighted Canadian heritage, citizenship and Flag Day themes. In 2000, in order to focus attention on citizenship and immigration themes, CIC assumed sole responsibility for the week. The week was renamed Canada’s Citizenship Week (CCW), and it was moved from February to October, in an effort to increase participation among schools and community groups and to provide a warmer temperature in which to celebrate.

To date, CIC has organized three Canada’s Citizenship Weeks:

October 16–22, 2000 [note 1]

October 15–21, 2001 [note 2]

October 14–20, 2002

The theme of the annual event is Canada, we all belong! The week is intended to encourage all Canadians to reflect on the value of citizenship, what it means to be Canadian, and the rights, privileges and responsibilities of citizenship.

Study Methodology

There are two main focuses of this CCW evaluation. One is the analysis of the program design; the other is the evaluation of the activity guide developed annually for teachers and youth leaders.

There are three sources of information used for this evaluation:

1. Interviews with guide users: school teachers and youth leaders in the greater Ottawa region and the greater Gatineau region who had requested material were interviewed.
2. Interviews with informed officials: public servants associated with CCW policy, implementation planning, communications, distribution or operational use of material were interviewed.
3. Review of documentation: key documents, including well-organized records of previous years maintained by Integration Promotion (IP) were examined (see Appendix A).

Goals of Canada’s Citizenship Week

The stated goals of the week are to:

• increase awareness of the values, rights and responsibilities associated with Canadian citizenship;
• increase awareness of the value of Canadian citizenship among youth and newcomers;
• promote a sense of belonging through the programs We all Belong and Welcome Home, which are managed by IP and carried out by the regions; and
• promote active citizenship to all Canadians.

CCW Budget

IP has a total Operations and Management budget of $1,100,000, which is divided into the following four main functional areas.

$600,000: Integration publications (Welcome to Canada, Newcomer’s Introduction, A Look at Canada, other fact sheets)

$120,000: Promotional material and events (i.e., the purchase and printing of supporting material such as Canadian flags, pins, kit folders and reaffirmation kits)

$80,000: Transfers to regions in support of the national program (Atlantic – $10,000; Quebec – $15,000; Ontario – $25,000; Prairies – $15,000; Pacific – $15,000)

$300,000: Canada’s Citizenship Week.

$1,100,000: Total

The budgeting for CCW provides the following:

$113,500: Activity guide ($24,000 for development, $7,500 for design and layout, $12,000 for translation and $70,000 for printing)

$33,500: Poster printing (design cost is covered through the activity guide)

$7,500: Order form printing

$53,000: Distribution ($5,000 for mail-out to municipalities, $45,000 for staff in CIC distribution, $3,000 for insertion in Scout magazine)

$45,000: Other related products (Welcome Home poster, $5,000; Times of My Life, $40,000)

$47,500: Other requirements such as the advance preparation for next year’s material, the reprinting of material or the purchase of additional items such as flags and pins

$300,000: Total

Citizenship Week

The core of programming activities for the week is provided through the IP unit of the Integration Branch and CIC regional staff across the country. The Promotions Division and the Strategic Communications Directorate (SC) work with regional program and communications staff to promote and communicate CCW to key target audiences.

The week is designed to provide occasions at which to highlight the value of citizenship and immigration and to focus on the privileges, rights, responsibilities and obligations of citizenship. Although CCW falls in October, the IP message is that citizenship is a concept that deserves to be celebrated throughout the year.

Citizenship Week is intended to encourage all Canadians to reflect on the value of citizenship, what it means to be Canadian, and the rights, privileges and responsibilities of citizenship. There are four key program components. Each is described below.

Activity guide for teachers and youth leaders;

Citizenship ceremonies at which newcomers to Canada take the Oath of Citizenship and become Canadian citizens;

Reaffirmation ceremonies at which Canadians reaffirm their citizenship by reciting the Oath of Citizenship; and

Citation for Citizenship presentation ceremonies [note 3].

Activity Guide for Teachers and Youth Leaders

The program has developed a year-round activity kit targeted to primary, middle school and early secondary school teachers and youth leaders. This cross-curricular kit includes an activity guide, an activity poster and a sample of Times of My Life. The guide contains activities designed to help youth know themselves better and to connect to each other, their community and their country.

The activity guide is sent to school teachers who are on a distribution list or who have sent in an order form. It is also sent to various Girl Guide and Scout chapters.

Although the guide is officially available for school children in grades 5 through 13, it is targeted to students in grades 5 and 6.

The contents of the three activity guides produced to date are displayed in Appendix B. A comparison of the three guides reveals that the guide has changed significantly from year to year. Our interviews have revealed that activities from previous years are favourably remembered. The present approach to producing a new guide each year leaves behind previously successful activities and presents new activities each year. The activity guide may be viewed at www.cic.gc.ca/english/celebrate/index.asp, which is the home page for the Welcome Home activities and which links to the activity guide of each year.

Citizenship Ceremonies

Citizenship ceremonies mark the occasion of a newcomer to Canada officially becoming a Canadian citizen. The Oath of Citizenship is part of the ceremony. These ceremonies, held throughout the year, are officiated by citizenship judges. Some of the ceremonies are given higher profile than others by being held during CCW or on Canada Day, or by involving high-profile individuals who may attract media attention. CIC supports these ceremonies whether they are held in a Citizenship Court or at another location. In the latter cases (e.g., classes at a high school, a Kiwanis Club, a Scout troop), there is often a host involved and the host’s participation will also be supported by CIC personnel.

Reaffirmation Ceremonies

The core of a Reaffirmation Ceremony is the repeating of the Oath of Citizenship. These are typically group events to celebrate the prior acquisition of Canadian citizenship. CIC assists with these ceremonies through the provision of a sample program, reaffirmation certificates and detailed suggestions for hosting a ceremony.

Citation for Citizenship

These awards recognize those who help newcomers integrate into Canadian society. Initiated in 1987, the annual Citation honours a maximum of 20 Canadian individuals and organizations from the voluntary, private or public sectors. Individuals must have contributed to the integration of permanent residents or convention refugees in an exceptional way through activities that have had significant impact at the local, regional or national level [note 4].

Target Audiences

Even though Citizenship Week is aimed primarily at young children, the intention is to influence a range of different audiences. Target audiences (see Appendix C) for the communication tools for CCW include:

• youth (with an emphasis on school children in grades 5 and 6);
• newcomers to Canada;
• federal public servants;
• the media, especially community, ethnic papers; and
• all Canadians.

Programming Processes

Exhibit 1 (see below) presents the major processes that are in place within CIC. The two principal players are IP and SC. This process works well according to all who have expressed a view. However, SC is of the opinion that the objectives of the program would be better achieved if there was a more strategic approach to obtaining media coverage. One way to attract media attention is by having high-profile public figures in attendance at events such as selected Citizenship ceremonies.

The view of SC is that this intention would be enhanced if it were involved at the early stages when the social marketing approach is being developed and very much in advance of any events. SC suggests that a long time line is required for planning and that high-profile participants can be attracted if they are given the right message and if this is done before their agendas are too crowded to permit attendance.

Partnerships are created to leverage the efforts of IP. A significant example that has evolved over time is the partnership with the Royal Canadian Mint [note 5]. The first use of a Citizenship Coin was in 1999. The initiative was well-received and has continued every year since. Although CIC was not formally involved in the initial years of the Canada Day Coin Design Contest, the winner of the 2001 contest was announced at a Citizenship Ceremony. The reaction was so positive that CIC and the Royal Canadian Mint entered into a formal agreement to introduce the Canada Day coin in the 2002 CCW activity guide. Both partners are of the opinion that this is a symbiotic arrangement that serves their respective objectives and adds synergy to their efforts.

Other categories of partnerships are less closely linked but nevertheless potentially synergistic. These consist of paid advertisements, civic proclamations and appeals to enlist the support of political leaders and public servants. IP purchases advertising space or inserts in magazines such as The Leader Magazine, directed at Scout leaders, and obtains an appropriate audience for its CCW material. Municipalities are encouraged to proclaim the designated week as Canada’s Citizenship Week in their city and, in 2001, some 39 cities across Canada made such a proclamation. Notices are sent to Senators, Members of Parliament, constituency offices, provincial offices, Deputy Ministers and Assistant Deputy Ministers of other federal government departments, and representatives on the Interdepartmental Committee on Citizenship Initiatives to give extensive coverage to the existence of CCW and to encourage attendance at a Citizenship ceremony.

Although the process map (see Exhibit 1) shows a linking of planned activities to support CCW, it does not provide an evaluation of what actually happens. It does describe what is planned to happen. The planned process is judged to be complete and holds the promise of achieving the objectives related to organization and the dissemination of material. The reality is that the preparation and distribution of material appears to be achieved but that the performance measurement is deficient. Basic “uptake” statistics comparing target markets with orders and the actual use of material are not available. The performance review that does take place is informed by more casual feedback and the experiences of those involved in the processes.

A comparison of the mechanisms used for getting out the message and the target audiences to be influenced shows that the program depends upon very strong ripple effects to achieve its objectives. For example, school interventions for students in grades 5 and 6 are expected to impact the students, their families and the students in later life. Participants at ceremonies are expected to be influenced by the explicit positive value of Canadian citizenship, and this influence is expected to be transferred to the general population through media coverage. There is no empirical evidence at present that the exposure of the few to CCW events has a discernible influence upon the attitudes of the many in the general population.

Intended Results

The objectives of CCW are supported throughout the year by a number of activities carried out by the regions. These include the federal government’s Welcome Home Campaign, the We All Belong program and Canada’s Citizenship Tree. The latter is a hand-sculpted tree honouring and welcoming new Canadian citizens. The Welcome Home Campaign invites children to create messages of welcome for the thousands of newcomers who will take their Oath of Canadian Citizenship each year.

Exhibit 2 (see below) shows the CCW activities engaged in by IP and SC. These are designed to produce outputs (i.e., activity guides, school visits, speech modules, news releases and ceremonies) from which additional results are expected to flow. The activities are in three “layers” to show the expected sequential construction of results. The initial outcomes (i.e., participation by children, media messages disseminated and heard, and public servants better prepared) result from the outputs and are expected to produce the later outcomes (i.e., awareness of the rights, obligations and value of Canadian Citizenship; newcomer’s sense of belonging; and the practice of good citizenship). These outcomes would contribute to three key societal-level attitudes: the intrinsic value of Canadian citizenship, a sense of belonging and the contribution of immigrants.

The later outcomes and the expected societal changes are goals towards which the CCW works, but it must be recognized that achieving these results is not under the control of managers. On the one hand, the program budget of $300,000 is not a very large amount with which to influence more than 30 million Canadians, nor would it be even if it were doubled or tripled by creative managers finding additional funding sources. On the other hand, there is a wide range of other influences acting upon the same intended results such that, even if very positive results were demonstrated, it would be difficult to estimate the degree to which results could be attributed to CCW.

Some examples of other influences are:

Literature in the form of books, newspapers and magazines, non-print media messages, and radio and television, all of which convey messages with the same themes as CCW. A specific case is that of the recent volume Passages to Canada referenced in Footnote 5.

Another source of complementary messages is the programs of other government departments. At the federal level, these include the peace support programs of the Department of Foreign Affairs and International Trade and the Heritage Day programming at Canadian Heritage.

The reality is that the activities and messages of much of Canadian society support these messages of CCW. Trying to describe those influences is a little like asking a fish to describe the water in which it swims or Canadians to describe the culture into which they have been inculcated.

Results Measurement

Identifying the intended results for CCW (see Exhibit 2) makes it possible to consider whether it is feasible to evaluate the program and, if so, how best to do that in a cost-effective manner. Toward that end, Appendix D has been developed. In Appendix D is found a listing of the full set of intended results and an initial consideration of the types of measurement required and how such measurement would be used in an evaluation study. It must be noted, as part of any consideration of measurement, that demonstrating positive findings in results areas is a necessary but not sufficient condition for the issue of program attribution. In addition to measurement considerations, an evaluation must use a methodology that allows the impact of a program to be estimated. In this case, such methodology would establish a causal link between CCW and the later outcomes and societal changes identified in Exhibit 2. That would require methodology that contrasts the attitudes of children exposed to CCW activities with others not exposed. Moreover, it would have to do that in a way that sorts out the impacts from the multiple sources of messaging focused on the same objectives as those of CCW.

In this section, we consider only the measurement of results.

There are four main categories of measurement identified in Appendix D. They are listed here with an assessment of their utility and availability.

Uptake Statistics—Uptake refers to the extent to which the material provided by IP is requested by the target population, i.e., what portion of the intended users actually request and receive material? Uptake is a basic requirement for an evaluation of this type of program. There are two main types of uptake measures needed for CCW:

• Use of communications products. What proportion of print and electronic media that receive news releases and story ideas publish the release, a story based on the release, or any story or article on the topic of CCW or citizenship?
• Use of didactic material by teachers and youth leaders. Of the populations given notice that material is available, how many request each of the available guides, pins, flags, etc.?

Situation Report—Uptake statistics are essential for program monitoring. Neither type of uptake measure is gathered on an ongoing basis. It is feasible, at a relatively low cost, to put in place data collection systems that will provide the information needed for performance monitoring.

Public Opinion Polls—Public opinion polls refer to the asking of questions of representative samples of a designated population. The Later Outcomes and the Societal Outcomes of CCW require measuring both attitudes (e.g., belonging) and knowledge (e.g., immigration’s contribution, value of citizenship, obligations of the citizen) of the Canadian public or of designated portions of this population. Given the cost of full population measurement of attitudes and knowledge, a search for existing data should be made before any new measurement is considered. It is most unlikely that costly new measurement would be warranted by a program the size of CCW [note 6].

Situation Report—Public opinion polling data is available for a number of relevant questions. These have not been systematically examined and, therefore, it is not known which questions were asked in which year, whether the same questions have been asked over time and whether there is a chronological record that is interpretable. If available, this data will be most useful.

User Reviews—User review refers to the critical analysis provided by persons who have used the material produced by IP. Typically, this information will be collected by some form of survey instrument delivered by interview, paper, internet or telephone. These approaches provide:

• comments spontaneously offered by the user when they choose to answer questions published as a “Please Comment” insert to a booklet or a Web page;
• comments provided by the user in response to an interviewer, either in person or by telephone;
• answers, including comments, offered to a written set of questions, presented either in hard copy form or electronically; and
• comments offered during a focus group session. Focus groups (which may be virtual or physical) with provincial curriculum representatives would usefully bring material for teachers more in line with school curriculum goals Focus groups with teachers and youth leaders could explore in depth both the content and messages being delivered and the form in which this content is best delivered.

Situation Report—User review is a valuable source of information. It can measure knowledge and attitudes, and can obtain suggestions for improvement. Such reviews are not currently available from any designated large population base.

Opinions of Program Officers—CIC regional staff across the country are an integral part of the support for CCW activities. Their opinions, based on experience delivering ceremonies and material, are an important source of feedback on what works, as well as suggestions for increasing efficiency and effectiveness.

Situation Report—The opinion of regional staff is not systematically collected. It would be desirable to do so using a standardized data collection form. The information received should be reviewed and a dialogue should be continued, as appropriate, with regional staff. The views expressed each year could be collected to allow a time-line analysis by which to monitor program delivery changes.

_______________

[1] Following the February 1999 session, there was a 20-month interval until the next session. The unusual interval occurred as a result of the shift in calendar placement from February to October and the decision not to respond for October 1999, eight months after the last Citizenship and Heritage Week. In 2000, Citizenship Week was held against a backdrop of uncertainty caused by the introduction of a new Citizenship Act in Parliament and the anticipated timing of a federal election.

[2] In 2001, Citizenship Week was given more emphasis due to the September 11 terrorist attacks on the U.S.A.

[3] Not offered in October 2002.

[4] Citations were not awarded in 2002.

[5] Other active partners of IP also exist but are not integrated into the CCW effort. For example, The Dominion Institute, in 2001, with CIC as one of its sponsors, issued Passages to Canada, a sourcebook for the themes of immigration, citizenship and belonging. The use of the same themes as CCW and the focus on the high school years make Passages to Canada complementary to the Activity Guide.

[6] Program size and program importance are not correlated. The individuals interviewed for this study would hold the view that a program that fosters good citizenship is very important regardless of its budget.

<< Contents | Previous | Next >>