Formative Evaluation of Canada’s Citizenship Week

2.0  Evaluation Findings

Findings
Output Results
Outcome Results
Teachers
Guide and Scout Leaders
Programming Suggestions
Key Change Suggestions

Findings

1. CCW is Not a Week

The key material, events and messages associated with Canada’s Citizenship Week (CCW) are year-round both in terms of their utility and availability. CCW is merely a week designated as being of higher profile than most of the rest of the year.

The discussion within Citizenship and Immigration Canada (CIC) regarding the programming efforts for CCW centres around the activity guide and one or more high-profile Citizenship ceremonies held during CCW. The order form for material, which is timed for CCW, provides space for ordering the following:

• The Spirit of Home activity kit—a year-round activity kit that includes an activity guide, a colourful poster and a sample of Times of My Life.
• Reaffirmation Ceremony kit—a Reaffirmation Ceremony is a formal event where people who are already Canadian citizens repeat the Oath Of Citizenship to express their commitment to Canada. The kit, designed for a group of 30, includes information on how to organize a reaffirmation ceremony, a sample program, 30 reaffirmation certificates and 30 “O Canada” bookmarks.
• Welcome Home activity poster.
• Canadian hand flags.
• O Canada” bookmark.
• Times of My Life (a family tree activity).

A review of these items shows that none of the material is specific to CCW. In fact, the activity guide is advertised as appropriate on a year-round basis and the other material is obviously useable at any time of the year [note 7]. Obvious occasions for the hand flags and the “O Canada” bookmarks are Canada Day and Flag Day. This is not to suggest that the items are inappropriate for CCW, but it does underline the fact that CCW does not have unique material.

The point of this observation is that the evaluation of CCW cannot be done in an isolated fashion. CCW is an important element of IP, with which it shares common objectives.

2. Lack of Integrated Program Delivery

On the positive side, there is a spirit of open communication, a willingness to share plans, and a cooperative spirit among all of the key actors involved with CCW. There is very clearly a shared vision (i.e., Canadians and newcomers are to be aware of what it means to be a Canadian citizen) and a shared set of activity objectives. There are four key groups for whom cooperation and shared understandings are needed: Integration Promotion (IP), Headquarters Distribution Centre, Communications and the regions. This evaluation finds that while these groups are in communication and seek to be cooperative, they do not succeed to the satisfaction of all. For example, there is a slowness in advising and in seeking advice from the other groups of forthcoming events, so that all groups will have sufficient time to influence the event planning. Also, there has been a tendency to make changes from one year to the next without explaining them to all groups. The clearest example of this is the decision not to continue the Citation Ceremonies this year, which was made without explaining the change to those who were anticipating the event.

But a further stage is possible. The good will evidenced among those involved with CCW suggests that a higher level of integration is possible. More advance common planning is called for by several of the individuals involved. In the words of one, "We still operate in silos. Yes, we talk across silos but it would be marvellous if the walls could come down and there could be true sharing of what can be done." Additional strategic planning may serve to:

• reinforce the CCW role in the broader context of fostering national pride and awareness of rights and responsibilities of Canadian citizenship among all Canadians;
• garner ideas from all levels of CCW program delivery in support of additional operational efficiency and program effectiveness; and
• maximize media coverage, which is today the primary means of influencing the attitudes of the adult general population.

The offering of these suggestions should not be construed as indicative of current program failure. The primary observation was of a spirit of open communication and cooperation. The additional strategic planning suggestions are put forward for the purpose of achieving even more.

3. Teachers Use Activity Guide Selectively

Appendix E presents the results of a teacher survey (the interview themes are in Appendix F). The following is a summary of the survey findings.

The activity guide most often finds its way to teachers in regular classrooms at the elementary grade levels. This is the case for both English and French schools. About one-quarter of the classrooms are focused on the teaching of the other official language or, less frequently, they are classrooms for newcomers.

The guide is ordered most frequently by teachers, followed closely by principals.

Three-quarters of those who answered the interviews are also those who ordered material. The other teachers had the material ordered for them, most likely by a principal.

By the end of October of the school year, relatively few of the teachers had actually used the guide in the classroom (five of 24 English and five of eight French), although material sent to the school for CCW was used more extensively, since posters, bookmarks and flags were also put to use.

The level of detailed knowledge of the guide is very low. Teachers were not comfortable comparing this year’s guide with that from last year. Teachers were not able to rate the themes used in the guide. Those teachers using the guide selected specific activities for use. They were not interested in the overall organization of the guide.

When asked for an overall opinion, most teachers who answered were very positive, but they were unwilling to judge the guide because they said that they had not yet adequately examined its contents.

It is clear from the survey responses that the users of the activity guide do not consider CCW as the prime time during the school calendar for use of the activities of the guide. Teachers consider the content of the guide to be another resource for them in delivering curriculum. Users have also said that promotional items such as flags, pins and bookmarks are very useful as “interest-pegs” on which to hang teaching plans.

This finding (i.e., that the various promotional items and the activity guide are resources to be encouraged throughout the year) complements the current practice of IP to ask for a higher profile event every month if at all possible. The findings of this study reinforce the utility of this practice.

Output Results

The first, and key, output to consider is that of uptake — the extent to which the target populations complete order forms and receive material.

In theory it should be easy to have uptake measurement of the key outputs. All that is required is the numbers of requests in relation to the target populations of teachers, Guide and Scout leaders, and other youth leaders.

In practice, uptake calculations require significant work. Normally, an accurate count of the numbers of orders can readily be obtained. What is difficult is the estimation of the population size from which the orders come. In other words, what would be the size of a 100 percent uptake?

For teachers, the task will be largely one of validating the list of schools that is used as the population to which to send order forms. Sample studies can readily establish the extent to which this list has errors of omission (schools eligible to receive an order form but which are not on the list) and of commission (schools on the list but which should not be listed).

For the Scouts and Guide population, the task is more difficult because there is not a list on hand at CIC. It is likely that lists can be obtained from the national organizations. At present, order forms come from Scout and Guide leaders who find a form in their monthly magazines, or who know of the activity guide and seek the form from the CIC website, or who are sent a form because they were on the CIC mail-outs in an earlier year. If a list can be obtained, it will require the same type of validity check as would be required for the list of schools.

The current evaluation study was provided by a manual search of the order forms, the names of schools in Ottawa and Gatineau that ordered CCW material. Because the mailing list for filling order forms to schools was also available, it was possible to calculate those uptake statistics. The list itself was not validated and, therefore, the statistics should be recognized to be approximations.

These 313 Ottawa schools were sent order forms and invited to order the CCW material that they wanted. Requests for material were received from 62 schools in Ottawa (20 percent of the number that had been sent order forms). Without a standard or other benchmark against which to compare this uptake, there is no norm against which to evaluate the performance. Nevertheless, a subjective assessment suggests that one-in-five is a good uptake rate.

There were 70 schools in Hull, Gatineau, Aylmer and Buckingham that were sent order forms and invited to order the material that they wanted. Requests for material were received from 15 schools (21 percent of the number that had been sent order forms). This is very similar to that for Ottawa and, again, it must be stated that there was no standard or other benchmark against which to compare this uptake. Nevertheless, as before, a subjective assessment suggests that one-in-five is a good uptake rate.

Outcome Results

This evaluation does not provide evidence on whether CCW succeeds in encouraging Canadians to recognize the value of citizenship and immigration, nor does it measure whether it encourages them to focus on the privileges, rights, responsibilities and obligations of citizenship.

It does offer evidence that those users of the activity guide who were interviewed had positive views about the guide and its utility for teaching situations.

Teachers

Based on the judgments of those few teachers surveyed who had actually used the Guide, the activities presented in it were found to be valuable for use both in the regular classroom and the second-language teaching classroom. Unfortunately, this conclusion is limited by geographical and by calendar constraints due to the nature of the survey. Teachers in only two cities, both in the same region, were telephoned. Several of the teachers asked that the activity guide be sent earlier in the school year.

Guide and Scout Leaders

Of the 13 youth leaders in Ottawa and Gatineau who received the guide, five were interviewed (see Appendix G). They reported the activity guide to be very good. They found it useful, clearly written, concise, written at a level that was accessible to the Guides themselves. The leader of a younger Girl Guide group was especially happy that it contained topics that could be used for yearly activities. Times of My Life (Family Tree) and the Reaffirmation Ceremony were the two components that these leaders had used so far. All of the leaders interviewed plan to use other parts of the booklet during the remainder of the year. Several leaders said they had also used the 2001 book and will continue to order the activity booklets and other material each year. These youth leaders said that the items ordered should be sent out early, no later than August, so that the leaders can have time to read the material before organizing their yearly programs.

Programming Suggestions

Many of the individuals involved with the CCW activities and the distribution or use of materials made suggestions for improvements and/or comments on the utility of material. These are presented in Appendix H. They are placed in an appendix rather than in the body of this report, because they should not be taken as evaluative conclusions about the material or procedures. Rather, they are suggestions from interested participants and, as such, are very meaningful. That said, they do not have the rigour of systematically collected findings and serve more as hypotheses for further consideration than as judgments about CCW components.

Key Change Suggestions

A. View CCW as a Contributing Component

CCW is not a stand-alone program. Its objectives are integrally linked to Integration Promotion (IP), which itself is directly linked to the Citizenship Program. They all share common objectives pursued on a year-round basis. CCW is but one week designated to be of relatively high profile. It and its parent programs seek to encourage Canadians to recognize the value of citizenship and immigration, and to encourage them to focus on the privileges, rights, responsibilities and obligations of citizenship.

Management Response

We agree with this recommendation.

This philosophy has been an integral component of our planning and messaging since the establishment of Canada’s Citizenship Week, and several initiatives are currently under way to strengthen this resolve.

B. Integrate Planning

There is a need for even more strategic planning and advance cooperation among the four key groups for whom cooperation and shared understandings are needed: IP, Headquarters Distribution Centre, Communications, and the regions. This likely requires proactive management at a senior level.

Management Response

We are neutral on this recommendation.

We feel that these four groups do cooperate well together, given competing demands for attention along with time and monetary constraints. We will, as always, endeavor to improve internal communications within the parameters of realistic expectations.

C. Maximize Access to Curriculum Resources

View the annual guides as successive editions of collections of youth activities for the classroom and elsewhere. Teachers and youth leaders find the suggestions valuable but need them available on demand, in a manner that caters to educators and youth leaders making activity choices under time pressures and at all times of the year. Make it easy to pick and choose activities from the large collection already developed. The ongoing (part-time) services of an education-focused, Web-literate librarian could prove very useful in this key requirement.

Management Response

We agree with this recommendation.

We attempt to ensure that all of our products are easily accessible and user friendly, and will continue to make this a priority.

D. Use Symbiotic Relationships

Coordinate with provincial ministries of education and national organizations servicing youth such as the Girl Guides and Scouts. Partnerships are a valuable route by which to leverage the bringing of citizenship messages to Canadians.

Management Response

We agree with this recommendation.

Education is a provincial responsibility; this, therefore, presents difficulties in federal-provincial jurisdiction issues.

E. Monitor Performance

Uptake statistics are the basic requirement for performance management of CCW. The database used for advertising the availability of material, for receiving requests and for filling orders should be made capable of providing uptake rates by categories of target populations.

Beyond uptake statistics, results measurement is needed. Samples of users and non-users should be monitored for use of communications products and for use of didactic materials by teachers and youth leaders. Longitudinal attitude measurement should be maintained. Existing public opinion polls can continue to be searched for relevant existing measurement over time and, if not adequately available, a limited set of questions should be added to an omnibus survey.

Management Response

We agree with this recommendation.

One of our key short-term priorities is to develop a database to facilitate better uptake statistics.

F. Management Response to Evaluation as a Whole

This evaluation brings into focus the need to review and identify the best strategies for the effective and efficient promotion of citizenship issues across government and for CIC’s contribution to this objective. It is recommended that a future evaluation be undertaken to study this question and make recommendations for the best use of funds within CIC, taking into consideration other activities of the Government of Canada that promote related themes.

_______________

[7] Page iv of the activity guide notes, “Although the guide is launched during Canada’s Citizenship Week …, please use it throughout the year.”

<< Contents | Previous | Next >>