Evaluation of the Nationally Standardized Data Collection Strategy on Hate-Motivated Crime

2. Methodology

2.1. Evaluation issues and questions

The evaluation of the Data Collection Strategy examined issues related to relevance, performance, and alternatives. Table 2-1 details the evaluation issues and questions addressed in the evaluation (see Appendix A for the complete evaluation matrix, which also includes specific indicators and methodologies for each evaluation question).

Table 2-1: Summary of evaluation issues and questions
Evaluation issue Evaluation question
Relevance
  • Is there a continued need for a nationally standardized Data Collection Strategy on hate-motivated crime?
  • Does the Data Collection Strategy continue to be consistent with departmental and government-wide priorities?
  • Is the Strategy consistent with federal roles and responsibilities?
Performance
  • Do police services and other stakeholders understand the utility and necessity of collecting hate-motivated crime data?
  • Do police services have the training and tools necessary to identify and collect standardized hate-motivated crime data?
  • Is Statistics Canada collecting high-quality, nationally standardized hate-motivated crime data?
  • To what extent is nationally standardized hate-motivated crime data made available?
  • Do stakeholders have an understanding of the nature and extent of hate-motivated crime provided by Statistics Canada?
  • Have stakeholders applied their knowledge of hate-motivated crime to improve programs, policies and services?
Alternatives
  • Is Statistics Canada the most suitable delivery mechanism for the data strategy?

2.2. Data collection methods

The evaluation of the Data Collection Strategy included the use of multiple lines of evidence and complementary research methods to help ensure the reliability of information and data collected. The following data collection methods were used to gather data for the evaluation:

  • stakeholder interviews;
  • document review;
  • review of administrative data; and
  • survey of police services.

Each of these methods is briefly described in the following section. Appendix B provides additional detail on these methods.

2.2.1. Stakeholder interviews

A total of 24 interviews were completed for the evaluation. Interviews were undertaken with four key stakeholder groups: CIC’s Citizenship and Multiculturalism Branch; CCJS; the Police Information Statistics Committee (POLIS); and other stakeholders, including non-governmental organizations (NGOs), academics, other government departments (OGDs), and international representatives. The number of interviews completed for each interview group is illustrated in Table 2.2 (see Appendix C for a list of interviewees and Appendix D for the interview guides).

Table 2-2: List of interviews by interview group
Interview Group Number of Interviews Conducted
CIC Citizenship and Multiculturalism Branch 3
Canadian Centre for Justice Statistics 2
Police Information Statistics Committee 9
Other Stakeholders 10
Total 24

2.2.2. Document review

Documentation was reviewed primarily to inform an assessment of the relevance of the Data Collection Strategy and to determine whether any feasible alternatives exist. (Appendix E contains a list of documents that were reviewed for the evaluation.) The following types of documentation were reviewed during the evaluation:

Corporate, accountability and political documents: included CIC’s Departmental Performance Reports and Reports on Plans and Priorities, agreements between CIC and CCJS, and Speeches from the Throne.

Materials produced by the Strategy: included CCJS annual project reports, presentations and training materials.

Hate Crime Reports and Academic Papers: included the CCJS annual hate crime reports and academic literature that provided information and perspectives on hate crime in Canada.

International Reports and Conventions: included documents related to Canada’s participation in any world conferences on racism and Canada’s response to international reporting commitments.

2.2.3. Survey of police services

A survey was administered to municipal, provincial and federal police services to gather information on whether they believe there is a need to have hate crime data collected in a standardized way, whether and how they find the data useful, and their perceptions of the training provided by CCJS. The Canadian Association of Chiefs of Police (CACP) administered the survey on CIC’s behalf. The CACP has 428 active members from police services across Canada. The active membership list was cleaned to ensure that the survey was targeted to the appropriate population. The survey was pre-tested with selected POLIS interviewees that included federal, provincial, and municipal police services. (See Appendix F for the survey.)

The CACP e-mailed the survey to a total of 368 members.Footnote 5 In a separate e-mail, 26 First Nations police services were sent the survey (using a distribution list located on the First Nations Chiefs of Police Association website). In total the survey was sent to 394 individuals (Table 2-3). In efforts to increase the survey coverage, the communiqué that accompanied the survey asked that the survey be forwarded to anyone else within the police service that might be able to respond. A total of 92 responses were received. Every effort was made to ensure wide coverage, however, a low number of responses were received from certain types of services. Due to the approach used for survey administration (i.e., recipients were asked to forward the survey to others), a response rate for the survey cannot be calculated.

Table 2-3: Number of survey recipients and responses, by type of police service
Type of Police Service Number of Recipients Number of Responses
Municipal 248 77
First Nation 26 6
Royal Canadian Mounted Police 78 4
Ontario Provincial Police 11 4
Sûreté du Québec 18 4
Royal Newfoundland Constabulary 5 0
Military Police 8 1
Total 394 92

2.2.4. Administrative and financial data review

A review of administrative data was completed to establish the level of activities/outputs of the Strategy. These data included: the number of training and workshops delivered, the results of training feedback forms, the number of presentations or conferences attended, and the number of polices services using UCR2.2. This information was extracted from the annual CCJS project reports and the annual CAPAR reporting template.

Financial information from the Data Collection Strategy was also examined to determine the total costs of the initiative.

2.3. Limitations and considerations

There are several methodological limitations and considerations that should be noted. In light of these considerations, the evaluation was designed to use multiple lines of evidence (e.g., interviews, survey, administrative data, documentation) to strengthen the reliability and validity of the evaluation results.

The sample for the survey of police services did not include all police services across Canada.

No complete contact list of all police services and detachments across Canada was identified during the evaluation. While effort was undertaken by the evaluation team to include wide coverage of police services in the survey sample, not all police services were surveyed, as not all services were either active members of CACP or included in the list of First Nations police services. Furthermore, as the e-mail communiqué encouraged police services to forward the survey invitation to other relevant members of their police service, and as survey responses could not be tracked by specific police service, the coverage across police services represented by survey respondents is not known. Survey respondents were most likely to be employed at municipal police services, and relatively few respondents were employed at RCMP detachments or with provincial police services. As a result, the survey of police services is not representative of the universe of police services across Canada.

Survey and interview respondents had varying degrees of familiarity with the Data Collection Strategy and may not have represented all stakeholders making use of the information produced by the initiative.

The evaluation aimed to obtain input from a wide range of stakeholders through both the survey of police services and interviews with academics, NGOs and OGDs. However, survey and interview respondents demonstrated varying degrees of familiarity with the Data Collection Strategy and responses reflected this range of knowledge. While effort was expended to obtain the input from the most suitable representatives, it is not known whether respondents represented those most knowledgeable about the Data Collection Strategy at their organization. This was perhaps an issue at larger police services, for example, where there may be multiple staff members (including research analysts) familiar with the available crime statistics in Canada. The survey respondent, who in many cases was the Chief of Police, may not have been the main user of the hate crime information products.

Identifying academic researchers and other stakeholders who had made use of the information produced by the Data Collection Strategy proved to be a challenge. While effort was made to identify and interview relevant academics, government policy makers and staff of non-governmental organizations, it is possible that some were missed. Since no lists of those who had accessed the information products, or received CCJS training, were available to the evaluation team, it is not known whether there were major beneficiaries of the Data Collection Strategy who were not consulted for the evaluation.

Page details

Date modified: