Evaluation of the going to Canada Immigration Portal Initiative

Appendix B: Evaluation matrix

Evaluation Questions Indicators Potential Data Sources Report Section
Relevance

1. Does the initiative continue to be consistent with departmental and government-wide priorities?     

Degree of alignment with Departmental Strategic Outcomes

Degree of alignment with the Government of Canada priorities

  • Document review
  • Key informant interviews

3.1.2

2. Does the GTC-IP initiative continue to address a demonstrated need?

Evidence of GTC-IP as information source for prospective and new immigrants

Perceptions of program partners and stakeholders with respect to continued information needs.

  • Web analytics (e.g. Website ranking, traffic)
  • Key informant interviews
  • Survey of prospective and new immigrants
  • Focus groups
  • Document review
  • Literature review
  • External demand data

3.1.1

3. Are CIC and HRSDC the most suitable delivery mechanisms for these information products and tools?

Perception of stakeholders regarding alternative approaches

Level / degree of overlap or duplication with other sources of information (e.g. P/Ts, municipalities, FCRO)

  • Key informant interviews
  • Focus groups
  • Web research
  • Key informant interviews (program, partners)

3.1.3

Performance

4. Is the Portal functional and user friendly?

User perceptions of functionality and user-friendliness (taxonomy, navigation)

IM/IT professional evaluation of quality of information architecture / taxonomy / navigation (e.g. appropriate selection and use of meta data)

# of errors/malfunctions on the site

Scanning analysis of Entry Pages (i.e. pages visitors are using to access the Websites/Portal – Are users coming to the site at the expected pages?)

Performance of on-line service delivery  (uptime and time to fix downtime)

  • Online user survey
  • Web analytics /Log File Analysis
  • External Demand Data
  • Usability testing
  • Client inquiries
  • CLF2 compliance testing
  • Program documentation
  • Search Engine Research
  • On-line surveys
  • Web analytics /Log File Analysis – link analysis (links to/from Website)
  • Help desk tickets raised
  • Web analytics /Log File Analysis
  • Service Level Definitions
  • Operations Centre Metrics

3.2.6

5. Is the governance structure, internally within each department (CIC and HRSDC), between CIC and HRSDC, and between GoC and P/Ts) for the Portal effective?

Perceptions of the effectiveness of the governance structure

  • Including comparisons of CIC’s GTC-IP governance structure with that of HRSDC’s and other similar government websites

Senior management perceptions of governance structure

GTC-IP stakeholder and partner understanding / clarity of roles and responsibilities

Timeliness of decision-making

Communication structures in place

Adequacy of meetings to address issues and documentation to record decisions

  • Interviews with key informants (partners, program, management)
  • Document review (i.e. other departmental website evaluations)
  • Interviews with key informants (management)
  • Interviews with key informants (partners, program, management)
  • Document review
  • Interviews with key informants (partners, program, management)
  • Document review
  • Interviews with key informants (partners, program, management)
  • Document review
  • Key informant interviews

3.2.1

6. Is the Portal URL the most appropriate, efficient and effective to meet the needs of the target population?

Stakeholder perceptions of appropriateness of location (P/Ts, service organizations)
Technical implications (feasibility, efficiency etc)

Branding implications
Usage patterns on Portal vs. CIC main site

  • Key informant interviews
  • document/file review (of any correspondence)
  • Key informant interviews
  • Technical documentation
  • Focus groups
  • Key informant interviews (with stakeholders)
  • Web analytics

3.2.10

7. Is the target population aware of the GTC-IP? Are they using the GTC-IP? Why or why not?

Extent to which Portal communications products, promotional and outreach activities are targeted/tailored to the primary audience

Profile and awareness of GTC-IP among prospective and new immigrants

Usage patterns for the GTC-IP

  • Trend analysis of referral patterns/traffic patterns (total visits, number of pages per visit, time per visit etc)
  • Usage patterns in comparison with other immigration, settlement and labour market information sources

Visits by date stamp compared to marketing activities

Impediments to use

  • Document review
  • Focus groups
  • Interviews with key informants
  • Web analytics
  • Survey of prospective and new immigrants
  • Key informant interviews (Stakeholders)
  • Web analytics (CIC)/ Log Files (HRSDC)
  • Literature review
  • Focus group
  • Log file analysis
  • Key informant interviews (partners, stakeholders)\
  • Focus group of users

3.2.5

8. Is collaboration and priority setting between and among the federal, provincial and territorial partners effective?

Instances of sharing of best practices and tools

Evidence of information sharing between P/Ts and other stakeholders such as municipalities, employers, immigrant service organizations

Level of partner satisfaction regarding effectiveness of collaboration

Level of partner satisfaction with mechanisms for engagement

Evidence of meetings, consultations, working groups, MOUs and other formal agreements with partners and stakeholders

  • Document review
  • Key informant interviews
  • Document Review
  • Key informant interviews (P/Ts, stakeholders)
  • Key informant interviews (partners, stakeholders)
  • Key informant interviews (partners, stakeholders)
  • Document review

3.2.2

9. Do the provincial/territorial portals contain and provide links to relevant, up-to-date and understandable information on:

  • immigrating, settling, living,  visiting and studying in the P/Ts?
  • working in the P/Ts?

Number of P/T/municipal websites developed and launched

Quality of P/T Portal content as assessed by public/service providers and subject matter experts

User satisfaction with information provided on P/T Portal

Plain language benchmarks

Visitor traffic patterns

  • Web site review
  • On-line user survey
  • Key informant interviews (stakeholders)
  • Focus groups
  • Assessment by subject matter experts
  • On-line user survey
  • Interviews (stakeholders)
  • Focus groups
  • Public opinion research
  • Web analytics for links to and from portal
  • Review and ranking by a plain language expert
  • Web analytics / Log Files

3.2.7
3.2.8

10. Does the Portal contain and provide links to relevant, up-to-date and understandable information on:

  • immigrating, settling, living,  visiting and studying in Canada?
  • working in Canada?

Quality of Portal content as assessed by public/service providers and subject matter experts

User satisfaction with information provided on Portal

Average time since last page update

Plain language benchmarks

Content enhancements and releases

Visitor traffic patterns

  • On-line user survey
  • Key informant interviews (stakeholders)
  • Focus groups
  • Assessment by subject matter experts
  • On-line user survey
  • Interviews (stakeholders)
  • Focus groups
  • Public opinion research
  • Web analytics for links to and from portal
  • Review and ranking by a plain language expert
  • Program files
  • Web analytics / Log File Analysis
  • Web analytics / Log Files

3.2.7
3.2.8

11. Does the target population gain knowledge and settlement information regarding provinces, territories and communities from across Canada?

Users identified increased knowledge of settlement information

Information gaps identified

# of users going from the GTC Website to the provincial/territorial Websites

# of users on pages containing settlement information

  • On-line user survey
  • Survey of prospective and new immigrants
  • Key informant interviews (stakeholders)
  • Focus groups
  • Provincial web analytics
  • Web analytics

3.2.9

12. Does the target population gain knowledge of immigrating to Canada and, living and working in Canada?

% of surveyed users identifying increased knowledge of living, working in and immigrating to Canada

Information gaps identified

# of users who get a WIC report

# of users who get a SPO tool report

  • On-line surveys (users)
  • Survey of prospective and new immigrants.
  • Key informant interviews (stakeholders)
  • Focus groups
  • Web analytics

3.2.9

13. Does the Portal contribute to the target population’s ability to make informed immigration decisions, prepare for the immigration process and integrate into Canada upon arrival?

% of surveyed users identifying the Portal as informing their decision- making process

% of surveyed users identifying the Portal as facilitating integration

Usefulness as identified by intermediaries (P/Ts, domestic and international immigrant service organizations, Canadian Embassies and Consulates)

  • On-line User Survey
  • Survey of prospective and new immigrants
  • Focus groups (possibly comprised of recent new immigrants and a comparison group (possibly international) (both bullets)
  • Key informant interviews

3.2.9

14. What have been the unintended outcomes of the GTC-IP, if any?

Identified unintended outcomes

  • Key informant interviews
  • Focus groups

3.2.2
3.2.4

15. Is delivery of the GTC-IP efficient and cost-effective?

Extent of planned to actual resource (FTEs, O&M, contribution funds) use by planned activity

Ratio of management costs to funds contributed to P/Ts (management burden on contribution funds)

Cost per user (cost of initiative/# of users)  and trend of that cost

Cost variance with private sector options

CMS cost-effectiveness and efficiency (timeliness, user friendliness)

  • Document review
  • Document review
  • Document review
  • Web analytics
  • Document review (of Business Cases, MC, etc.)
  • Secondary research, case studies and international comparisons
  • Key informant interviews
  • Document review (e.g. Close Out Reports, project documentation)
  • Secondary research, case study, benchmarking

3.2.3
3.2.11

Page details

Date modified: