Evaluation of the Overseas Orientation Initiatives

Appendix A: Evaluation matrix for pre-departure orientation initiatives

Question Indicators Int Admin Data FSW Survey COA Survey Focus Groups Site Visits   Doc Review
Participants Non-
participants
Relevance
1. Is there a continued need to provide pre-departure orientation overseas to newcomers destined to Canada? 1.1 Evidence that pre-departure orientation is needed (differences between immigration categories and source countries)               x Research and literature
1.2 Perceptions of the need for pre-departure orientation x   x x   x x    

2. What is the federal role in the provision of pre-departure orientation overseas to newcomers destined to Canada?

What role do provinces and territories play and to what extent is this role complementary?

2.1 Alignment of provision of pre-departure orientation services with federal responsibilities               x Relevant Acts, legislation, international commitments
2.2 Existence of provision of similar services by P/Ts and other organizations and the extent of complimentary/duplication with federal services x             x Program documentation (from P/Ts)
2.3 Perceptions on the role of the federal government, P/Ts and other organizations in provision of pre-departure orientation services x                
3. How does the provision of pre-departure orientation align with the objectives and priorities of the Government of Canada? 3.1 Alignment of provision of pre-departure orientation with the Federal Government’s stated objectives and priorities regarding settlement               x Foundation documents, Speeches from the Throne, federal budgets
3.2 Alignment of provision of pre-departure orientation with CIC’s stated objectives and priorities regarding settlement x               Departmental RPPs, PAAs, foundation documents
Design and implementation

4. How do COA, AEIP and CIIP align with each other and with other CIC settlement program streams?

How does this approach to delivering pre-departure orientation sessions compare to approaches from other countries?

4.1 Extent to which COA, AEIP and CIIP complement/duplicate each other in terms of reach (geographic areas covered, immigrant classes targeted), scope and depth of information provided (topics, level of detail, length), and linkages to settlement services x           x x Initiative documents (annual and quarterly reports)
4.2 Extent to which pre-departure orientation initiatives build upon and align with other CIC settlement initiatives x             x Settlement program documentation
4.3 Characteristics of overseas orientation programs provided in other countries x             x Program documentation (other countries)
4.4 Stakeholder perceptions on the current approach for delivering pre-departure orientation sessions and alternative approaches x                

5. How effective are current COA, AEIP and CIIP governance structures?

Are they appropriate?

5.1 Description of current governance structures in place in Canada and overseas to support overseas orientation initiatives (e.g., roles and responsibilities, accountability structure, decision-making processes) x           x x Foundation & accountability documents, committee/working group TORs, meeting minutes
5.2 Extent to which there is coordination among the three initiatives (e.g., regular communication, meetings, committees, joint planning) x           x x Initiative documents, TORs for committees /working groups, meeting minutes
5.3 Perceptions on the appropriateness and effectiveness of the initiative governance structures and coordination amongst them x                
5.4 Evidence of a rationale and applied criteria for the allocation of funding to the three initiatives x             x Program documentation
5.5 Identification of issues that may have affected the allocation of resources or the production of outputs x           x x Program documentation
6. To what extent is policy development and initiative management supported by effective tools, resources, information-sharing and coordination, both in Canada and overseas? 6.1 Existence of resources, training and tools to support policy development and initiative management (including for overseas facilitators) x           x x Program budgets, training curriculum, other tools used
6.2 Evidence of procedures and mechanisms in place (committees, working groups) to share information, coordinate policy development and manage responsibilities x           x x Initiative documentation, terms of reference for committees/working groups
6.3 Evidence of linkages between third-party delivery agents and service provider organizations in Canada and between third-party delivery agents and other domestic stakeholders (e.g., designation/regulatory bodies) x           x x Agreements, MOUs, partnership documents
7. Is pre-departure orientation being offered in the right locations and to the right target groups? 7.1 Proportion of landings in Canada, by source country and immigration category in relation to where and to whom pre-departure orientation is offered   x              
7.2 Rationale / strategy is in place for the selection of the location and target groups for orientation offerings x             x Contribution agreements
7.3 Perceptions on whether pre-departure orientation is being offered in the right locations to the right target groups x           x    
8. How effective are current tools and mechanisms to reach potential participants and to promote pre-departure orientation offerings? 8.1 Nature of promotional materials for orientation services (i.e., types, placement, target groups, languages) x           x x Initiative promotional material
8.2 Extent to which potential participants were aware of the orientation services     x x          
8.3 Methods used by potential newcomers to access information on pre-departure services     x x   x      
8.4 Proportions of individuals participating in orientation sessions by immigration class, per location   x           x Contribution agreements
8.5 Perceptions on the effectiveness of promotional tools and mechanisms x   x x   x      
8.6 Evidence of barriers to participation in the orientation sessions (e.g., access, cost, session availability, language)     x x   x x x Research/study papers
Performance
9. To what extent is the pre-departure information provided during orientation sessions appropriate, timely, and useful? 9.1 Participant satisfaction with the effectiveness of the learning environments (length, focus, delivery method) and performance of facilitators           x x x Session feedback forms (as completed by participants)
9.2 Average time elapsed between session date and departure for Canada     x   x x x    
9.3 Perception of whether participants receive orientation within an appropriate timeframe prior to departure x   x     x x    
9.4 Newcomer satisfaction with pre-departure orientation information (usefulness, timeliness)     x     x   x AEIP and CIIP participant feedback forms
9.5 Extent to which pre-departure orientation influences newcomers’ decision-making     x            
9.6 Newcomer perceptions on the usefulness of orientation information to prepare for the trip to Canada     x           AEIP and CIIP evaluations
10. To what extent have COA, AEIP and CIIP contributed to newcomers’ understanding of life in Canada, and their ability to access settlement services? 10.1 Perceptions of whether pre-departure orientation sessions impacted newcomers’ knowledge of life in Canada upon arrival, by type of offering   x x     x x x AEIP and CIIP evaluations
10.2 Perceptions of whether newcomers receive accurate information and develop realistic expectations about life in Canada as a result of receiving pre-departure orientation, by type of offering     x     x x x AEIP and CIIP evaluations

10.3 Views of newcomers who have participated in the COA, AEIP and CIIP with respect to their ability to:

  • deal with a culture shock
  • access settlement agencies
  • access community services
  • settle in (readiness for weather, housing, banking, obtaining SIN/health card, enrolling children in school)
  • enrol in language training (if needed)
    x x   x x x AEIP and CIIP evaluations
10.4 Perception of gaps in pre-departure orientation information x   x   x x x    
10.5 Degree and nature of the difficulties encountered by newcomers in the first 6 months after arrival x   x x   x   x Research and literature
11. To what extent have COA, AEIP and CIIP contributed to newcomers’ preparation for employment in Canada?

11.1 Degree to which COA, AEIP and CIIP participants believe the sessions provided them with information to assist them in:

  • understanding the Canadian labour market (including local labour markets)
  • understanding the steps to take to find employment
  • having credentials and qualifications recognized
  • taking the steps to find employment
  • finding employment commensurate with their skills and experience
  x x     x x x AEIP and CIIP evaluations
12. How efficient is the current approach to providing overseas orientation to newcomers? 12.1 Total CIC budgets and expenditures (O&M, salary, Gs&Cs) by initiative (COA, AEIP, CIIP) by year (cost of delivery for CIC, 3rd parties, and total)   x              
12.2 Range and median cost per session by initiative by year (cost of delivery only for CIC, 3rd party and total)   x              
12.3 Range and median cost per participant by initiative by year (cost of delivery for CIC, 3rd party and total)   x              
12.4 Proportion of clients served compared to the number of landings to Canada, by country where orientation is offered   x              

Features

Page details

Date modified: