Appendix A: ICCRC Evaluation Matrix

Type Question Indicators Methods / Data sources
Relevance What need is the regulation of immigration consultants aiming to address? Reasons why the industry is being regulated (e.g., level of fraud, immigration practitioner issues, use of ‘ghost consultants’) Document review
(Relevant industry reports, Standing committee reports, POR)
Key informant interviews
(CIC, ICCRC, CBSA, IRB, ESDC, CAPIC, CBA)
ICCRC member survey
# and % of immigration applicants who use the support of immigration consultants (by mission, immigration category) Review of program data
(CAIPS, FOSS, GCMS)
Is the regulation of immigration consultants aligned with federal roles and responsibilities and GoC and CIC priorities? Existence of federal legislation and obligations related to the regulation of immigration consultants Document review
(Federal legislation and regulations (E.g., IRPA))
Alignment of the objectives of the regulations with GoC and CIC priorities Document review
(Speeches From the Throne / Budgets, CIC reports on Plans and Priorities, CIC Annual Reports)
Performance Are the appropriate governance and management processes in place to achieve program outcomes? Mechanisms are in place for coordination, communication and decision-making within CIC, between CIC and other government departments, and between CIC and ICCRC Key informant interviews
(CIC, ICCRC, CBSA, IRB, ESDC)
Review of program data
(Committee terms of reference, Meeting records)
Stakeholder perceptions on the effectiveness of mechanisms for coordination, communication, and decision-making Key informant interviews
(CIC, ICCRC, CBSA, IRB, ESDC)
Processes and tools are in place for CIC and other government departments to validate bona fide representatives Key informant interviews
(CIC, ICCRC, CBSA, IRB, ESDC)
Review of program data
(CIC processing manuals)
Processes and tools are in place for CIC and other government departments to report misrepresentation Key informant interviews
(CIC, ICCRC, CBSA IRB, ESDC)
Review of program data (GCMS)
# of complaints received through the complaints mailbox (ImmigrationRepresentatives@cic.gc.ca) on authorized and/or unauthorized consultants Review of program data
(OMC data)
Is ICCRC a viable, transparent, accountable, and well-managed organization? (Capacity) Capacity building activities and outputs are delivered as planned, with explanation of variances Key informant interviews
(CIC, ICCRC)
Review of program data
(ICCRC output and outcome reports, Annual reports, Committee terms of reference and records of decision, Financial reports, Meetings minutes, Organizational charts, By-laws, Code of ethics, Membership)
Evidence of public reporting on organizational structure and activities, processes, and investigations Review of program data
(ICCRC annual reports, Web publications, Communication products)
ICCRC balance of revenues against expenditures, by fiscal year Review of program data
(ICCRC financial reports)
Stakeholder perceptions on the appropriateness of governance and management structures that have been put in place Key informant interviews
(CIC, ICCRC, CBSA, IRB, ESDC, CAPIC, CBA)
ICCRC member survey
Stakeholder views on the transparency and accountability of ICCRC Key informant interviews
(CIC, ICCRC, CBSA, IRB, ESDC, CAPIC, CBA)
ICCRC member survey
Has the ICCRC adequately informed stakeholder groups on the immigration consulting sector? (Communication) Communication activities and outputs are delivered as planned, with explanation of variances Key informant interviews
(CIC, ICCRC)
Review of program data
(ICCRC administrative files, Output and outcome reports, Communication plan, Communication products, Web material)
% of information products made available in both official languages and in target applicant languages Review of program data
(ICCRC administrative files, Output and outcome reports, Communication plan, Communication products, Web material)
Stakeholders are aware of, and have access to, ICCRC information products Key informant interviews
(CIC, CBSA, IRB, ESDC, CAPIC, CBA)
ICCRC member survey
Stakeholders believe that ICCRC information products are appropriate and effective Key informant interviews
(CIC, ICCRC, CBSA, IRB, ESDC, CAPIC, CBA)
ICCRC member survey
Are members receiving accreditation from the ICCRC? Are they receiving professional development opportunities to develop competencies and improve qualifications? (Competencies) Competency activities and outputs are delivered as planned, with explanation of variances Key informant interviews (CIC, ICCRC)
Review of Program Data
(ICCRC administrative files, Output and outcome reports, Documentation on accreditation process / certification procedures, Training tools and activities, Professional development tools)
# of new ICCRC members, by month Review of Program Data
(ICCRC annual reports, Membership records)
# and % of members certified Review of Program Data
(ICCRC membership records, training records)
# and % of members meeting continuing education requirements Review of Program Data
(ICCRC annual reports, Training records)
Stakeholder perceptions on the accreditation process Key informant interviews
(CIC, ICCRC, CBSA, IRB, ESDC, CAPIC, CBA)
ICCRC member survey
Stakeholder perceptions on the sufficiency of professional development opportunities Key informant interviews
(CIC, ICCRC, CBSA, IRB, ESDC, CAPIC, CBA)
ICCRC member survey
Stakeholder perceptions on the quality of training provided Key informant interviews
(CIC, ICCRC, CBSA, IRB, ESDC, CAPIC, CBA)
ICCRC member survey
Review of program data
(Training feedback forms)
To what extent is there a fair, transparent, and accessible complaint and discipline mechanism in place to regulate member conduct? (Compliance) Compliance activities and outputs are delivered as planned, with explanation of variances Key informant interviews (CIC, ICCRC)
Review of program data
(ICCRC administrative files, Output and outcome reports, Code of ethics, Complaints procedures, Investigations procedures, Enforcement mechanisms, Liability insurance plan)
# of complaints filed with the ICCRC, by type of complaint Review of program data
(ICCRC annual reports, Complaints reporting)
# and % of complaints referred to other stakeholders, by type of complaint Review of program data
(ICCRC annual reports, Complaints reporting)
# and % of investigations conducted by the ICCRC, by type of complaint Review of program data
(ICCRC annual reports, Complaints reporting)
# and % of complaints that are heard by the complaints committee; and discipline committee, by type of complaint Review of program data
(ICCRC annual reports, Complaints reporting)
# and % of infractions issued by the ICCRC, by type of complaint Review of program data
(ICCRC annual reports, Complaints reporting)
% of complaints launched and finalized within 3 months Review of program data
(ICCRC annual reports, Complaints reporting)
# and % of complaints or disciplinary decisions appealed by either party Review of program data
(ICCRC annual reports, Complaints reporting)
Stakeholder awareness and understanding of compliance and discipline mechanisms Key informant interviews
(CIC, ICCRC, CBSA, IRB, ESDC, CAPIC, CBA)
ICCRC member survey
Stakeholder views on adequacy of compliance and discipline mechanisms Key informant interviews
(CIC, ICCRC, CBSA, IRB, ESDC, CAPIC, CBA)
ICCRC member survey
Resource utilization Have program resources been used appropriately to achieve program outcomes? ICCRC planned versus actual costs, and reasons for variations Key informant interviews (CIC, ICCRC)
Review of program data
(ICCRC financial reports)
Stakeholder perceptions on whether program resources have been allocated appropriately to achieve program outcomes Key informant interviews
(CIC, ICCRC)

Page details

Date modified: