Audit Report - Audit of the Designated Medical Practitioner Management Control Framework

(Ottawa Region – North America)

Audit Report
Internal Audit and Accountability Branch
Citizenship and Immigration Canada
July 2006

Table of Contents

• Executive Summary
  • Summary of Findings
  • Summary of Recommendations
• 1.0 Introduction
• 2.0 Audit Objective and Scope
• 3.0 Audit Approach
• 4.0 Audit Reporting and Assessment
• 5.0 Detailed Observations and Recommendations
  • 5.1 Governance
  • 5.2 Organizational Capability
  • 5.3 Risk Management
  • 5.4 Performance Management
  • 5.5 Operational Delivery
• 6.0 Conclusion
• Appendix A: Management Control Framework
• Appendix B: Audit Criteria
• Appendix C: DMPs, Ottawa Region
• Appendix D: Management Action Plan

Executive Summary

Designated medical practitioners (DMPs) are qualified physicians appointed by the Medical Services Branch (MSB) of Citizenship and Immigration Canada (CIC) to conduct medical examinations on all applicants for permanent residence and some visitors who apply to enter Canada. During the summer of 2004, following a risk analysis, senior management and the departmental Audit and Evaluation Committee agreed that the focus of this DMP audit would be on the management control framework (MCF) of the MSB-Ottawa Region, which has responsibility for managing DMP activity in North America. The audit was initiated in late September 2004, at around the time a new director general was appointed for MSB. The audit was conducted between September 2004 and March 2005.

The Internal Audit and Accountability Branch is reporting on the DMP MCF using a four-stage development scale (a description of each of which is provided in section 4.0). Table 1 presents our summary assessment of the stage of development for each of the five elements of the MCF: governance, organizational capability, operational delivery, risk management and performance management.

Summary of Findings

The governance element of the DMP MCF had the fundamentals in place.

The organizational capability element was undergoing a major reorganization during the time of the audit. The roles, responsibilities and accountabilities of the managers involved in the DMP process were being examined for redistribution.

The operational delivery element had the fundamentals in place. For example, we found that there was a systematic process in place to select qualified DMPs. These DMPs were provided with orientation tools, including a DMP handbook, the primary guidelines for DMPs. Since 2002, MSB has conducted regular onsite audits of DMPs, covering at least 75% of the DMPs in the Ottawa Region.

We found that there was no formal process in place for reviewing DMP appointments that are made for a five-year period, nor did we find formal procedures or criteria for determining satisfactory performance. We also noted that the medical officer (MOF) guidelines had not been updated since 1992.

CIC developed an integrated risk management framework (IRMF) in December 2002 and support was provided to departmental operations to establish formal risk management practices. We did not find documented risk assessment strategies and related processes and practices, information on country-specific health risks, or regular reporting to senior departmental management on DMP-related risks and how those risks are being managed.

Performance management relies on using meaningful measures and indicators of performance. We found that a significant amount of DMP performance information was captured in the DMP appointment process, the immigration medical examination (IME) process, the annual audit (self-evaluation form) and the onsite DMP audit. This information has been captured in key systems. We did not find key measures and indicators of performance defined and regular performance reports produced for management, nor were we able to determine how the information is used for DMP performance rating or reappointment decisions.

Overall, MSB has made significant progress and is continuing to implement various elements of the DMP MCF. The fundamentals are in place regarding the governance and operational delivery elements, and improved articulation of risks and the reporting of performance are required. Progress is being made in implementing important organizational changes and MSB senior management has committed to continued efforts to implement the DMP MCF.

Summary of Recommendations

• MSB should continue to implement risk management practices and assign responsibility to a manager for formalizing risk management practices for the DMP process. These risk management practices should include risk articulation and documentation of country-specific risks, mitigation strategies and reporting on risk management.
• Suitable performance measures and indicators for management and operational requirements should be established with associated validation and reporting procedures.
• The MOF guidelines should be updated and the systems and the processes for DMP reappointment should be implemented.

1.0 Introduction

The Immigration and Refugee Protection Act (IRPA), in force since June 28, 2002, provides the legislative framework for delivering on Canada ’s commitment to immigration. This commitment has ranged from 220,000 to 245,000 new permanent residents for fiscal years 2003–2004 and 2004–2005. Before an applicant enters Canada, an application is submitted to an officer who will ensure that the applicant is admissible and meets the requirements of the Act. The goal of the medical program at CIC is to protect Canadian public health and safety and to limit the cost and the strain on the resources of the Canadian health and social services system.

The MSB is responsible for protecting the health and safety of Canadians by providing the immigration officer with an assessment regarding the inadmissibility of applicants on health grounds. Applicants are inadmissible on health grounds if their health condition:

1. is likely to be a danger to public health;
2. is likely to be a danger to public safety; or
3. might reasonably be expected to cause excessive demand on health or social services (IRPA S38).

DMPs are qualified physicians appointed by the MSB of CIC to conduct medical examinations of all applicants for permanent residence and some visitors who apply to enter Canada. A CIC medical officer, relying on the medical assessment done by a DMP as well as lab test and X-ray results, will determine whether the applicant is inadmissible on health grounds.

In the summer of 2001, CIC created the Medical Services Branch under a director general. The Branch was assigned responsibility for the development of operations and policies related to the medical program at CIC. It brought together existing expertise in the International Region, the Selection Branch and Departmental Delivery Network. MOF positions were upgraded, new MOFs were recruited and regular MOF meetings were introduced. MSB, jointly with the Internal Audit and Accountability Branch, developed an MCF for the DMP process in the fall of 2002 and field-tested it in the winter of 2003. The DMP MCF was approved by the departmental Audit and Evaluation Committee in January 2004. A high-level overview of the MCF is presented in appendix A. Other actions taken by MSB related to DMP included:

A management control framework represents a set of policies and procedures to ensure that results are achieved and program objectives are met.

The MCF presented in appendix A is consistent with frameworks promulgated by professional organizations involved in governance issues.

1. The conduct of annual audits at the DMP level (this is a self-assessment form completed by the medical practitioner);
2. The conduct of on-site audits (a CIC medical officer carries out on-site examinations);
3. Updating of the DMP handbook; and
4. The development of systems to capture DMP information.

In September 2004, at the time the audit was initiated, a new director general was appointed. During the audit, new senior managers were also appointed (in late 2004) and new directorates were created (in early 2005). The roles and responsibilities of key management positions were being redefined and plans for replacing the two systems containing medical information (the Immigration Medical System [IMS] and the Field Operations Support System [FOSS]) and the Global Case Management System were under way.

A risk analysis was conducted in the summer of 2004 in consultation with MSB, and both CIC senior management and the departmental Audit and Evaluation Committee agreed that it would be appropriate to focus the DMP internal audit activity, during fiscal year 2004–2005, on the MCF for the DMPs that fall under the direct responsibility of the Ottawa Region (North America).

2.0 Audit Objective and Scope

The objective of the audit of the DMP MCF is to assess it and to provide assurance on its implementation. The audit report provides recommendations for improvement, as appropriate.

The audit scope included all activities related to DMPs that are under the direct responsibility of the MSB Ottawa Region. This region covers North America, and consists of 302 DMPs who have conducted 139,352 medical assessments in 2004–2005. Appendix C provides a breakdown of the DMPs by residence and the immigration classes for the medical examinations.

3.0 Audit Approach

The audit was conducted in accordance with the Government of Canada’s Policy on Internal Audits, as well as auditing standards prescribed by the Institute of Internal Auditors. The audit utilized criteria based on the DMP MCF. Appendices A and B provide a summary of these criteria. The audit was conducted from September 2004 to March 2005.

The audit involved interviews with MSB staff, management, MOFs and other CIC management officials. It included reviewing documents to validate the information gathered during interviews, examining other documents on policies and practices related to the management of DMPs, and conducting a file sample of the on-site and annual audits of DMPs undertaken by MSB for the Ottawa Region.

4.0 Audit Reporting and Assessment

Our assessment of the DMP MCF uses a four-stage development scale (table 2) and this audit report provides management with an objective assessment of progress made on the DMP MCF, as well as making recommendations for continued progress.

Overall, MSB has made significant progress and is continuing to implement various elements of the DMP MCF. Table 3 shows our summary assessment of the DMP MCF, which must be read in the context of the environment in MSB presented under Introduction, section 1 of this report. Details are provided under Detailed Observations and Recommendations, section 5 of this report.

5.0 Detailed Observations and Recommendations

5.1 Governance

Audit criteria related to the governance structure of the DMP MCF included expectations with regard to MSB strategic direction, legislation and policies, and the DMP handbook. Our examination found that many expectations were met and well understood and that MSB was now looking at further refining and improving the governance over the DMP process.

We believe that senior management has made significant progress in developing the DMP governance structure. We note that management is committed to continuing to strengthen the governance structure by refining and clarifying roles, responsibilities and accountability as well as seeking additional resources to ensure MSB sustainability.

The existing components of the governance structure include the following:

• The mandate and mission of MSB, including the DMP process, are clearly communicated in the Integrated Business and Resource Planning document;
• There is an appropriate policy and legislative framework in place;
• Regular monitoring of DMP activities is carried out by MSB;
• Operations memorandums on changes to the immigration medical assessment process are issued, and MOF tools, such as IRPA instructions, guides, and links to references and health care in Canada are made available; and
• The DMP manual (2003) has been issued and is also available on the Web.

Monitoring systems, also an important element of governance, are discussed under section 5.4 (Performance Management) and section 5.5 (Operational Delivery) of this report.

Overall, we have assessed the governance element of the MCF as having the fundamentals in place.

5.2 Organizational Capability

Audit criteria related to the organizational capability element of the DMP MCF covered aspects such as organizational structure, responsibilities and accountabilities, authorities, roles and operational policy capacity.

At the time of our audit, we found that management positions were being replaced with new acting senior managers and that MSB restructuring of roles, responsibilities and accountabilities related to the DMP process was under way. The delivery of the DMP process was the subject of an internal study by one of the senior managers.

Our assessment of organizational capability was limited because of the restructuring taking place at the time of our audit. Given the early stage of the reorganization, no recommendation is made at this time.

5.3 Risk Management

We would expect that risk management for DMP includes the articulation of pertinent strategies and implementation of proven processes and practices, as well as the identification and assessment of country-specific health risks. CIC provides support to departmental operations in establishing formal risk management practices, and it developed the IRMF in December 2002 to add a systematic and rigorous approach.

The IRMF document provides information and guidance on risk management, where risk management should be applied, the limitations, roles and responsibilities for implementing and maintaining sound risk management processes, and examples of various tools and techniques. More specifically, the document “Health Canada Decision-Making Framework for Identifying, Assessing and Managing Health Risks” (August 2000) is an important resource for MSB in implementing risk management.

We expected to see formal risk management practices applied in the DMP process, including a risk profile, mitigation strategies, country-specific risks and reporting of important risks to senior MSB and CIC management. At the DMP process level, we did not find documented and articulated risk management practices or a risk methodology. We were informed that risk-related issues were discussed at staff meetings. We noted that the rate of inadmissibility was significantly under one percent but it was not clear whether this represented a low risk for the program and the Department or a significant risk and why. It is essential that the program articulate and communicate how this measure is linked to risk management and whether the observed level is acceptable from a risk perspective.

We found that six individuals from the 2003–2004 MSB management committee were provided with training on risk management.

One of the initiatives under way in the DMP process was the consolidation of existing processes and tools to remove redundancies and inconsistencies in the tombstone information of the DMPs in the current systems. This new system is to replace four different repositories containing the tombstone data of the DMPs. We noted that the multi-year plan for this system included a phase for an IME risk management plan, scheduled to be completed by the end of 2005. MSB has accumulated a significant amount of information on IMEs and DMPs that will allow MSB, when summarized, analysed and reported, to identify the risks for operational as well as management action.

We noted that MSB has made progress in implementing risk management practices in its operations. However, formal documentation through the use of such tools as risk information sheets was not yet implemented at the DMP process level. We did not find documented risk assessment strategies, proven processes and practices, and articulation of country-specific health risks. With a fully implemented formal risk management approach, MSB should be positioned to proactively manage the overall DMP process as opposed to reacting to individual DMP situations.

Recommendation 1

MSB should continue to implement risk management practices and assign responsibility to a manager for formalizing risk management practices within the DMP process. These risk management practices should include risk articulation and the documentation of country-specific risks, mitigation strategies and reporting on risk management.

Management Response

Agreed. MSB recognizes that the current quality assurance and risk management processes can be strengthened. This has to be done in coordination with an in-depth review and strengthening of the entire management accountability framework of the DMP program. However, due to the recent major staffing reorganization and current lack of personnel, our immediate priority has been to maintain our operational processing capacity.

MSB created a new directorate in 2005 (Program Management and Controls) and is in the process of identifying an EX-01 director. One of the tasks of the director will be to supervise the review and the development of a management framework for four programs, including the DMP management program. MSB has also recently created an HR structure dedicated to the management of the DMP program (PM-06 manager, PM-02 program assistant and CR-04 support). The Branch is currently involved in the selection process for the nomination of a manager with experience in the areas of risk management, quality assurance and utilization review management, who will be responsible for developing and formalizing a comprehensive quality assurance and risk management strategy for the DMP management program.

Once the manager is identified, a working group, including experienced MOFs, will be formed and tasked with risk identification and articulation, identification and documentation of country-specific risks, and the development of mitigation strategies. The manager will be responsible for not only reporting to the DG and the director of Program Management and Controls, but also for ensuring that organizational goals are clearly communicated to all MSB employees so that organizational commitment to this process is achieved.

5.4 Performance Management

Performance management includes having meaningful measures or indicators of performance, a means to capture both quantitative and qualitative performance information, and a system to track, monitor and report performance information at the operational delivery level. It also includes summarizing the performance information for reporting to management and executives and for use in accountability reporting. Equipped with quality performance information, management is better positioned to demonstrate due diligence, accountability, effectiveness and efficiency, and to take corrective action when warranted.

We found that MSB had captured a significant amount of performance information on the DMP appointment process, the IME process, the annual audit (self-evaluation form) and the on-site DMP audit activity. The information from the IME is captured in two key systems—IMS and FOSS. During our audit, technical enhancements for the DMP systems (IMS and FOSS) were frozen due to their eventual replacement by the Global Case Management System.

MSB has ad hoc databases to capture basic DMP information and the DMP audit activity. This makes it difficult for MSB to manage the activities of DMPs. These databases, along with the DMP hard copy files, were used at the operational level for such things as following up on new DMP appointments and on incomplete or questionable IME issues, and for planning on-site DMP audits. We did not find regular reporting on DMP performance or evidence that any formal procedures were in place to use the information in the ad hoc databases for DMP performance rating or reappointment decisions.

MSB initiated a project—the DMP information management system—to improve the capture of DMP information, At the time of our audit, this project was at the conceptual design stage. However, this system may have the potential to greatly increase the efficiency of processes and the reliability of information for performance management. In addition, MSB needs a formal strategy in place to cover all aspects of a DMP quality management system that would generate key performance information for monitoring and reporting purposes.

Recommendation 2

MSB should establish suitable performance measures and indicators for management and operational requirements with associated validation and reporting procedures.

Management Response

Agreed. The manager/program specialist of the DMP management program, in consultation with the directors of Program Management and Controls, Operations, and Strategy, Policy and Business Effectiveness, will be responsible for the development and implementation of a DMP integrated quality assurance program. The goals will be to enhance performance monitoring by making use of standards, indicators and criteria to better evaluate the DMP program and accountability. MSB will move toward an integrated quality assurance program by combining the once separate entities of quality assurance, risk management and utilization review management. The DMP manager/program manager will be responsible for reporting to senior management.

A statistician, engaged by the director of Strategy, Policy and Business Effectiveness, will provide statistical analysis of the data gathered and will ensure the validity and reliability of the measures, indicators and statistics.

5.5 Operational Delivery

DMP operational delivery consists of two processes: DMP appointment, and the medical examination and assessment process. Our examination consisted of documentation review and analysis, interviews, mapping DMP processes and testing.

For DMP appointments, we found that there is a systematic process in place to select qualified DMPs and to provide them with orientation and tools. We reviewed a sample of files for DMP appointments in 2004 and found adequate documentation to support the appointments. The DMP handbook states that DMP appointments are for a period of five years, yet we found that there was no formal process in place to review appointments after this period, nor did we find procedures or criteria for determining satisfactory performance.

The primary guidelines for operational delivery are the DMP handbook and the MOF guidelines. The DMP handbook was revised in 2003 and reflects the June 2002 IRPA. We found that there was only an informal system in place to track improvements for future updates due to the uncertainty over responsibility for the DMP handbook.

The MOF guidelines are dated 1992 and work on revisions was not completed due to a lack of resources. We found that the MOFs supplemented the MOF guidelines through other available professional resources. MSB should ensure that a systemic process is developed for the update of operational tools such as the MOF guidelines.

In terms of DMP monitoring and quality control, since 2002, the MSB has conducted more on-site DMP audits in the United States and Canada. We found that at least 75% of the DMPs have been subject to an on-site audit since the fall of 2002. In 2004, there were 114 DMP on-site audit visits undertaken, and plans were under way to continue with more DMP visits in 2005. These audits are part of the MSB quality assurance for the program. We note that this is a significant improvement over previous years.

Audits are performed by MSB medical doctors or, in some cases, by MSB nurses. While conducting on-site audits, MSB used the opportunity to provide DMPs with information and training on IRPA and other issues identified by the DMPs. MOFs complete a standard on-site audit guide for each DMP.

We compared the MSB on-site audit guide to a generic model for a medical office liaison visit, which had been developed during the preliminary survey phase based on what was in use by the United States and Australia. Based on that comparison, we concluded that the DMP on-site audit guide provided adequate coverage as a tool to monitor DMP performance. We also tested the completeness of the information provided on 35 of the DMP on-site audit forms for 2004 and found that they were generally complete, with only a few minor omissions in some cases.

In 2004, MSB also initiated an annual audit (self-administered evaluation) of all the DMPs who conducted the IMEs for the Ottawa regional medical office (RMO). At the time of our audit, MSB was conducting a thorough analysis of the responses.

We were informed that MSB has yet to develop a formal strategy to cover all aspects of the quality assurance management of DMPs, such as responsibilities, resourcing, and the reporting and analysis of results. Taking this next step should further contribute to the continuous improvement of DMP processes and the integrity of IMEs.

Recommendation 3

MSB should update the MOF guidelines.

Management Response

Agreed.

Recommendation 4

MSB should implement the processes for DMP reappointment.

Management Response

Agreed. The manager of the DMP management program, in consultation with the directors of Program Management and Controls and Operations, will establish a formal process for the review of DMP appointments and reappointments. The goals will be to develop appropriate data collection methods, redesign monitoring tools, test monitoring tools, select monitoring strategy, collect data, use the data collected to tabulate results, and interpret and use results in a timely and effective fashion. The following procedures will be formalized: the determination of geographic demand for DMP services, credentialing, and quality assurance and reporting activities.

6.0 Conclusion

Overall, MSB has made significant progress and is continuing to implement various elements of the DMP MCF. Fundamentals are in place regarding the governance and operational delivery elements, and improved articulation of risks and the reporting of performance are required. Progress is being made in implementing important organizational changes and MSB senior management has committed to continued efforts to implement the DMP MCF.

The following recommendations are made:

• That MSB continue implementing risk management practices and assign responsibility to a manager for formalizing risk management practices for the DMP process. These risk management practices should include risk articulation and the documentation of country-specific risks, mitigation strategies and reporting on risk management.
• That suitable performance measures and indicators for management and operational requirements be established, with associated validation and reporting procedures.
• That the MOF guidelines be updated and the processes for DMP reappointment implemented.

Appendix A: Management Control Framework

Designated Medical Practitioners (DMP) Process for the MCF

Text version: (DMP) Process for the MCF

Appendix B: Audit Criteria

1. Governance
   1. The key components of a governance regime, including the MSB policy framework, the DMP governance framework and the accountability framework, should be in place and clearly understood.
   2. MSB should monitor activities and performance with regard to DMP activity against organizational objectives and intended results.
2. Organizational Capacity
   1. The key components of organizational capacity, including a well-defined organizational structure, should be in place with clearly defined authorities and responsibilities and with sufficient policy capacity.
3. Operational Delivery
   1. Operational delivery should be supported by well-disseminated operational policies, directives and forms.
   2. The required qualified resources should be available for the conduct of operations at the corporate (MSB), local (RMO) and first-line (DMP) levels.
   3. The medical procedures and tests should be conducted by the DMPs according to CIC requirements, with adequate quality control over DMP activities.
4. Risk Management
   1. MSB and the MOF at the RMO should manage risks relative to the integrity of the process through pertinent strategies and proven processes and practices in order to minimize risk to public health and safety and excessive demand from prospective immigrants.
5. Performance Management
   1. The performance of DMPs should be monitored to assess their competence and integrity and the quality of their medical examinations.
   2. The medical assessment program should be evaluated periodically to ensure that objectives are being met.

Appendix C: Designated Medical Practitioners, Ottawa Region

Source data from MSB systems.

Appendix D: Management Action Plan