Audit of Immigration-Contribution Accountability Measurement System (iCAMS) - Phase 1

Final Report
Internal Audit and Accountability Branch
Citizenship and Immigration Canada
April 2007

Table of Contents

• 1. Introduction
• 2. Objectives
• 3. Audit Scope
• 4. Methodology
• 5. Background
• 6. iCAMS Challenges
• 7. Independent Review of iCAMS
• 8. Conclusions
• 9. Observations and Recommendations
  • 9.1. Processes and procedures in support of ongoing data integrity
  • 9.2 iCAMS 9.2 information used in decision making

1. Introduction

The Internal Audit and Accountability Branch completed a risk-based audit plan for 2005–2006 that identified the need for an audit of the Immigration–Contribution Accountability Measurement System (iCAMS) that focuses on the integrity of the information used for decision making.

Our audit will be conducted in two phases. Phase one reports on objectives identified in the next section. Phase two will look at the accuracy and completeness of the data and the extent to which the iCAMS reports are being used for monitoring purposes. We will also look at the reporting and performance framework related to the new settlement funding.

2. Objectives

The objectives of phase 1 of this audit are to assess:

• the adequacy of management processes and procedures in place to support ongoing data integrity; and
• the adequacy with which iCAMS information is used for decision making.

3. Audit Scope

The audit scope covered the information contained in iCAMS where the program is delivered by Citizenship and Immigration Canada (CIC). We examined the current processes and procedures in place to support the ongoing data integrity of the iCAMS production environment. In terms of the adequate use of iCAMS information for decision making, we looked at current planning and reporting materials.

Methodology

This audit of iCAMS included an examination of the management processes and procedures in place to support ongoing data integrity and a review of planning and reporting documents to determine the extent to which information contained in iCAMS was used in management decision making by the Integration and Refugees branches respectively.

The sources of criteria for this engagement included the Settlement Manual and iCAMS Security Requirements for Service Provider Organizations Manual.

The audit of management processes and procedures included three lines of enquiry: access controls, input controls and monitoring.  This portion of the audit involved documentation review, interviews and audit testing of control practices. A random sample of 42 contribution agreements signed with service provider organizations (SPOs) was selected and user access was tested to ensure access controls were functioning as intended. Furthermore, a review of documents was undertaken to ensure that adequate input controls were in place and that adequate monitoring of access and input was occurring.

The audit of the adequacy with which iCAMS information is used for decision making included two lines of enquiry: planning and reporting.  This portion of the audit involved document review, interviews and the examination of information contained in planning and reporting documents as they pertain to information derived from iCAMS.

The audit was conducted in accordance with the Government of Canada’s Policy on Internal Audit, as well as auditing standards prescribed by the Institute of Internal Auditors. Examination work was conducted in August and September 2006.

Background

CIC manages the delivery of settlement and resettlement services through four programs:

• Language Instruction for Newcomers to Canada (LINC);
• Immigration Settlement and Adaptation Program (ISAP);
• Resettlement Assistance Program (RAP); and
• Host Program.

The collective goal of these four programs is to support the settlement, adaptation and integration of newcomers to Canada by assisting immigrants and refugees in learning about life in Canada, accessing public services, acquiring basic language skills in one of the official languages, and becoming participating and contributing members of Canadian society.

CIC manages the delivery of these four programs in all provinces and territories, with the exception of Quebec (Canada-Quebec Accord, 1991), British Columbia (immigration agreement, 2004) and Manitoba (immigration agreement, 2003), through standardized federal contribution agreements with SPOs. The SPOs in turn, deliver services directly to permanent residents of Canada. In the province of Quebec, responsibility for the delivery of all settlement and resettlement services (LINC, ISAP, RAP and Host) belongs to the province while in B.C. and Manitoba, only resettlement services (RAP) continue to be managed by CIC.

In 1999, CIC launched the development of the Contribution Accountability Framework (CAF) aimed at ensuring the accountability of settlement and resettlement expenditures through the monitoring of service delivery and the evaluation of program effectiveness. The framework has five components:

• Performance measurement;
• Evaluation;
• Contribution agreement process;
• Management control framework; and
• Provincial/territorial accountability.

In support of the CAF, the Department developed iCAMS, an Internet-based database, to collect performance measurement data from SPOs delivering services on behalf of CIC. Using this system, SPOs report on services delivered as part of the four settlement and resettlement programs through direct online input. iCAMS is not a financial management nor an operational management system. Management has stated that iCAMS is expected to enable SPOs to provide consistent and reliable program data to CIC to determine the number of existing program recipients, the number of immigrant-servicing agencies, and the types of services available or delivered within a particular region and the local community. Management has also stated that this information will ultimately assist SPOs and CIC in providing better services to clients and increasing program effectiveness.

At present, the number of SPOs using iCAMS is 237. The Integration Branch has advised us that for the last two fiscal years, the number of clients served by the various programs in iCAMS was as follows.

Source: Integration Branch.
* These numbers represent unique RAP clients nationally.

Originally conceived in the late nineties, iCAMS development began in 2000. The project charter articulated iCAMS’ long-term goal as the creation of an electronic data collection system that allows program data from SPOs across Canada to be consistently and reliably collected for contribution management and evaluation purposes. This was to be accomplished by the creation of two information technology (IT) solutions making up iCAMS: an Internet (Web-based) application and a database interface for existing systems.

The iCAMS development was phased with the release of a module for each of the four contribution programs as follows:

• RAP, 2002;
• LINC, 2003;
• ISAP and Host, 2004.

Each module evolved through a pilot project phase involving a select number of SPOs to full release. At the time of our audit, all four modules for data capture were in full operation, the reporting module being released internally in August 2006.

iCAMS was originally designed to minimize excessive reporting requirements on SPOs by drawing select information from CIC corporate legacy systems. As the Department is in the process of a major IT initiative to replace these legacy systems and others with one large system called the Global Case Management System, this will have a significant impact on iCAMS design. That being said, the project is well aware of this issue and is working to ensure iCAMS will continue to function throughout this process.

Management estimates that from the initiation of the iCAMS project in 1998–1999 until 2005–2006, the costs reached $8,676,236 (Integration Branch: $4,816,878; Information Management and Technologies Branch (IMTB): $3,859,358). iCAMS funding for future years, (section removed), has been estimated at an additional $6,746,553. Of this, $5,408,654 would be derived from funds provided as part of the Canada-Ontario agreement negotiated by the federal and provincial government, and $1,337,899 would come from new monies requested for additional settlement funding (section removed).

6. iCAMS Challenges

Since its inception, iCAMS has been confronted with many challenges, which have had an impact on the development of the system. This section briefly describes the challenges the project has faced to date.

The development of iCAMS proceeded without permanent funding being dedicated to it. Rather, the funding for iCAMS was determined on an annual basis, leading to some uncertainty as the project was conceived as a multi-year project. Consequently, work carried out in any given year had to take into consideration that future development might not receive funding.

Prior to iCAMS, the Department attempted to implement a settlement information management system, but CIC was unable to implement it for many reasons, including lack of buy-in from staff and SPOs. Moreover, other regional systems had been created in the absence of a national one, the most notable of these being the Automated Reservation System (ARS) used in Ontario to manage the LINC program. As a result, resistance to iCAMS was high at the outset, with the Contribution Accountability Team (CAT) devoting a large amount of time to getting buy-in from various stakeholders, which had not been taken into account when iCAMS was conceived.

IMTB has been a key partner in the development of the system. Consequently, environmental factors affecting IMTB have also in turn affected the development of the system. These factors include, but are not limited to, downsizing, management changes and the replacement of CIC legacy systems included in the iCAMS design.

7. Independent Review of iCAMS

As a result of the issues discussed in the previous section, the Integration Branch engaged the services of an independent consultant in February 2006 to perform a review of iCAMS. This review sought to analyse iCAMS, perform a gap analysis and provide recommendations for the project to move forward.

The review conducted by the consultant found that management believed iCAMS provided the necessary volumetric information to support an accountability assessment but that qualitative information for outcomes evaluation was outside its scope. The consultant concurred with this assessment and found that while iCAMS provided quantitative outputs, not outcomes, it could provide inputs to program evaluation and assist with program management and policy development.

As a result, the review concluded that iCAMS was an appropriate tool for performance measurement in the CAF and a useful tool for program management and policy development, with some refinement. The review also concluded that iCAMS objectives were being met despite considerable resource constraints at national headquarters (NHQ).

8. Conclusions

As a result of our examination, we found that the iCAMS access and input controls were adequate. However, monitoring controls to ensure ongoing completeness and accuracy were not in place. We also found that iCAMS was used in limited instances to support decision making and that results were not reported. However, by granting access to the reporting module, management has taken steps to help this move forward.

9. Observations and Recommendations

9.1 Processes and procedures in support of ongoing data integrity

As part of our audit of processes and procedures that support ongoing data integrity, we examined three areas. These were access to the system, data input controls and monitoring of data input.

We expected that:

1. access to the system would be controlled and access procedures would be documented;
2. input controls would be designed into the system to ensure that the data were both accurate and complete; and
3. there would be an effective monitoring regime in place to ensure that the data were complete and accurate.

The following section describes our observations in each of these areas.

9.1.1 iCAMS access

As iCAMS is an Internet-based system, users may access the system wherever the Web is available. Given the sensitive nature of the information being reported, we expected there would be identification and authentication protocols in place to protect this information. The access protocol is documented in the iCAMS Security Requirements for Service Provider Organizations Manual. To protect the system, users are required to obtain a user ID and password to access the system. Without an ID, individuals may not enter or view information on iCAMS. The ID acts as a key to the system. We were advised that SPOs were not obliged to report in the same manner. Therefore, SPOs with multiple contribution agreements offering services at multiple locations may report each site individually or globally for the SPO as a whole.

As a part of our audit, we selected a random sample of 42 SPO contribution agreements and tested the access protocol (e.g., formal authorization and security clearance) for each SPO’s list of users. We found that there were 39 SPOs that had iCAMS users, for a total of 389 user accounts. One of the other three contribution agreements was for an SPO with two agreements selected among the sample of 42 and reported globally. For the remaining two contribution agreements, these two SPOs did not report through iCAMS and therefore had no users. Of the 389 users, 288 were found to have documents to demonstrate that they had obtained their account in compliance with the access protocol.

Access protocols for iCAMS have evolved over time and consequently account for this high level of non-compliance. Our testing indicates that those who did not comply obtained their accounts early on when consistent access protocols had not been established.  In addition, in performing our audit test, we noted some irregularities with respect to user IDs (e.g., inactive accounts, duplicate accounts and irregular formatted user codes) which we discussed with management. These issues are assessed in section 9.1.3 on iCAMS monitoring.

9.1.2 iCAMS data input

System design

The design of the system has been documented in various specification reports. Moreover, input controls have been designed into the system in accordance with the data definitions and data models developed by the project. The combination of these documents acts as a blueprint for the system by defining which information can be entered in a prescribed input area as well as the sequence of input. Using these documents, the system developers incorporated defined controls in the system coding. These include fields that only accept certain types of data (e.g., numbers in number fields) and mandatory fields. Industry standard controls like these field edits are the main input control for iCAMS, and they increase the likelihood that information entered by SPOs is accurate and complete.

One key data input control of note designed into the system is the validation process of the system. iCAMS, depending on the module, allows for individual (client-level) or aggregate reporting of SPO services delivered. In both situations, SPOs must enter valid client details. This is accomplished by validating the client details against CIC databases. If the client is not validated by CIC databases, then the SPO cannot enter service details related to the clients. Including this validation process prevents an SPO from reporting on services delivered to non-existent permanent residents. We were informed that the current validation rate is 96%.

In cases where client information is entered into iCAMS and certain aspects of the data cannot be verified (e.g., immigration number, temporary resident or minister’s permit number), the record is deemed “non-validated” and although the “tombstone data” are recorded by iCAMS, the SPO is unable to record any services against that client. While there is an automated “cull” to ensure that the non-validating clients do not represent duplicates of validated clients, there is currently no follow-up mechanism for the true non-validating clients to understand what they are, what made them invalidated and what follow-up, if any would be required. The lack of follow-up mechanisms for non-validating clients has the effect of increasing the risk of incomplete data in iCAMS.

In 2003, the CAT, in cooperation with IT staff, examined the issue of non-validation to obtain a better understanding of this area. This review determined that a large number of non-validations were occurring due to issues involving validation of a client’s name. As a result, the CAT review recommended that the validation be changed and that the number of client details that are compared be decreased. Currently, SPOs continue to enter the client’s name, but validation is sought only for the client’s date of birth and CIC identifying number.  We are concerned that without determining the cause of non-validation, some valid data may not be entered into the system. The CAT informed us that this was part of the monitoring responsibilities of local offices. It is discussed in more detail in Section 9.1.3 on iCAMS monitoring.

In the Ontario Region, LINC SPOs report this information through a regionally developed system that predates iCAMS called the Automated Reservation System. The ARS is developed and maintained by an Ontario SPO. Information captured in ARS is transmitted through an ARS/iCAMS interface which compares iCAMS data elements to comparable data in ARS. Prior to transferring the data to the iCAMS database, three key steps are taken to ensure the accuracy of the information. The steps are:

• Examination of ARS data – checking the ARS database daily and ensuring that the data are logical and accurate by running several Structured Query Language (SQL) queries. The queries check for logical client tracking events and exceptions to the ARS business rules.
• Generating data transfer – once the examination of ARS data is complete, the transfer of data begins by using a customized “tool” (ACCESS software) created by the ARS team to generate the data set needed for the specific months requested.
• Final examination of data – as the ARS data are transferred to the iCAMS database, they are double-checked for accuracy against the iCAMS business rules. During this process, SPOs may be notified by e-mail or telephone regarding any errors in the data. Many files are corrected and run through the transfer tool for examination a final time. If client records do not pass validation at this final examination point, they are not included in the transfer of the data set to the iCAMS database.

These steps are an additional control to ensure that information transferred from ARS to LINC is accurate and complete.

Training

The initial roll-out of the different modules was accompanied by the large training efforts of the CAT, which act as another control to ensure accurate and complete information. This included developing training manuals and user manuals and delivering training courses to SPO users across the country. This larger than normal training effort was done to ensure a consistent interpretation of data entry and a good understanding of the system. This also helped with stakeholder buy-in.

Any additional training would be administered on an “as requested” basis. Industry best practices consider the “train-the-trainer” model a valid way to disseminate training materials to a large audience. Delivering these courses was a large task that has resulted in the CAT considering alternative formats for training to move forward (e.g., CD self-learning and online training). However, it was noted during our audit that iCAMS users had online access to training materials while accessing the system.

In the case of CIC, however, the approach has been modified somewhat, given the diverse population. Typically, a train-the-trainer approach is structured with specific training responsibilities assigned to designated field trainers. These individuals are responsible for keeping the “users” current and training new users as they come on board. In the case of iCAMS, the approach is less structured and follows a “train-the-successor” model. In this model, individuals could be called upon to train their successors at any time in the future.

Based on our interviews, it appears that SPOs have not always trained the successor, and in some cases where they have, errors have been passed from one employee to the next. We are concerned that the initial control established by the initial wave of training may no longer be in effect as the system has been in operation for some time and has gone through system updates. Furthermore, training materials may not remain current. However, by providing online access to these materials, management has established a compensatory control that will ensure that users are informed about any system updates.

9.1.3 iCAMS monitoring

Local office monitoring of SPO data has not been occurring as intended. The Settlement Manual produced by the Integration Branch sets out the expectation that settlement officers will be responsible for the monitoring of SPOs’ iCAMS use as they deal directly with the SPOs. This includes data input to ensure it is being done completely and accurately.

In this context, local offices would use reports generated by iCAMS to verify the accuracy and timeliness of information reported by the SPOs in iCAMS. Initially, these reports were provided to the regions on a quarterly basis to allow local offices to monitor SPO reporting, with the last set provided to the regions in 2004. These reports have been adjusted to a new format and include details about non-validation. However, the development of these new reports has been delayed and regional access to the reporting module within the Department was only granted in August 2006, with access to be extended to SPOs later in the year. We noted that sample reports provided to the audit team contained details on the non-validation of information by SPOs in the reporting period.

Since management reports are a key source of information on performance for local offices, we are concerned that they were not provided this information in order to fulfil their monitoring requirements. Without this monitoring, a key control over the integrity of data in iCAMS was absent and we are concerned that this increases the risk of inaccurate or incomplete data during the period that access to this information was not granted. Moreover, as access has only just been granted, it is not yet clear how effective the monitoring procedures are. Providing access to this information to stakeholders will allow them to see its value and therefore improve their buy-in.

In the interim, the CAT carried out data accuracy reviews of all four programs and concluded that there were no problems in the iCAMS reporting mechanisms, and that they accurately reflected the data entered by SPOs. Furthermore, two LINC ARS data verification tests were undertaken by the CAT to verify the integrity and accuracy of Ontario LINC data transmitted through the ARS/iCAMS interface. These reviews found that 96% of all client records in ARS were successfully validated in iCAMS. No further reviews of data integrity have been planned.

Moreover, in our sample of 42 SPO contribution agreements taken as part of our audit of user access, we found that 30% (116 of 389) of accounts were never used to access the system. We also found that 37% (68 of 186) of the accounts that had accessed the system at some point had not done so within the past six months. We were advised that the system deactivated a user account after 180 days of inactivity and that the accounts of former staff were not deleted. Instead, they were deactivated to allow them to be permanent records of users. However, we were informed by staff at NHQ that no monitoring of user accounts is undertaken at this time. Earlier, we noted some irregularities regarding the user accounts and now note that if monitoring were carried out on users, many of these issues would likely have been caught.

We also note that while maintaining a list of users allows for accountability, no monitoring of this information has been done and we are unaware of any plans to do so in the future. Furthermore, if access to the system requires a user account to protect the system, maintaining inactive and unused accounts increases the potential risk to the system, as these are potential access points which could be compromised. However, given other system-designed controls and the nature of the information in the system, this risk is low. In addition, maintaining these accounts implies that the list of users will continue to grow even if the number of SPOs does not. This increases the need to monitor iCAMS user accounts.

The iCAMS Security Requirements for Service Provider Organizations Manual requires SPOs to update user information. This audit found that no monitoring of these accounts, which is part of the overall iCAMS monitoring that local offices are expected to do, is undertaken at this time. This is seen as a weakness in the process, although it is considered to be a low risk. While the Settlement Manual clearly indicates that monitoring is the responsibility of the local offices, NHQ should ensure that this monitoring is occurring.

In the spring of 2006, the Department launched a new organizational structure to better reflect the reality of its new role and mandate.  As part of this new structure, the Department created the Operational Management and Coordination (OMC) Branch to support the objective of coherent and consistent program delivery throughout Canada and abroad. The transition to the new organizational structure was a phased process, completed later in 2006. This resulted in program monitoring functions, which previously resided in policy development branches, moving to the newly created OMC, including those that formerly resided in the Integration and Refugees branches.

Once an adequate monitoring regime is in place and functioning as intended, we would expect that the results from the monitoring of SPOs would be rolled up on a periodic basis. This way, management could gain insight into the overall effectiveness of the controls. This could also serve as a guide for management to determine where to focus their attention for further investigation or efforts toward program redesign.

Recommendation 1
We recommend that the OMC Branch ensure that local offices not only utilize the recently released iCAMS reports to monitor SPO data entry into iCAMS, but monitor SPO iCAMS use as articulated in section 52.13 of the Settlement Manual.

Management response

OMC agrees to ensure that local offices incorporate information from iCAMS reports into their regular monitoring activities and into their decision-making process for continued funding. iCAMS reports have the ability to provide counsellors with quantitative data on client services and characteristics that can be compared against the objectives agreed to by the participating parties within the contribution agreement. The use of iCAMS reports benefits the SPO by decreasing their reporting burdens.

OMC agrees in principle that there is a need to monitor SPO data entry into iCAMS to maintain data integrity. The development of monitoring tools and the enhancement of procedures will facilitate this activity, but they must take into account local CIC capacity and any additional burdens that may be placed on staff.

9.2 iCAMS information used in decision making

iCAMS is expected to enable SPOs to provide consistent and reliable program data to CIC on services delivered. Management has stated that this information will ultimately assist SPOs and CIC in providing better services to clients and increasing program effectiveness. As part of our audit of the adequacy with which iCAMS information is used for decision making, we examined two areas: reporting on performance and the use of information in planning.

We expected to find that information contained in iCAMS was used for reporting on performance and for planning purposes and that it therefore supported management decision making. As a part of this audit, we interviewed staff and reviewed departmental and branch reporting and planning documents.

The following sections describe our observations under each of these areas.

9.2.1 Reporting on performance

We found that at the departmental level, no information from iCAMS was reported for the 2004–2005 and 2005–2006 periods in the Departmental Performance Report (DPR), the Department’s main performance-reporting document. As noted earlier in this report, iCAMS experienced a number of challenges, such as uncertain funding and the unsuccessful implementation of a prior settlement information system. These factors resulted in delays in the implementation time frame. As well, additional concerns over data accuracy and completeness meant that the Department was unable to provide complete and accurate information from iCAMS in the DPR. With these issues now resolved, it is management’s intention to provide information on the integration programs starting in the next reporting period.

9.2.2 Use of information in planning

Just as for reporting on performance, we found that at the departmental level, no information from iCAMS was reported over 2004–2005 and 2005–2006 in the Department’s Report on Plans and Priorities. At the branch level, neither the Refugees Branch nor the Integration Branch had included this information in their business plan. In terms of planning, it may not be realistic to expect this information to be included in a planning report, but given the objective of the system, at a minimum, we expected that iCAMS information would be considered when developing the plan. However, management informed us that they will include this information in planning documents in the future.

In discussions with management, we were advised that iCAMS was used in limited situations as it was felt that not enough time had elapsed for a sufficient amount of data to be captured, and that the Integration Branch wanted to ensure that the data were correct prior to their release. We were also advised that this partly explained the delay in releasing iCAMS reports. All iCAMS data capture modules have been in production and capturing data since 2004. Our examination found that while iCAMS information was indeed used for planning purposes in some limited situations, access to it was restricted and consequently, it was not as widely used as we would have expected. As the system has been in operation for quite some time, we would have expected that the Integration and Refugees branches would be more familiar with the information available in iCAMS and therefore able to utilize it more effectively. We would expect that with the availability of the reporting module and increased access to reports, the use of iCAMS information would increase.

Recommendation 2
We recommend that the Integration and Refugees branches incorporate more iCAMS information in their planning and reporting in order to better support decision making and demonstrate accountability for the four contribution programs.

Management response: Integration

The Integration Branch will continue to incorporate iCAMS information in policy development, analysis and evaluations.

Management response: Refugees

The Refugees Branch commits to incorporating more iCAMS information in its planning and reporting process in order to support decision making and demonstrate accountability for the RAP.

The Refugees Branch will use iCAMS information in the development of client-centred approaches to meet the integration needs of refugees.

• Data from iCAMS will be analysed with the aim of improving the national monitoring of refugee needs and outcomes.
• Gender-based analysis will be performed on resettlement programs using iCAMS data.

Information from iCAMS will be used to help evaluate whether RAP is meeting the program objectives.

Appendix A: Management Action Plan