Audit of The Administration of Settlement Programs

3.0 Observations and Recommendations

3.1 Governance

The audit examined the adequacy of three areas of the governance framework in place over settlement programs. These are governance and strategic direction, accountability and risk management.

Overall, we found that the governance framework and processes in place for settlement programs were appropriate and operating as intended. We noted some opportunities for improvement that could strengthen the framework.

Specifically, the audit noted the following:

  • The governance structures in place provide for appropriate representation, consultations, engagement and communications to achieve objectives, though some clarification of roles and authorities is needed;
  • The protocols for escalation of issues were well understood and there was significant evidence of stakeholder engagement;
  • The key committees (the Contribution Management Steering Committee [CMSC], the Settlement and Resettlement Management Committee [SRMC] and the Settlement Management Committee of Ontario [SMCO] in the Ontario Region) were viewed positively by interviewees;
  • CIC has documented the overall framework and strategy, including governance, roles and responsibilities, performance measurement blueprint and an evaluation strategy in May 2008 as part of its approach to modernize the delivery of the settlement program; and
  • While a new framework has been developed for the new program, a formal overall plan to implement the framework with explicit accountabilities, deliverables and time lines is missing.

3.1.1 Governance and Strategic Direction

In assessing governance and strategic direction, we expected to find that processes were in place to develop plans that align with departmental objectives, and that communication protocols existed for both internal and external stakeholders. As part of the planning activity, given the substantial increase in resources in recent years, we also expected that the Department would have a reasonable indication of the “baseline” in terms of newcomer settlement needs.

We found that the settlement programs were aligned with departmental strategic outcome 3 in the 2009–2010 Report on Plans and Priorities (i.e., successful integration of newcomers into society and promotion of Canadian citizenship). There is a business planning process within the Department that requires integrated business and resource plans at the branch, sectorial/regional and departmental level.

CIC has established various mechanisms for consulting with provincial and territorial partners, with SPOs and certain umbrella organizations that represent them, and with TBS and other federal partners. Furthermore, the provinces and territories are consulted prior to departmental decisions on funding allocations to ensure transparency and accountability. We found that CIC staff were aware of issue escalation procedures and exception protocols. Communications protocols exist through committee memberships and more formally through OMC at the departmental level. In addition, communications protocols exist for external stakeholders.

We have noted issues with respect to two key areas in the context of planning as discussed below.

Baseline on newcomer needs and demand

Funding allocations to provinces and territories are based on consultations and include consideration of various factors, as outlined in section 1.1.1, the most significant factors being parity among provinces and territories and immigrant intake. While the Department has developed a program logic model that describes and links the key activities relating to newcomer settlement, we noted that the Department had not sufficiently developed a “baseline” as to the level of need and demand in regard to the settlement of newcomers, either regionally or nationally. What we mean by a baseline is an estimate, based on available evidence, of the funds that would be required to reasonably meet the basic settlement needs of a newcomer.

A clear baseline would assist in the determination of the adequacy of investments being made with respect to settlement programs across the country in order to achieve the desired outcomes.

Recommendation 1

The Department should initiate the development of a baseline for needs and demand with respect to newcomer settlement.

Management Response

The Department agrees that setting a baseline for the needs of newcomers is appropriate.

As part of the implementation of the Modernized Approach to Settlement Services, the Department is developing and testing new means of assessing and planning for the settlement needs of newcomers once they have landed in Canada. The Department also intends to pilot a needs assessment tool in 2010–2011. This tool will focus on family and individual settlement planning in order to better define CIC outcome expectations, increase potential uptake of CIC settlement services, and better empower newcomers to identify and meet their settlement goals. The pilot, once complete, is expected to provide valuable information on the level of demand for settlement services in each region, including a potential baseline for newcomer needs, and assist the Department in its planning and priority-setting activities, including determining funding allocations.

The time lines for this initiative are:

  • Development of the needs assessment tool, by September 2010
  • Testing the new tool, by December 2010
  • Rolling out the new tool across the regions, by April 2011

Settlement funding outside Quebec is a fixed envelope (currently set at $650 million in 2010–2011). Over the past few years, parity among various provinces and territories has been the driving force in developing the current settlement allocation. As outlined in section 1.1.1 of the audit report, the current formula takes into account the higher needs of refugees (who are the biggest users of settlement services), but does not distinguish between the needs of newcomers and those of the family and federal skilled worker classes. The current formula, which was approved by Cabinet, does take into account the need to provide some baseline funding for smaller provinces by adding a capacity-building amount.

CIC is in the process of implementing a new module that could assist in developing a baseline on newcomer needs by providing financial data and cost unit. The Financial Management Branch will support the Integration Branch with the costing aspect of the needs assessment initiative with a view to providing an estimate of the funds that would be required to reasonably meet the basic settlement needs of a newcomer by December 2010.

As part of the settlement review, CIC will review the current settlement allocation formula and will take this recommendation into account.

Implementation of the new Accountability, Risk and Audit Framework

CIC developed a new Accountability, Risk and Audit Framework (ARAF) for the settlement program in May 2008, and this document was submitted to the Treasury Board as part of the approval of new terms and conditions and CIC’s commitment to modernizing program delivery. The new framework puts forward high expectations (e.g., new focused program initiatives, administrative simplification, requirements with respect to the new TBS Policy on Transfer Payments, performance measurement strategy, etc.) and it is envisioned to be implemented over time. Implementation has already begun on several initiatives and some of the elements of this framework are included in the work plans of the CMSC and SRMC.

However, there were two noteworthy gaps from a governance perspective, as follows:

  • There is no formal documented overall plan for the implementation of this new framework identifying the key steps, specific deliverables, related accountability and time frames; and
  • While the new framework suggests that the Integration Branch is responsible for ensuring that there is alignment of departmental settlement programs with strategic and operational plans and priorities, this review mechanism has not yet been implemented.

A documented overall implementation plan for the new ARAF would help ensure that the commitments therein are achieved in a complete and timely manner. Without the contemplated departmental review of the alignment of settlement programs with strategic and operational plans and priorities, the assurance with regard to this alignment of plans with departmental objectives is diminished.

Recommendation 2

The Department should formally establish an overall implementation plan for the 2008 ARAF with key steps, explicit accountabilities, specific deliverables and time lines.

Management Response

The Department agrees with the recommendation.

The Department has developed an overall implementation plan for the Modernized Approach to the Settlement Program that included major elements of the ARAF. This plan was first tabled at the Departmental Management Committee meeting in October 2008, and subsequent updates on the implementation plan were presented to the CMSC in May 2009 and December 2009.

A formal ARAF implementation plan setting out clear accountabilities, roles and responsibilities, deliverables and time lines is being developed and once completed, will reflect the implementation of new settlement themes replacing the former LINC, ISAP and Host brands.
It is expected to be completed in June 2010.

The Department’s Settlement, Resettlement and Multiculturalism Management Committee (SRMMC)—a group comprising directors from the relevant business lines implicated in the management of these programs—will be updated every month on the progress of the implementation plan, as will the newly formed Business Operations Committee (BOC) when  required.

Recommendation 3

The Department should implement a process to review the strategic and operational plans relating to settlement programs and assess their alignment with departmental priorities and results for Canadians.

Management Response

The Department agrees with the recommendation.

The Department will be adopting a new planning process for the settlement program. The proposed strategic plan for settlement programs, which will align with departmental priorities, will be developed by the SPP Sector and tabled for approval at an Executive Committee (ExCom) meeting and will set out plans and priorities regarding settlement outcomes and associated investments. It will be tabled in October 2010.

The related operational plan to implement the above strategic plan will be developed by the Operations Sector and submitted to the BOC for approval in December 2010.

Using the Grants and Contributions Dashboard tool, the BOC will also monitor on an ongoing basis the progress made toward the plans, and bring any necessary corrective measures.

This process will allow the Department to identify gaps and to take measures to address them so that comparable outcomes are achieved across Canada.

Changes are being introduced to iCAMS to revise data collection and respond to the need to amalgamate all data on the new settlement themes into one source (e.g., Enhanced Language Training and the Foreign Credentials Referral Office’s tracking of overseas orientation graduates) and address identified gaps in reporting. These important changes to iCAMS will standardize the reporting on activities input by SPOs, thus allowing CIC to report on comparable outcomes across the regions and implement a renewed and expansive reporting structure.

Concurrently, the SRMMC has been facilitating the harmonization and consistency in program implementation. This forum has also provided and will continue to provide oversight and guidance on priorities and gaps in settlement services from both an NHQ and a regional perspective, and serving as a conduit for funnelling information to the BOC and, ultimately, ExCom. This group will also be used strategically to validate settlement planning assumptions and the overall approach to planning. Under the auspices of the SRMMC, the Department also established a Call for Proposals (CFP) Working Group in late 2009–2010. This serves as a forum to discuss and share information on best practices, process improvements, changes to requirements and other issues in managing a call for proposals cycle at both the NHQ and regional levels. The CFP process is the principal vehicle for the dissemination of funds to support the delivery of the settlement, resettlement and multiculturalism programs.

3.1.2 Accountability

We expected to find that processes existed to ensure that accountabilities and responsibilities were clearly delegated and communicated and adequately discharged, and that a clear organizational structure was established and documented.

Our review found that overall, CIC had defined the accountabilities and responsibilities through the functional guidance model, the ARAF, the mandates of various committees, the agreements with the provinces and territories and other guidelines, including manuals. Delegation instruments and work descriptions define accountabilities at the individual position level. We also found that clear organizational structures were in place to manage these programs.

A summary of the various committees and organizational units is provided in Appendix D. The committee structures are appropriate, their membership ensures appropriate representation from NHQ and the CIC regions and local offices, and the committees are linked to the departmental management framework and decision-making bodies. We found that the governance structures allowed for the engagement of both internal and external stakeholders, directing decision making to the appropriate level of authority and addressing the need for communications and tools to enable effective program delivery. They also provide a vehicle to ensure accountability for program resources. Several interviewees at both NHQ and in the regions indicated that the CMSC had a significant positive influence on the administration of settlement programs. Our review of the minutes of key committees such as the CMSC, SRMC and SMCO, as well as the ExCom records of decisions, confirms that relevant issues are discussed and direction given with regard to areas of importance.

The overall accountability with regard to CIC settlement programs at the time of our audit was understood to be with the Associate ADM of Operations, who co-chairs the CMSC and is also a member of ExCom. The CMSC provides leadership in the management of contributions. The role of the OMC Branch is also defined in the centralized functional guidance model introduced in 2006, while the departmental delegation instrument defines the delegated authorities.

At the time of our audit, the Department was creating the new Business Operations Committee to oversee the CIC operational program delivery. It was also creating the Settlement, Resettlement and Multiculturalism Management Committee, which would be responsible for the oversight of all grants and contributions (G&C) programs within the Department and would report to the BOC.

Our analysis has identified opportunities for improvement in the following areas:

  • Decision making and approval authority for committees;
  • Clarification of roles and responsibilities; and
  • Accountability of the Board of Directors of SPOs.

Additional details are provided below.

Decision making and approval authority for committees

Various committees were established by CIC to improve governance by providing oversight and facilitating communications. However, they impose significant demands on operations in terms of the need for information, approvals, etc. While the committees have formal mandates (draft or approved), these mandates do not clearly state whether and to what extent these committees have approval, recommendation or decision-making authority. This is particularly important given that resources may or may not always reside with the line manager (e.g., for information system enhancements such as iCAMS, responsibilities are distributed among three separate NHQ branches, creating challenges in priority setting and decision making).

Some of the individuals we interviewed indicated that there is a risk with respect to the efficiency of the decision-making process if issues and decisions do not get tabled at the right level (the right committee) and the appropriate level of discussion does not take place. In addition, the evolving nature of the programs and organizations creates the need for periodic updates of roles and responsibilities to enhance governance and accountability.

Clarification of roles and responsibilities

The roles and responsibilities of the Integration Branch, the OMC Branch and the regions, as they pertain to settlement programs, are defined in the new ARAF developed in May 2008. However, they need to be reviewed and updated based on the operational experience to date, as both the organizational structure and the programs are rapidly evolving. This is particularly important as several interviewees reported that there were grey areas between program policy and the operational implementation policy.

Accountability of the Board of Directors of SPOs

CIC delivers its settlement programs through SPOs. Under the modernized approach, with increased funding and scope of services to improve immigrant outcomes, the average size of an individual agreement is likely to increase and the number of SPOs may also increase and become more diverse. In line with this, CIC will also be playing a key role in strengthening the capacity of some of the less experienced SPOs. Generally, the SPOs are headed by an executive director and oversight within these organizations is provided by a board of directors, which often includes community volunteers. The accountability of board members in the governance process is very important, especially in terms of an independent effective oversight challenge and a due diligence function. Our discussions with CIC regional officials suggest that CIC should be taking steps to emphasize the accountability of the board of directors of SPOs by ensuring that they are familiar with CIC funding arrangements and expectations so that they may properly exercise their due diligence and oversight role. It is noted that CIC does engage the board of directors if significant issues arise.

If the accountability of the board of directors is not reinforced, there is an increased risk that governance of some of the SPOs may not meet CIC’s expectations of good governance or that CIC monitoring of an SPO may need to be stepped up. Reinforcing their accountability puts CIC in a stronger position to hold them accountable if significant issues arise.

Recommendation 4

The Department should ensure that accountabilities and responsibilities for settlement programs are strengthened by:

  • clarifying the decision-making and approval authorities of the committees associated with settlement programs;
  • updating the roles and responsibilities of the OMC Branch, the Integration Branch, the Financial Management Branch and the regions based on operational experience; and
  • taking steps to reinforce the accountability of the board of directors of SPOs with regard to CIC funding.
Management Response

The Department agrees with the recommendation.

The terms of reference for both the SRMMC and the BOC were recently approved. The proposed Program Management Continuum, which clarifies the roles and responsibilities of SPP, the OMC Branch and the Chief Financial Officer (CFO) regarding settlement programs, will be presented to ExCom for approval in June 2010.

The recent in-depth revision and significant rewriting of most of the settlement manual provides guidance and tools to staff both on the management of the settlement and resettlement programs,roles and responsibilities and on the management of contribution agreements generally. The revised manual will be released in July 2010.

The Department will also develop guidelines covering all financial aspects of the contribution agreements. These guidelines will clearly define the roles and responsibilities of the Financial Management Branch, the financial and settlement officers managing the financial aspects of the contribution agreements (both at NHQ and in the regions) and the recipients. The guidelines will be issued by March 2011.

There will be two main components to CIC’s planned action, education and process enhancements:

  • The CIC Ontario Region, the largest domestic region with settlement responsibilities, is developing generic board governance training for its settlement staff. This will help officers to identify the various board structures and signs of board competence, and equip staff to assess and determine the risk, capacity and approach associated with a particular organization’s board. Externally, the Ontario Region is also offering orientation and board governance training for CIC-funded organizations as a proactive measure to increase the effectiveness and involvement of the board and to raise every organization’s awareness of the scope of their responsibilities and the funders’ expectations. This model will be developed by September 2010 and will apply to other regions.
  • From a process perspective, to ensure procedural fairness, the Department will actively engage the senior management of SPOs and their board of directors upon signing an agreement. The goal will be to outline the organization’s obligations and responsibilities (through a statement of expectations) under the contribution agreement. CIC will advise the board of any findings resulting from a financial audit and the monitoring results. The expected date is March 2011. This will ensure that the board is aware of and accountable for addressing performance issues.

The Department will also pilot an approach whereby senior members of the board of directors (for example, the chairman or the treasurer) will be made mandatory signatories to any CA.

3.1.3 Risk Management

In examining risk management, we expected to find that processes were in place to identify, assess, mitigate and monitor risks; that management appropriately communicated risk and risk management strategies to key stakeholders; and that planning and resource allocation considered risk information.

We found that, while an overall risk assessment on settlement programs had been performed, the accountability for the regular monitoring and reporting of these risks had not been defined. Further, the risk assessment methodology for SPOs in place at the time of our audit was under revision in order to reflect the realities of the current programs and the new programming approach.

In May 2008, CIC management identified seven program-level risk areas and conducted an assessment of them and articulated the mitigation strategies. The risk areas are relatively broad. One area (performance measurement) was assessed as high risk while the remaining risk areas ranged from significant to moderate. We found that this risk information was communicated to key stakeholders, including the TBS. However, it was not clear from our interviews who at CIC was accountable for regularly monitoring the risks and reporting the results of this work. Several interviewees not only indicated they were not certain how risks relating to settlement programs were monitored, but that they were uncertain how the risk management framework for the settlement programs operated.

We did note that CIC management was aware of the risk relating to lapsed program funds and had implemented additional monitoring requirements to manage this risk.

At the operational level, CIC has a methodology in place to conduct risk assessment of contribution agreements with SPOs. This methodology has been in place since 2004 and was under revision at the time of our audit. Management advised us that the revision sought to update the risk assessment methodology and reflect the realities of the current programs and the new programming approach. Discussions are taking place at various committee meetings to perform a front-end assessment of SPOs and establish a prequalification process to better manage risks at the SPO level. The OMC Branch is currently developing a risk management framework for contribution programs. The proposed dashboard reporting on settlement is also intended to include information on progress with respect to monitoring and the level of risk at the SPO level.

It is too early to assess whether planning and resource allocation considers risk information adequately. The allocation of program funds is primarily based on a formula subject to consultations, although consideration is given to capacity building. For operating funds, a business case has to be submitted if incremental resources are required and the business case receives close scrutiny.

Our main concern with regard to the lack of formal accountability for risk management of the settlement programs is that the risks at the program level may not be consistently understood and adequately monitored on an ongoing basis. At the SPO level, there is a potential for the resulting risk assessments to not accurately reflect the changed operating environment and consequently, the Department may not be establishing a monitoring plan that is either realistic, in terms of its capabilities, or cost effective. This is discussed in more detail in section 3.3.

Recommendation 5

The Department should establish clear accountability for monitoring risks associated with settlement programs and include the reporting of this risk-related information in the dashboard reporting.

Management Response

The Department agrees with the recommendation.

The Department is currently building a broader risk assessment strategy as part of CIC’s corporate plan. The risks associated with the settlement program will be incorporated into this departmental strategy and, once completed, will be presented to the Management Accountability Committee and ExCom.

For non-financial risks, the Department recently established a risk and monitoring strategy that is now being used by the OMC Branch at NHQ. This strategy and framework, which take into account elements pertaining both to the SPO and to the agreement itself (e.g., dollar value), facilitate the assessment of risk for all agreements and the determination of a risk rating that indicates the frequency and type of monitoring which program officers should conduct during the life cycle of the agreement. The framework also allows regular reporting of risk information as part of the dashboard reporting. While every agreement is currently subject to at least one activity monitor, upon full implementation of this risk and monitoring framework, the Department will be better able to systematically monitor and further improve the management of contribution agreements.

With respect to financial risks, the Department will develop a risk assessment tool, which will be used to determine the frequency and type of reporting for recipients, and the level of financial monitoring and audit activities on the part of the Department. This tool will be used consistently in each region and at headquarters, for all programs. The Financial Management Branch will coordinate and monitor the audit schedule. The Department will review the G&C dashboard to ensure it provides relevant and strategic risk information on the financial aspects of the G&C. These initiatives will be implemented by March 2011.

3.2 Performance Measurement

The audit assessed the adequacy of the performance measurement framework in place for settlement programs. Specifically, we expected to find that processes existed to ensure that planned results and outcomes were linked to organizational objectives; that management monitored and reported on performance and considered performance information in decision making; and that controls were in place to ensure that performance measures were complete and accurate.

Performance measurement is a work in progress driven by the need to report at the outcome level, the need for key information at the management level, and the need for evidence-based information to guide the development of new programming initiatives. The audit found that while  information at the output level was available, CIC was in the early stages in terms of overall performance measurement of settlement programs at the outcome level. As a result, the performance measurement area only partially met our expectations.

Various information at the output level is captured in departmental information systems with regard to the existing programs (LINC, ISAP and Host) delivered by CIC. Our audit found that a significant number of measures articulated in the May 2008 TBS submission that included the new terms and conditions for the revised settlement program were not yet captured and these related primarily to outcomes. We did note that at the local level, SPOs may be reporting to some extent on these aspects, but this information remains at that level (see section 3.3). At the time of our audit, the Department was in the process of developing dashboard reporting at the program level which would include quarterly reporting on financial and output information. In addition, a performance management plan was presented to the CMSC in the spring of 2009 and the goal was to have a settlement national performance plan by 2011 and a refugee performance measurement framework by 2010. A significant number of resources from new settlement funding were allocated for research and evaluation activities, and some of the settlement programs were  undergoing evaluation at the time of our audit.

With regard to internal controls that are in place to ensure that the information is complete, we examined the controls over iCAMS data entry by comparing the number of SPOs who should report in iCAMS to those who were actually reporting. We found that approximately 17% of SPOs who should report in iCAMS were not doing so. Our discussions with OMC staff indicated that, while they review some data and provide administrative support to users, the primary responsibility for ensuring the accuracy and completeness of iCAMS data remained with the local CICs. If there are insufficient controls in place to ensure that the data are both accurate and complete, there is a risk that management may not have complete information.

The implementation process with regard to the performance measurement strategy and systems is currently under way and in our opinion, there remain significant risks and challenges, including data availability, costs associated with data collection, and attribution of outputs to results and immigrant outcomes. The implementation of an appropriate performance measurement framework for the settlement program is an onerous but necessary task. Performance measurement supports reporting on evidence-based outcomes to assist management decision making and guide new program initiatives. In our opinion, it is too early to assess the extent to which CIC will be able to report effectively at the outcome level with respect to its settlement programs because of the challenges noted and the complexity of the delivery structure.

Recommendation 6

The Department should ensure that the current performance measurement strategy that is being implemented will clearly articulate the approaches, methodologies and outcome information that will be provided in order to guide the future programming decisions related to the departmental integration programs.

Management Response

The Department agrees with the recommendation.

As part of the performance measurement strategy, the Department is working to define the elements of the performance measurement framework. This includes aligning outputs and outcomes with performance indicators that can be measured across all jurisdictions, regardless of program design. The approach and development of methodology under the leadership of the Integration Branch will allow the Department to make informed decisions on the settlement program (i.e., data sources, data collection methodology, target sample, frequency of reporting, performance targets and accountabilities).

Next Steps and Deliverables

  • The Policy Committee to approve the methodology for data collection of performance measurement framework for the immediate and intermediate level outcomes portion in September 2010.
  • The Business Operations Committee to approve the implementation plan for outputs for the performance measurement framework in December 2010.

3.3 Internal Controls

The audit assessed the adequacy of internal controls over the administration of transfer payment programs to ensure compliance with applicable policies. We expected to find that appropriate controls were in place to ensure that:

  • an adequate review process was in place for reviewing proposals;
  • an appropriate process was in place to approve agreements;
  • adequate monitoring of agreements was occurring; and
  • there was compliance with financial administration processes.

In general, we found that CIC had well-documented processes in place to ensure that there was a proper review and approval of agreements. As well, practices and procedures with respect to the monitoring and processing of claims were also clearly established. However, our examination of agreements showed that while the processes were in place, they were not being consistently applied in all the regions. We also noted that many of the SPOs had had agreements with the department for many years and that program officers generally had a good knowledge of SPO operations.

Consequently, we concluded that internal controls only partially met our expectations as we found several areas where the processes in place did not match departmental guidelines. However, we found that many of the guidelines related to settlement internal controls had not evolved in line with the growth of these programs, and some control processes might no longer be suitable given the level of maturity of the settlement programs. Revisions were under way at the time of our audit.

3.3.1 Contribution Agreement Process

Our audit reviewed controls in place over the contribution agreements management process depicted in Appendix B. We examined 121 contribution agreements across the country.

In general, we found that CIC had well-documented processes in place to ensure that there was a proper review and approval of agreements. As well, practices and procedures with respect to the monitoring and processing of claims were clearly established. However, our examination of agreements showed that while expectations for these processes were in place, they were not being consistently applied in all the regions.

In terms of the CA individual files, we noted that:

  • all CA files had a signed contribution agreement on file which included evidence and review of the proposal;
  • regular communication with SPOs was occurring and had improved since our last audits; and
  • regular ongoing activity and financial reporting by SPOs was occurring as expected.

We also note that the majority of the SPOs have had agreements with the Department for several years and that program officers generally had a good knowledge of the SPO operations and the nature of the expenditures being submitted for reimbursement. With respect to the financial administration of the claims, for the sample of claims we examined, we found instances of incomplete or missing documents related to the purchase of items and the reimbursement of certain costs. Our detailed observations are provided below.

Proposal assessment: We found that all regions had a different tendering process, but that a formal review mechanism was in place and was adequate. Generally, this process was documented and contained an element of managerial review.

Contribution agreement signing: We noted above that all agreements we examined had been signed. However, we found that 15% of the contribution agreements had a retroactively signed initial agreement, amendment or both. Additionally, we also found that 13% of CA files had agreements or amendments that did not include a schedule of services to be delivered (Schedule 1). By not having agreements in place that document the services to be delivered, there is the risk  that services may not be delivered as intended.

Monitoring of agreements: Program officers within the Department are responsible for the formal monitoring of the financial and program activity of the SPOs. The extent of monitoring is established according to a prescribed monitoring plan, as per the CA. Additionally, the majority of the SPOs have had contribution agreements with the Department for several years and program officers had a good understanding and knowledge of their operations, were in regular communication with SPOs and were aware of the ongoing activities being carried out by them.

However, our audit found that formal monitoring was not occurring as intended in the monitoring plans. In 17% of our sample agreements, we found no updated monitoring plan on file and in 45% of the agreements we examined, the required monitoring was not occurring as intended. If monitoring is not occurring as intended, services delivered may not be occurring as planned in terms of frequency and quality. We noted that the SPOs were periodically reporting on activities as required, but we found limited documentation of reviews of these activity reports to ensure satisfactory progress on results or that the quality of services rendered was satisfactory.

We also noted that as part of SPO periodic activity reporting, some qualitative information is reported by SPOs. Generally, only quantitative information reported by SPOs was reported up through the Department while qualitative information remained at the local level. Some of this qualitative information may provide on-the-ground information about the outcome results of these services in the communities served (as discussed above in performance information). However, in the absence of formal monitoring, we did note that program officers were in regular communication with SPOs and aware of the ongoing activities being carried out by them. We also noted that the majority of the SPOs have had contribution agreements with the Department for several years, and program officers had a good understanding and knowledge of their operations.

iCAMS: As part of their responsibilities under the contribution agreements, SPOs are required to enter data on results into iCAMS. iCAMS is a reporting tool used by the Department to capture results information on its settlement programs. Individual SPOs have the responsibility of inputting the data, while CIC programs officers are required to monitor the input of data to ensure their accuracy and completeness. We found that in 66% of the files we examined, we could not find any documented evidence that the monitoring of iCAMS, including data entry, was occurring as intended. This does not mean that SPOs did not enter the information, but that there was no evidence on file to demonstrate that officers were reviewing this information to ensure it was both accurate and complete. If iCAMS is not monitored, there is a risk that the information will be incomplete or inaccurate. This information is used by management to support its decisions on the programs.

Financial administration: Under the conditions of the contribution agreements, payments are periodically made to SPOs as costs are incurred and satisfactory results are achieved. In some circumstances, advances may be made to SPOs. There is also normally a 10% annual holdback within each contribution agreement.

For the agreements and claims (activity during fiscal year 2008–2009) we examined as part of this audit, we found:

  • Agreements that contained inconsistent wording and conflicting clauses related to advance payments; 
  • Claim files that had incomplete or missing documents related to the review of receipts for items being purchased;
  • Instances of costs being reimbursed that were not originally contemplated within the agreement and with no supporting documents; 
  • Incomplete documentation when the payment of claims differed from the amounts originally forecasted; and
  • Instances where there were end-of-year prepayments being incorrectly expensed in the year of recording.

These findings reinforce the requirement for proper documentation to ensure the completeness of contribution agreement files. Other risks relate to the accuracy of financial data and the payments for items not eligible for reimbursement.

Close-out: We were unable to conclude on close-out procedures as many of the agreements included in our file review were still active. In some cases, this was due to the timing of our visit.

We note that new programs and services would be more susceptible to these potential risks than existing services.

3.3.2 Other Considerations

Management advised that there was a significant increase in funding without a similar growth in resources to manage the programs and that this may account for some of the above observations. We also note that the following may have had an impact on these observations:

  • A new approval process requiring broader consultation and phased approvals, implying a longer time to approve an agreement;
  • Incongruent planning and reporting horizons; and
  • Guidelines not being up to date and reflecting the current state of the programs.

The compensating controls that we found in place were phased approvals in the proposal assessment process, which narrowed the gap of understanding despite the lack of an agreement in place; the use of pre-existing service providers who are familiar with our programs, thus decreasing the learning curve; and increased contact with SPOs, which served to approximate formal monitoring.

Overall, many of the guidelines related to settlement had not evolved in coordination with the growth of these programs. Consequently, some control processes may no longer be suitable given the level of maturity of the settlement programs. Management advised us that a revised settlement manual was in progress but had not yet been approved for release.

Based on our review of the guidelines and manuals in effect at the time of our audit, the observations emanating from the Blue Ribbon Panel work had not been fully incorporated into departmental guidelines. However, we saw some areas of the process which could be refined to take advantage of the observations raised by this committee. Some areas that may benefit from refinement include the following.

  • Approval process: This process has become lengthier and requires greater consultation, resulting in more time to negotiate and sign agreements. The revised settlement program has yet to be implemented and the previous programs remain in place. Currently, the process is for SPOs to sign agreements by program, which may result in multiple agreements per SPO, further adding to the administrative burden. As the Department moves to incorporate these old programs into the new structure, CIC has opportunities to simplify the approval process and should consider phasing these new programs in with well-performing SPOs to ease the changeover.
  • Risk assessment: The criteria used to assess risks require updating. One of these criteria defines agreements as high-risk based on a dollar threshold established at the time of our previous audit work. Most agreements in effect at the time of our current audit exceed this threshold and therefore, a higher than perhaps necessary level of monitoring was set. In addition, risk ratings were established by contribution agreement rather than SPO. Under the current process, a single SPO may be awarded multiple CAs resulting in multiple risk assessments and multiple monitoring plans for a single SPO. Doing risk ratings at the SPO level would allow for decision making at the SPO level rather than that of the CA and would allow for more efficient management.
  • Monitoring: Monitoring is only generally defined in the settlement manuals and tools to carry out monitoring are broad in focus and touch on many issues in little depth. The development of guidelines may provide more flexibility on monitoring by allowing more targeted monitoring requirements and a more efficient use of monitoring resources. For example, creating a modular monitoring program covering a range of topics may allow an officer to target monitoring efforts to areas of concern rather than to a wide range of areas with little depth.
  • Reporting requirements: Currently, SPOs report specific quantitative data through iCAMS and both qualitative and quantitative data through periodic narrative reports. There is some duplication of effort. However, there is also inconsistency across SPO narrative reports, which do not allow for roll-up or comparison. While we acknowledge that there could be some leeway to allow for variation in services and delivery, common standards should be established to allow CIC to take advantage of these data at higher levels for program design considerations, in particular SPO qualitative data.
  • Close-out procedures: Current guidelines in this area predate the use of multi-year agreements and consequently, it is not clear whether these procedures should be done on an annual basis or at the end of the full term of an agreement. Anecdotally, we saw instances where unresolved issues (which are likely to grow over time) remain from past years despite new CAs having been signed. As funding is conditional on annual appropriations, this should be done on an annual basis so that adjustments can be made, taking into consideration SPO performance.
Recommendation 7

The Department should review its current proposal approval process and risk assessment process and consider revising and streamlining these areas as appropriate. These changes should be incorporated into the revised settlement manual to be issued later this fiscal year. 

Management Response

The Department agrees with the recommendation.

For the financial aspects of the G&Cs, the Financial Management Branch will develop a formal decision process to be fully integrated into the initial risk assessment and the related reporting and monitoring strategies. This work will reinforce current monitoring approaches and place them on a solid footing. The expected date is March 2011.

In addition to the improvements described in responses to recommendations 4 and 5 pertaining to the revisions of the settlement manual and the development of a new risk assessment process, the Value for Money Working Group was establishedin2009–2010 under the auspices of the SRMMC with the goal of determining the specific criteria that will be used during the assessment phase for proposals submitted in response to CFPs. The group has been working to harmonize and ensure consistency in the approaches currently in place across the Department. Some of this work is now completed. Because it contains information on the factors that are crucial to how settlement officers discharge their responsibilities, the settlement manual will remain the primary vehicle for these elements. The expected date is September 2010.

Recommendation 8

The Department should review its overall monitoring approach as well as its close-out procedures and consider revising and streamlining these areas as appropriate. These changes should be incorporated into the revised settlement manual to be issued later this fiscal year.

Management Response

The Department agrees with the recommendation.

The Department has finalized a monitoring strategy that is being used at NHQ for national and international agreements. In addition, the Department has developed a new close-out form and procedures that are being piloted in 2010–2011 at NHQ. All this has been incorporated into the new version of the settlement manual and is scheduled to be implemented in the regions in fiscal year 2010–2011.

Finally, to improve the management of situations where the Department may need to close an agreement before the conclusion of the negotiated time period—for performance or other reasons—the Department has developed an escalation process that sees progressively senior departmental officials engaged in determining a final decision.

In addition to the improvements described in the response to recommendation 5, the Financial Management Branch is developing tools such as audit programs to ensure that the financial monitoring of agreements is consistent in the regions and at NHQ. These tools will be available by March 2011. The Financial Management Branch will ensure that the prescribed monitoring plans occur in a timely manner, through ongoing coordination and monitoring. The Branch will also ensure that the Department takes action as needed and in a timely manner based on the results of the audits or reviews of SPOs.

Recommendation 9

The Department should ensure that expectations regarding monitoring practices are clearly communicated and that policy requirements are adhered to.

Management Response

The Department agrees with the recommendation.

The changes to iCAMS and the implementation of a performance measurement framework and planning strategy will contribute to improvements in monitoring outputs and outcomes of settlement planning, and ensure adherence to policy requirements. More importantly, the essential information provided by the monitoring activities will be communicated to inform any required policy development and ensure the continued integrity of the settlement program. Again, the SRMMC, with its regional representation, will be an appropriate forum to maintain communication lines between monitoring practices and policy requirements.

The Department’s current monitoring strategy will further strengthen ongoing communications with the SPOs and inform them of the new monitoring practices that are being instituted. Both CIC officers and SPOs will receive the relevant training as new procedures are implemented by September 2010.

Training in monitoring will occur for program officers and financial officers and will also include an SPO component. This will occur from September 2010 to March 2011.

Date Modified: