Internal Audit of the Operation Syrian Refugees (OSR) – Settlement

Internal Audit & Accountability Branch

21 February 2017

Table of Contents

1. I. Background
2. II. Audit objective, scope and methodology
3. III. Audit findings and recommendations
4. IV. Conclusion
5. Appendix A – Management response to recommendations

I. Background

Introduction

Syrian refugee resettlement and settlement

10. The Settlement Network is part of the IRCC Operations Sector and is responsible for overseeing the delivery of all IRCC settlement and resettlement grants and contributions programs including the Resettlement Assistance Program and the Settlement Program.
11. To ensure that the grants and contributions programs are managed with integrity, transparency and accountability, IRCC entered into contribution agreements (CAs) with service provider organizations to deliver the Resettlement Assistance Program (RAP) and Settlement services. The CAs identify the terms and conditions by which the service provider will receive funding from the Department. An electronic and/or hard copy record of the CA is required to be retained and the Department is expected to retain records of the management and monitoring of the CA in compliance with the Treasury Board Policy on Transfer Payments.
12. Resettlement Assistance Program (RAP). This program supports the immediate and essential needs of Government-Assisted Refugees upon arrival, and is delivered during the first four to six weeks after arrival by service provider organizations that have CAs with IRCC, and in the province of British Columbia, by third party service providers who are under contract with a service provider organization that has a CA with IRCC. The funding is used to help pay for items such as meeting refugees at the port of entry, temporary accommodations, assistance in finding permanent accommodations, and general orientation to life in Canada. Income support under the program is provided for up to the first twelve months after arrival, and in some cases up to two years. RAP income support funds provisions for rent, food, and basic household items. There were 41 RAP service providers that delivered services to Syrian newcomers as part of OSR.
13. Settlement Program. Settlement services provided under the Settlement Program include needs assessments and referrals, information and orientation, employment related services, community connections, and language services. This program is available to all new immigrants including resettled Syrian refugees. The program objective is to provide newcomers with the information required to make informed decisions, develop language skills adequate to their settlement and integration goals, and provide the support needed to build networks within their new communities. There were 297 Settlement service providers that delivered services to Syrian newcomers as part of OSR.
14. For OSR, additional funding was approved by the Treasury Board to enable the service provider organizations to meet the increased demand for Syrian refugee resettlement and settlement services. Project officers in IRCC regional offices were required to manage and monitor the CAs and any subsequent amendments to ensure that the delivery of resettlement and settlement program services was performed in accordance with the agreements in place.

II. Audit objective, scope and methodology

Audit objective and scope

15. The audit objective was to assess whether processes and procedures were implemented to administer Syrian refugee settlement and resettlement programs.
16. The examination focused on the internal controls in place to transfer information on Syrian refugee settlement requirements to service provider organizations; and the processes in place to ensure that CAs were managed and monitored. The scope included the settlement and resettlement of Syrian refugees that arrived in Canada between November 4, 2015 and February 29, 2016 as part of Operation Syrian Refugees.
17. The audit did not include an examination of resettlement services delivered to Syrian refugees destined to the province of Quebec since the province is responsible for administering its own refugee sponsorship programs including resettlement and settlement services as part of the Canada-Quebec Accord relating to Immigration and Temporary Admission of Aliens (February 1991). In addition, the audit did not include an examination of the Resettlement Assistance Program Income Support program which is directly administered by IRCC. Furthermore, the audit did not include an examination of Resettlement Assistance Program services delivered at the ports of entry. Finally, the audit comments on IRCC administration and does not comment on any other organization that was involved in, or that supported the resettlement of refugees.

Methodology

18. The following audit procedures that were performed:
    • Reviews and analyses of program delivery documentation such as policies, procedures, and manuals;
    • Interviews with IRCC Headquarters officials; IRCC Regional Office officials; as well as service provider organization officials;
    • Walkthroughs of the electronic systems in place to support program delivery - the Grants and Contributions System (GCS) and the Immigration Contribution Agreement Reporting Environment (iCARE) system;
    • A sampling strategy was developed that included:
      • Thirty Syrian refugee application files to assess whether the operating procedures for the processing of Notification of Arrival Transmissions were implemented;
      • Thirty-six CAs to assess whether processes and procedures were implemented to manage and monitor program delivery - 8 CAs with Resettlement Assistance Program service provider organizations; 28 CAs with Settlement Program service provider organizations; and
      • Seventy-two CA amendments that were made during the OSR time line.
    • Reviews and analyses of the information in the GCS for 36 CA files.

Statement of conformance

19. The audit is in conformance with the Internal Auditing Standards for the Government of Canada as supported by the results of the quality assurance and improvement program.

III. Audit findings and recommendations

Transfer of information to settlement Service Provider Organizations

20. Criteria: It was expected that established internal controls were implemented to transfer Syrian refugee settlement requirement information to service provider organizations to enable service delivery.
21. Conclusion: Internal controls were implemented to transfer Syrian refugee settlement requirement information to service provider organizations to support Operation Syrian Refugees program delivery.

Transfer of information

22. Syrian refugee applications were processed by immigration officers at visa processing centres overseas. Once an application was approved, information was sent to the IRCC Refugee Matching Centre located at IRCC National Headquarters. The matching centre was responsible for matching Government-Assisted Refugees to communities that would best meet their individual requirements while Privately-Sponsored and Blended Visa Office Referred refugees travelled to the destination of their approved sponsor. An immigration officer then prepared a Notification of Arrival Transmission to provide information such as sponsor names and special medical requirements to help prepare for the arrival of the refugee to their final destination. The standard process required the Notification of Arrival Transmissions to be sent to the matching centre at least 10 working days prior to the refugee arrival date and then sent to local IRCC offices to be distributed to resettlement service providers accordingly.
23. The Refugee Matching Centre implemented a special Standard Operating Procedure for OSR to manage the coordination and communication of information to resettlement service provider organizations or private sponsors in anticipation of the arrival of a high number of Syrian refugees in a short period of time. The review of the implementation of the procedures in place for OSR indicated that the process generally worked as intended. The Notification of Arrival Transmissions were sent to the service provider organizations or the private sponsors prior to the refugee’s date of arrival. When there were delays, IRCC and the service provider organizations worked together so that the Syrian refugee settlement requirement information was shared to support program service delivery.

Managing unique individual needs

24. A Case Management Unit was established at IRCC National Headquarters for OSR to manage unique individual refugee cases in advance of their travel to Canada or travel within Canada to their final destination to respond to unique needs in an effort to support their resettlement. These types of cases included special medical requirements, arranging travel for missed flights, assessing requests for alternate destinations, and re-destining refugees to other cities.
25. A Standard Operating Procedure was developed to manage these types of cases. Overall, there were 475 unique individual cases that had to be addressed during the period under review. Officials managed these types of cases by using the Global Case Management System and by setting up an electronic mailbox to track and monitor all requests. Documentation was recorded and retained to demonstrate that the capacity of the receiving service provider organization was confirmed prior to the determination of where the individuals would be sent – this included information related to the Notification of Arrival Transmission sent to the service provider organization. All cases reviewed as part of this audit were approved by an official with the appropriate delegated authority and almost all (90 percent) were resolved within 10 days.

Managing and monitoring contribution agreements in place with Service Provider Organizations

26. Criteria: It was expected that processes and procedures were implemented to ensure that contribution agreements with service provider organizations were managed and monitored to support program service delivery to Syrian refugees.
27. Conclusion: Processes and procedures to manage contribution agreements were implemented and IRCC officials generally monitored program delivery to Syrian refugees. Areas for improvement were noted regarding the consistent use of the Grants and Contribution System to manage the contribution agreements; assessing whether the hub and spoke model continues to meet the needs of the Department; and improving the monitoring of the service provider organizations and the use of performance information for the resettlement and settlement programs.

Management of contribution agreements

28. When IRCC enters into CAs with service provider organizations to deliver resettlement and settlement services, there are processes and procedures in place to manage and monitor the expected delivery of resettlement and settlement services included in the agreements. Through the management of the CAs, project officers are required to be aware of the service provider’s progress and how their service activities relate to overall program outcomes. The purpose of monitoring CAs is to assess the extent to which the expected results of the agreement are being met and whether they are being met in an effective and efficient way. Monitoring the CAs also provides an opportunity to identify the service provider organization’s successes and challenges and to provide them with support.

Grants and Contribution System to manage contribution agreements

29. The Grants and Contributions System (GCS) is the electronic system used by IRCC officials to monitor and manage CAs and program officers are expected to use GCS for their routine work. The GCS was initially rolled-out across the Settlement Network between January and March 2015 and was implemented to increase program standardization, and improve CA service delivery tracking and reporting. The Department continues to update the system to improve functionality. Prior to the launch of GCS, there was no single repository for information about the CAs and information was retained primarily in hard copy paper files.
30. For the examination of the processes and procedures used to manage CAs, a sample of 36 agreements with 72 related amendments was reviewed. As GCS is the main repository for managing and monitoring the CAs, the review of the sample agreements was conducted primarily by examining GCS records. When information was not available in GCS, the hard copy paper file was also reviewed.
31. The audit examined a number of key records in GCS which demonstrate due diligence by officials to manage the CAs and amendments related to OSR. The review looked at records related to the dollar value, the delegated authority sign-off, capacity assessment and other sources of funding.
32. The review identified that the dollar value of the CA amendment was not recorded in the GCS file in 42 percent of the CAs. During OSR, the system did not have a field to identify the amendment value; this field was added to the system in June 2016. In addition, a record of other sources of funding was not captured in GCS for any of the agreements in the sample. Without a record of the amendment value and other sources of funding in the system, there is a risk that the Department does not have accurate information in GCS regarding the total funding provided to the service provider organizations.
33. The GCS includes a field where the delegated authority electronically approves the commitment and the transaction authority for an amendment to the CA. According to the GCS guidance, a project officer may complete the field on behalf of the delegated authority and a copy of the original signed version of the CA is expected to be scanned into GCS.
34. Electronic approvals of amendments to CAs made to address OSR funding were added as a function in GCS to further reduce payment processing times and the amount of paper that IRCC staff needed to print. The examination of the GCS files identified that, in almost half of the CAs (47 percent), the approval field was not filled in. In addition, for more than half (61 percent) of the amendments examined, a scanned copy of the signed amendment was not in the GCS file as required.
35. The Department recognized that the capacity of the service provider organizations in the communities receiving Syrian refugees was critical to successful resettlement and settlement. Capacity refers to the ability for the service provider organizations to successfully deliver their programs and services with the appropriate resources, infrastructure, and funding. Amendments were made to existing CAs to provide additional funds to support and build capacity to address the surge of refugees being resettled and settled as a result of OSR. As such, the audit reviewed the sample of CAs and amendments to determine if capacity of the service provider organization was considered when recommending an amendment during OSR. In GCS, the recommendation and rationale for an amendment is an essential record in the approval process.
36. In the examination of the 72 amendments, the capacity of the service provider organizations was not identified in 51 percent of the GCS files. Without including information about the assessment of the capacity of the service provider in the file, the Department cannot demonstrate that an adequate assessment was conducted of the service provider organization’s capacity for administering the activities and services to meet the expectations set out in the amendments for OSR.
37. Given the nature of program delivery in communities and cities across Canada, service provider organizations may be funded to provide services by other federal departments as well as provincial, territorial, or municipal levels of government. As such, service provider organizations are required to disclose potential sources of funding or in-kind participation for program activities and/or eligible costs related to the CA. They are also expected to inform IRCC of any changes in funding from other sources throughout the duration of the agreement. The audit identified that the file in GCS includes a section where other sources of funding are expected to be recorded. However, no information on other sources of funding was recorded in GCS for any of the CAs examined.
38. Overall, guidance on the use of GCS was not consistently applied. In many instances, program officials indicated that because of their regular communications with service provider organizations, they were aware of issues that were not documented in GCS. While the information that officials require to manage agreements is sometimes available in the paper files in the regional offices, the inconsistent use of GCS prevents a more strategic analysis of data and information held at the regional offices to assess the management of OSR related agreements across the Settlement Network. Furthermore, this may present challenges for IRCC senior management to obtain accurate, reliable information and data on overall program delivery.
39. Recommendation 1 – The Associate Assistant Deputy Minister, Operations should ensure that program officials record all required information in the Grants and Contribution System (GCS) in a consistent manner to maintain a complete and accurate record of the oversight of contribution agreements.

The hub and spoke model for resettlement services

40. IRCC used a unique approach (the Hub and Spoke model) in British Columbia to resettle Syrian refugees in cities outside the Greater Vancouver Area. An amendment to the CA was signed with the service provider organization (the Hub) on March 8, 2016 which permitted the organization to enter into Third Party Agreements with other service provider organizations (the Spokes) to deliver resettlement services to Syrian refugees in other cities across British Columbia. Syrian refugees were therefore directly destined to cities where the third party service provider organizations provided resettlement services.
41. The review of the management and monitoring of the Hub and Spoke model identified that, despite being a requirement under the amended CA, neither the electronic file nor the paper copy of the CA with the resettlement service provider organization included copies of the contracts between the service provider organization and the Third Party Agreement holders. [Note: copies of the contracts were later provided to the audit team upon request]. The review of the CA with the service provider organization also noted that the CA did not include the target number of refugees expected to receive resettlement services - either the overall target number or the number for the Spoke communities outside of the Greater Vancouver Area.
42. Furthermore, IRCC program officials do not have the information required to assess the capacity of the Third Party Agreement holders providing resettlement services outside of Vancouver. This is because IRCC does not monitor these service provider organizations without a signed CA with the Department.
43. Given the findings identified, there is a risk that the Department does not have the information needed to monitor the resettlement of Syrian refugees in the cities outside of the Greater Vancouver Area and that there is inadequate oversight of the resettlement services delivered through third party organizations.
44. Recommendation 2 – The Associate Assistant Deputy Minister, Operations should assess whether the hub and spoke model continues to meet the needs of the Department, and if so, ensure the appropriate accountability mechanisms are in place for departmental oversight of program delivery.

Monitoring service provider performance

45. IRCC is accountable for monitoring planned objectives and deliverables as set out in CAs and for the assessment of whether satisfactory results have been achieved.
46. For OSR, additional funding was received from the Treasury Board to support the transition of Syrian refugees from resettlement to settlement service providers by targeting immediate and urgent settlement needs. These funds were for service provider organizations to provide services to the Syrian refugees. The OSR funding had to be rolled out in a timely manner so that services could be provided as quickly as reasonably possible. Service provider organizations could only access funding once CAs or amendments were in place.
47. Recognizing the risks associated with the roll-out of Syrian initiative funds, IRCC officials issued internal guidance for program officers regarding the monitoring of OSR funding. Emphasis was placed on ensuring that information was captured in specific Schedules to the CAs for their Syrian and non-Syrian funded programming so that the intended results for OSR could be tracked. The audit examined schedules to the CAs to determine whether planned activities were specified before services were delivered to the refugees.
48. Schedule 1 of the CA identifies the service provider organization’s planned activities, the target number of clients to be served, and the intended results to be achieved for the funds received. In the review of agreements, 50 percent of the resettlement program CAs and 7 percent of the settlement program CAs did not include the target number of clients to be served.
49. Schedule 2 of the CA describes the eligible expenditures to monitor how Syrian funding would be spent. For OSR, service provider organizations were required to provide a breakdown of planned expenditures for programs specifically supported by Syrian funding and non-Syrian program expenditures. The review of the amendments identified that 17 percent of the amendments did not include the Schedule 2 breakdown of Syrian funding in the description of expenditures.
50. When a service provider organization begins the delivery of services expected under the CA, they submit a monthly (or quarterly) Forecast of Cash Flow statement. The Forecast of Cash Flow statement includes the expenditures for the planned and delivered services. However, the audit identified that the Department did not request the breakdown of planned and delivered expenditures by Syrian and non-Syrian funding in the Forecast of Cash Flow statement.
51. Without complete information regarding the planned activities, the target number of clients, the intended results to be achieved (Schedule 1) as well as the breakdown of how these activities are funded (Schedule 2 and Forecast of Cash Flow), the Department may lack information to adequately monitor service provider organizations.

Performance reports

52. The Immigration Contribution Agreement Reporting Environment (iCARE) system was implemented in 2013 to collect statistical client and service delivery information, and monitor client usage of settlement services. The iCARE system is also used to help track service provider results, successes and challenges.
53. Service provider organizations providing direct settlement services are required to submit Narrative Reports and Annual Project Performance Reports to IRCC officials who are responsible for monitoring the implementation of the CA through the use of iCARE. Officials are required to review and approve these reports through the iCARE system. The Narrative Reports are submitted on a monthly (or quarterly) basis, as identified in the CA. The Narrative Report provides qualitative information to support the statistics entered into iCARE, including the successes and challenges experienced by the service provider organization in delivering the program.
54. Currently, Narrative Reports and Annual Project Performance Reports for resettlement services under RAP are not captured in iCARE, and rather are submitted directly to IRCC officials who are responsible for monitoring the implementation of the CA.
55. The Department’s guidance states that the content of the Narrative Report assures the officer that all requirements of the CA are being met. As the iCARE system is not linked to GCS, the audit examined whether the narrative reports were included in the GCS files, given that GCS is the system for managing the CAs. The review of the CA electronic and paper files indicated that 6 percent of the narrative reports were uploaded to the GCS file and 53 percent of the paper files included hard copies of the narrative reports. In addition, the audit noted that there was no standardized format used for the narrative reports.
56. As a result, there is a risk that the Department cannot demonstrate that there is adequate oversight to ensure the terms and conditions of the CAs are being met and that challenges in program delivery experienced by service provider organizations are reported and addressed in a timely manner. Furthermore, the lack of standardized format makes it difficult to aggregate information to obtain a horizontal perspective of program delivery results.
57. The audit noted that IRCC collects output-type information regarding services delivered such as the number of language classes delivered and number of participants. There is an opportunity for the Department to use outcome-type information to have a better understanding of program results regarding the impacts of services delivered to OSR clients.
58. Recommendation 3 - The Department should reassess the information required to monitor service provider organization effectiveness. The review should include determining what information is needed and how this information is obtained so that useful data, in a standardized format, is available to report on OSR results.

IV. Conclusion

59. Overall, internal controls were implemented to transfer Syrian refugee settlement requirement information to service provider organizations to support Operation Syrian Refugees program delivery. Processes and procedures to manage contribution agreements were implemented and IRCC officials generally monitored program delivery to Syrian refugees. Areas for improvement were noted regarding the consistent use of the Grants and Contribution System to manage the contribution agreements; assessing whether the hub and spoke model continues to meet the needs of the Department; and improving the monitoring of the service provider organizations and the use of performance information for the resettlement and settlement programs.

Management has accepted the audit findings and developed an action plan to address the recommendations.

Appendix A – Management response to recommendations

Recommendation 1

The Assistant Deputy Minister, Operations should ensure that program officials record all required information in the Grants and Contribution System (GCS) in a consistent manner to maintain a complete and accurate record of the oversight of contribution agreements.

Management response

Agree. In May 2013 IRCC started the development of a comprehensive grants and contributions (Gs and Cs) information management and technology solution to modernize the management of Gs and Cs just as the Global Case Management System (GCMS) modernized citizenship and immigration business lines. The objective is to improve program integrity, further streamline processes, and promote national consistency in program delivery. The Grants and Contributions System (GCS) was planned to be launched in components with the first release in January 2015, and the latest major release in March 2017. As the system is still relatively new, IRCC is still in the implementation and change management process.

In June 2016, Settlement Network conducted a Quality Control exercise of data quality in GCS. The exercise provides baseline data related to compliance, and the Network has taken action to address issues, including enhancements to user support.

Recommendation 2

The Assistant Deputy Minister, Operations, should assess whether the hub and spoke model continues to meet the needs of the Department, and if so, ensure the appropriate accountability mechanisms are in place for departmental oversight of program delivery.

Management response

Agree. The new requirements will be included in any new contribution agreement that involves third-party service delivery partnerships.

Recommendation 3

The Department should reassess the information required to monitor service provider organization effectiveness. The review should include determining what information is needed and how this information is obtained so that useful data, in a standardized format, is available to report on OSR results.

Management response

Agree. The Department is looking at strengthening outcome reporting for all resettlement and settlement programming not just OSR. The Department currently collects a variety of data to support the analysis and measurement of both outputs and outcomes.

At the outset of Operation Syrian Refugees, the Department developed and implemented an Outcomes Monitoring Framework, comprised of performance measurement, evaluation, and research angles, to gather performance information and outcomes data on the Syrian refugees. It includes ongoing performance indicators, a Rapid Impact Evaluation, ongoing iCARE data analysis, Annual Project Performance Report (APPR) analysis and a Syrian Refugee Research initiative (26 co-funded research projects on Syrian integration). This is in addition to weekly reporting that continues through operational dashboards and other venues.

It should be noted that some outcomes for Syrian refugees and associated monitoring and reporting are longer-term, as the integration process can take time, and associated data will only become available after time (such as IMDB data for economic outcomes). The Department will consider survey options and new data linkages to further enhance the ability to report on early and midterm results for the Syrian population.