Report on the Evaluation of Canada’s Membership in the International Organization for Migration

3. Evaluation Framework

3.1 Introduction
3.2 Evaluation Matrix
3.3 Evaluation Methodology


3.1 Introduction

This chapter contains a description of the evaluation framework used for the evaluation of Canada’s membership in the IOM. The framework includes the evaluation matrix and a description of the evaluation methodologies.

3.2 Evaluation Matrix

Appendix B of this report contains an evaluation matrix for the evaluation. The matrix summarizes the key evaluation issues and questions, related indicators and data sources for the evaluation. In the following sections, we discuss the evaluation questions under each of the three evaluation issues and then provide an overview of the methodologies used to evaluate against these questions.

Evaluation Questions

As is evident from this matrix, we have grouped the evaluation questions under the following headings:

Governance and Accountability

The IOM was established by the member states to assist them in dealing with some very difficult migration issues following World War II. In the half-century since its founding it has grown dramatically in terms of membership and in terms of the nature and scope of its activities. Ostensibly, however, it remains an organization whose raison d’être remains one of providing migration-related services to its members. An important question, then, given its growth, is whether the member states are able to ensure the IOM continues to operate in accord with their interests and its mandate. Key questions in this regard are:

  • Do Canada and other member states exercise effective oversight over IOM programmes, budgets and memberships, the strategic directions of the IOM and other issues?
  • To what extent are current/planned IOM roles and activities perceived to be in the interest of Canada?
  • Does the IOM overlap with or duplicate the mandate, roles or activities of any other international organizations on migration issues?
  • How effective are current governance bodies and what, if any, changes should be made to the current governance structure to improve its effectiveness?
  • Do Canada and other member states have sufficient information to determine whether staff and office costs charged to the Administrative Budget are appropriate?
Benefits and Alternatives

The benefits of membership in international organizations tend to be somewhat intangible, relating more to non-quantifiable outcomes than to measurable economic or other benefits. As such, they are difficult to evaluate. It quickly became evident, in the course of developing this framework, that this is the case with Canada’s membership in the IOM, to some extent. However, the IOM membership was justified partly on the grounds of ensuring continued access to, and realizing savings from IOM’s services in the areas of medical examinations, and processing and transporting refugees and other migrants to Canada. Consequently, we developed our questions and related indicators to address both the “soft” and the more tangible benefits of membership in the IOM.

A related issue is whether Canada could achieve the objectives it is aiming at through this membership, through membership in other international organizations or through the services of another international service delivery agency.

We identified four key questions under this issue:

  • To what extent and in what ways has Canada benefited from its membership in the IOM in terms of migration issues?
  • Are there alternative ways Canada could more effectively address bilateral/multilateral migration issues?
  • What have been the benefits to Canada of accessing IOM service in the areas of transportation, medical assessments and orientation training?
  • Could Canada access IOM cost-recoverable migration services without being a member of IOM?
Relevance

Ultimately, Canada’s membership in the IOM is intended to be relevant to the objectives of Canada’s Immigration Program. These objectives include meeting its humanitarian commitments regarding the resettlement of refugees, managing access to Canada and benefiting from the global movement of people. We have identified one question related to relevance:

  • To what extent does continued membership in the IOM contribute to the strategic objectives of Canada’s immigration program?

3.3 Evaluation Methodology

The methodological options for evaluating memberships, such as Canada’s membership in the IOM, are limited for several reasons:

  1. The outcomes of Canada’s IOM membership, as noted earlier, are, for the most part, “soft” outcomes. There are very few tangible, quantifiable benefits that can be attributed to the membership.
  2. Even with respect to the few potentially quantifiable benefits — cost savings to migrants and to Canada from accessing IOM services — there is very little data available to measure these cost-savings or to determine what would have been the cost of alternative approaches to obtaining the services currently provided by the IOM.
  3. A detailed comparison of IOM costs with the costs of alternative approaches to obtaining IOM migration services was not feasible due to the difficulties of accessing cost information for alternatives and due to the high cost of this methodology. In any event, based on what was learned from this evaluation and in the course of a concurrent evaluation of the Canadian Orientation Abroad (COA), there is currently no other international organization with the reach of the IOM (over 110 countries) and with its operational experience in migration services.

Consequently, we relied heavily on qualitative research methodologies, including interviews with various knowledgeable stakeholders, reviews of key documents and observation of the IOM governing bodies for this evaluation, supplemented by quantitative methods where practical. As a consequence, many of our conclusions, especially regarding the “soft” benefits of IOM membership, are based on our considered judgment, taking into consideration the perspectives of stakeholders and our other lines of enquiry.

We employed the following methodologies.

Interviews with Key Stakeholders

We conducted interviews with the following stakeholder groups (see Appendix C for a list of interviewees):

CIC HQ and CIC Mission Officials

We interviewed officials of BSK at CIC HQ, the division responsible for managing Canada’s IOM membership. As well, we interviewed the Senior Immigration Counselor at the Canadian Mission in Geneva. These interviews also covered the entire spectrum of evaluation issues but were of particular value with respect to questions regarding the effectiveness of IOM governance structures; the extent to which members are able to exercise control over IOM directions and activities; the “softer” benefits of IOM membership, such as contributing to Canada’s ability to address bilateral/multilateral immigration issues; the extent to which the IOM duplicates or overlaps with other international organizations; and the extent to which IOM membership contributes to the strategic objectives of Canada’s immigration program.

Officials of Other Missions

We interviewed officials at the Geneva Missions of seven (7) other member states: Australia, Germany, Guatemala, Latvia, Netherlands, Sweden and the US. These individuals represent their state at the IOM. Most of those we interviewed had been involved with the IOM for some time. In one case we conducted a joint interview with the outgoing representative and their replacement.

These interviews covered governance and accountability; the softer benefits of IOM membership; overlap or duplication with other organizations; and the relevance of the IOM to member states. In a couple of cases, we were not able to cover all of the evaluation topics in the time available for the interviews.

IOM Officials

We conducted interviews with most of the senior management team of the IOM in Geneva, including the Director General. These interviews covered the entire gamut of evaluation issues, except in one case where the interviewee had to leave before the interview was completed. The interviews were conducted in person except in one case. In this case the two interviewees provided a completed questionnaire by e-mail.

For the most part, the interviews covered the entire range of evaluation issues but with a significant focus on governance and accountability issues, the IOM structure and operations, recent strategic initiatives, issues related to budgeting and finance and the benefits of membership. In the course of these interviews we also obtained data that was very helpful in evaluating the cost savings from IOM’s refugee transportation services.

Review of Documents

In the course of the evaluation we obtained and reviewed a wide range of CIC and IOM documents. A complete list of these is contained in Appendix D. The following documents, however, were central to the evaluation:

  • Canadian Government documents correspondence pertaining to the renewal of Canada’s membership in the IOM, dating from 1991.
  • CIC briefing note outlining current issues relating to Canada’s IOM membership.
  • IOM Constitution and Mission Statement.
  • IOM Programme and Budget (2003/2004/2005).
  • IOM Strategy: Current and Future — Migration Realities and IOM’s Role.
  • IOM summary of airline fees for transporting refugees to Canada.
  • IOM electronic and hard copy information sheets on organization and activities.

These documents contributed to the evaluation of most issues, including questions regarding the extent to which the IOM’s activities and strategic directions reflect the interests of members and the mandate given to the IOM by member states; the effectiveness of member states in overseeing the IOM; the benefits of IOM membership; overlap or duplication with other organizations; and accessibility of IOM services to non-members.

On-Site Observation

While in Geneva, we attended the meeting of the IOM SCBF. This provided an opportunity to observe the workings of a key governance body and to see how it arrived at decisions. This was helpful in evaluating the effectiveness of IOM governance bodies.

Comparative Data Analysis

One of the alleged benefits of IOM membership is the cost-savings resulting from IOM processing refugees, conducting medical examinations (or monitoring Designated Medical Practitioners conducting medical examinations of migrants) and arranging for the transportation to Canada of refugees and other migrants requiring assistance.

In the time frame and budget of this evaluation, a comprehensive, in-depth analysis of all of the cost-savings — including the savings in staff costs to CIC — was not feasible. Nevertheless, we did attempt to analyze the extent to which the IOM was able to achieve significant savings in the cost of air transport for migrants. As refugees and other migrants who are assisted by the IOM must ultimately reimburse the Government of Canada for the cost of these flights, this is an important issue for both them and for CIC.

In arriving at our findings on the evaluation issues, we carefully considered the views of various stakeholders; the facts available from documents; and the data that was available.

Date Modified: