Uncategorized

Award-Winning Canadian Immigration and Refugee Law and Commentary Blog

Blog Posts

Filling in Three Missing Peer Reviews for IRCC’s Algorithmic Impact Assessments

As a public service, and transparently because I need to also refer to these in my own work in the area, I am sharing three peer reviews that have not yet been published by Immigration, Refugess and Citizenship Canada (“IRCC”) nor made available on the published Algorithmic Impact Assessment (“AIA”) pages from the Treasury Board Secretariat (“TBS”).

First, a recap. Following the 3rd review of the Directive of Automated Decision-Making (“DADM”), and feedback from stakeholders, it was proposed to amend the peer review section to require the completion of a peer review and publication prior to the system’s production.

The previous iteration of the DADM did not require publication nor specify the timeframe for the pper review.  The motivation for this was to increase public trust around Automated Decision-Making Systems (“ADM”). As stated in the proposed amendment summary at page 15:

The absence of a mechanism mandating the release of peer reviews (or related information) creates a missed opportunity for bolstering public trust in the use of automated decision systems through an externally sourced expert assessment. Releasing at least a summary of completed peer reviews (given the challenges of exposing sensitive program data, trade secrets, or information about proprietary systems) can strengthen transparency and accountability by enabling stakeholders to validate the information in AIAs. The current requirement is also silent on the timing of peer reviews, creating uncertainty for both departments and reviewers as to whether to complete a review prior to or during system deployment. Unlike audits, reviews are most effective when made available alongside an AIA, prior to the production of a system, so that they can serve their function as an additional layer of assurance. The proposed amendments address these issues by expanding the requirement to mandate publication and specify a timing for reviews. Published peer reviews (or summaries of reviews) would complement documentation on the results of audits or other reviews that the directive requires project leads to disclose as part of the notice requirement (see Appendix C of the directive) (emphasis added)

Based on Section 1 of the DADM, with the 25 April 2024 date coming, we should see more posted peer reviews for past Algorithmic Impact Assessment (“AIA”).

This directive applies to all automated decision systems developed or procured after . However,

  • 1.2.1 existing systems developed or procured prior to  will have until  to fully transition to the requirements in subsections 6.2.3, 6.3.1, 6.3.4, 6.3.5 and 6.3.6 in this directive;
  • 1.2.2 new systems developed or procured after  will have until  to meet the requirements in this directive. (emphasis added)

The impetus behind the grace period, was set out in their proposed amendment summary at page 8:

TBS recognizes the challenge of adapting to new policy requirements while planning or executing projects that would be subject to them. In response, a 6-month ‘grace period’ is proposed to provide departments with time to plan for compliance with the amended directive. For systems that are already in place on the release date, TBS proposes granting departments a full year to comply with new requirements in the directive. Introducing this period would enable departments to plan for the integration of new measures into existing automation systems. This could involve publishing previously completed peer reviews or implementing new data governance measures for input and output data.

During this period, these systems would continue to be subject to the current requirements of the directive. (emphasis added)

The new DADM section states:

Peer review

  • 6.3.5 Consulting the appropriate qualified experts to review the automated decision system and publishing the complete review or a plain language summary of the findings prior to the system’s production, as prescribed in Appendix C. (emphasis added)

Appendix C for Level 2 – Moderate Impact Projects (for which all of IRCC’s Eight AIA projects are self-classified) the requirement is as follows:

Consult at least one of the following experts and publish the complete review or a plain language summary of the findings on a Government of Canada website:

Qualified expert from a federal, provincial, territorial or municipal government institution

Qualified members of faculty of a post-secondary institution

Qualified researchers from a relevant non-governmental organization

Contracted third-party vendor with a relevant specialization

A data and automation advisory board specified by Treasury Board of Canada Secretariat

OR:

Publish specifications of the automated decision system in a peer-reviewed journal. Where access to the published review is restricted, ensure that a plain language summary of the findings is openly available.

We should be expecting then movement in the next two weeks.

As I wrote about here, IRCC has posted one of their Peer Reviews, this one for the International Experience Canada Work Permit Eligibility Model. I will analyze this (alongside other peer reviews) in a future blog and why I think it is important in the questions it raises about automation bias.

In light of the above, I am sharing three Peer Reviews for IRCC AIAs. These may or may not be the final ones that IRCC eventually posts, presumably before 25 April 2024.

I have posted the document below the corresponding name of the AIA. Please note that the PDF viewer does not work on mobile devices. As such I have also added a link to a shared Google doc for your viewing/downloading ease.

(1) Spouse Or Common-Law in Canada Advanced Analytics [Link]

A-2022-00374_ – Stats Can peer review on Spousal AI Model

Note: We know that IRCC has also been utilizing AA for Family Class Spousal-Overseas but as this is a ‘triage only’ model, it appears IRCC has not published a separate AIA for this.

(2) Advanced Analytics Triage for Overseas Temporary Resident Visa Applications [Link]

NRC Data Centre – 2018 Peer Review from A-2022-70246

Note: there is a good chance this was a preliminary peer review before the current model. The core of the analysis is also in French (which I will break down again, in a further blog)

(3) Integrity Trends Analysis Tool (previously known as Watchtower) [Link]

Pages from A-2022-70246 – Peer Review – AA – Watchtower Peer Review

Note: the ITAT was formerly know as Watchtower, and also Lighthouse. This project has undergone some massive changes in response to peer review and other feedback, so I am not sure if there was a more recent peer review before the ITAT was officially published.

I will share my opinions on these peer reviews in future writing, but I wanted to first put it out there as the contents of these peer reviews will be relevant to work and presentations I am doing over the coming months. Hopefully, IRCC themselves publishes the document so scholars can dialogue on this.

My one takeaway/recommendatoin in this context, is that we should follow what Dillon Reisman, Jason Schultz, Kate Crawford, Meredith Whittaker write in their 2018 report titled: “ALGORITHMIC IMPACT ASSESSMENTS: A PRACTICAL FRAMEWORK FOR PUBLIC AGENCY ACCOUNTABILITY” suggest and allow for meaningful access and a comment period.

If the idea is truly for public trust, my ideal process flow sees an entity (say IRCC) publish a draft AIA with peer review and GBA+ report and allow for the Public (external stakeholders/experts/the Bar etc.) to provide comments BEFORE the system production starts. I have reviewed several emails between TBS and IRCC, for example, and I am not convinced of the vetting process for these projects. Much of the questions that need to be ask require an interdisciplinary subject expertise (particularly in immigration law and policy) that I do not see in the AIA approval process nor peer reviews.

What are your thoughts? I will breakdown the peer reviews in a blog post to come.

Read More »

A Fresh New Start – Edelmann and Co.

First week officially in the books. I look around at my office, out at Victory Square through one clear and one not so clear glass window (there’s a story behind this – for another day, no less), I feel as if I have finally settled in.

IMG_6316

I am grateful to have landed in a place of compassion and learning. I love it here. From having a brilliant office manager who I can already tell is a special soul to the equally brilliant colleagues I have. Everyday is filled with discussions about the law and doing good.

This journey won’t be easy. I’m shifting from what I will readily admit to now, a “business man practicing law” to a “lawyer who happens to operate a business.” The stakes are also much higher now as is the level of knowledge and finesse required. The clients that I have started to and well be starting to see often find themselves in vulnerable places, subject to an immigration system that increasing feels turned against them. Every conversed word and every step must now be more calculated than every before – often with lives and families hanging in the balance. It is not enough to simply show a client meets the requirements of a certain permit.

Learning to push myself yet be patient at the same time will itself be a process. Beyond myself, I need to put my faith in God and those around me. My heart is ready and the mind is making it’s way there – it has to overcome.

Over the next little bit, I will revive this blog – with a little bit of law and lot of love. Like the elephant being held up by balloons that sits in my office room I hope to take the heavy burdens in stride and focus on how much support I am grateful to have.

To the mentors who helped me make this decision, I am very thankful to each and every one of you. Now – let’s get back to the grind and do it for the people.

Read More »

Looking Back at Law School…. Five Things to Think About

I realize I am writing this piece on the coattails of two of my incredible mentors who have done so earlier this year. Yet, with so many individuals asking me and emailing me about their pending decisions/LSAT woes/life choices it only make sense to address it here.

Before you start, check out the pieces of my mentors, ironically Steve and Stephen who wrote the pieces below:

1) Steve Meurrens’s very insightful look at pieces of advice for law students http://meurrensonimmigration.com/10-pieces-of-advice-before-entering-first-year/

 

2) Stephen Ngo’s fantastic piece

Alumni perspective – Steven Ngo

Both of them provide very sage advice in their piece, from – keeping open minds (Steve M) to truths on reality

Without further ado, here are five things that I would think about. You will note that I am not calling this “advice.” Advice necessarily presumes that I did something right or wrong and I have some lessons to impart because of this. I don’t believe most things in my life, anyway, to be so clearly defined. If they are, I am still searching through the dictionary on this one.

1. “What Got You Here Will Get You Out of Here” – then second-year colleague, LK

When I started law school, I had an older colleague at this time – I’ll call her “LK” who was different than all the other mentors. While most older students approached us with their war stories and exam tips, LK kind of showed that kind of quiet, confidence from afar. I remember eventually having an opportunity to chat with her. The only one real thing she said to me was – “Look, there are a lot of people who have gotten here through many different paths. Everybody learns a different way. Everybody will succeed a different way. Remember, what got you here will get you out of here.”

It was a lesson that I should have heeded earlier. My initial motivation was to pursue international law (I had applied to a Masters-IR joint program but abandoned the application at the last minute). I ended up abandoning much of the advice LK provided in pursuit of the big firm pathway. I wanted it because it was considered elite, coveted, and unknown when it fact it wasn’t aligned with who I was and what would have made more sense .

I am grateful that eventually the path led me to going back to pursue my passion for culture, people, and histories as an immigration lawyer. I am also able to advocate on behalf of those individuals who traditionally have been dis-empowered. I use this arsenal of lived experiences and past experiences every day in my work assisting new and potential immigrants.

Too many people I know abandoned elements that made them uniquely themselves. Those that did not – are pursuing (if not as a primary job) efforts in niche areas or side hustles that supplement and/or become their day jobs.

2. Law Doesn’t Kill Creativity, You Do

There’s a misnomer that law school turns everyone into robots first, a pre-evolution stage of the billable robot we later become. Looking back for me, law school was actually a time of immense creativity. I was involved and captained a volleyball called “Denning Digs Dis”, played floor hockey, spent hours observing life in a cute Korean coffee shop, and learning Indian card games.

While first-year black letter law is a required rite of passage, it is only to build the foundation of the house. Second and third year is a time for designing and sprucing the space into something more liveable. Those of my colleagues that did internships/externships, took creative opportunities starting law clubs, went on overseas internships are some of the most successful today.

For myself, taking an optional course in project management was one of those things that really made a difference I felt. I learned about an entirely new area and how it could combine itself into law. I studied what some of the Firms are doing in this regard. While I don’t use it enough (admittedly still), the idea of treating a file as a project and engaging the end-user in the process and the product were lessons that have stayed with me. I very much treat my immigration applications, especially appeals, not as a commodity to be delivered but a game plan requiring various moving pieces to align. There is so much room to be creative within a system that rewards this over just the pure time spent and billed.

Creativity is an understated part of my work. Often times the solutions I come up for clients and strategies I plan out are now presented on immigration websites or discussed openly. My creativity draws me to look at the law as not confined space but one where corners can be navigated.

3. If You Leave Law School More Humble(d), You’ve Succeeded

I was humbled by law school. Like many of you preparing to go, I got through undergrad – a few bird courses. I received some good grades. While I stumbled during the LSAT, I thought that I could write my way to some semi-decent grades.

It couldn’t be further from the truth. I realized that I struggled to synthesize large amounts of information into small principles. Looking back at my undergraduate studies that were in history, I was doing the entirely opposite thing – taking smaller concepts or unknown events and making bigger statements.

In addition to exams, I also ended up being humbled by the journey of being in a class with incredibly talented and smart individuals. There were more than a few occasions that I really had to throw cold water on my face to snap out of thinking I did not belong. This was especially true when some of my colleagues had twenty-years of professional work experiences, graduate degrees from world-class universities, and many of them had even held previous jobs as engineers and consultants with top global companies. I was a wide-eyed youngster who’s resume consisted of working front desk at sport and recreation facilities. I was a nobody but eventually I learned to embrace this as a challenge rather than a weakness.

Through the process, I learned that law was ultimately a beast that I did not tame in my three years but that it was the actual taming of the beast (process) that was the work we would be doing.

Through law I also learned to accept myself – embrace my stronger characteristics but also recognize those weaker ones. In my day to day practice today, I am still aware of these and they keep me grounded and focused. In fact, right now I would argue that the biggest plague for younger lawyers today is not having too narrow of a focus but trying to overextend themselves and grow too quick for their current capacities. The same could be said about start-up firms.

4. Understanding Power Structures, Politics, and Being a Professional Through All Of It

I was at Heenan Blaikie during the downfall. I recently had a chance to read the book written by former co-managing partner, Norm Bacal on the events.

Starting from law school, I started to realize that certain privilege was access. There were certain students who had “in’s.” Some made those “in’s” very obvious. A majority of them hid it in their day to day. I was immune to it all. I did not know any lawyers.

There is a power structure. There is a bamboo ceiling. There are politics. No matter how you want to dice it up, there will always be individuals on the outside looking in within the legal system. In law school, my reaction (admittedly a mistake) was to try and fit in. I was determined to get to a Bay Street (type) firm. I was determined to be able to chat about yachts, boat trips, and golf scores. I was determined to know which of the five forks and spoons to use, even though I grew up utilizing chopsticks.

When HB fell, the reality was the writing was on the wall for first (the students), second those lawyer’s that did not have the business case to be there, and ultimately there were a core set of individuals who made it through based on existing relationships, mentorships, and connections.

HB was a wake up call that (1) I didn’t have any connections; and (2) that I needed to start making those connections for myself.

I did not take being let go and having my articles cancelled  professionally. I struggled. I cried (I can admit this now). I had frantic calls. Looking back I overreacted. I was a self-absorbed law student who felt that I could control everything.

There are structures, power structures, politics – ultimately beyond your control. You can either let these things consume you or you can keep your head up eye – take losses like future wins, and push through.

5. A “No” Is a Pathway to a Future “Yes” – if you are Grateful for the No

Even today, I learn more from failed applications and difficult cases than I do from cases that are successful. I wish in law school I had spent more time seeking  follow-up from my exams and assignments rather than just caring about the grade.

I didn’t see it that way then but the low LSAT scores, rejected law school admissions, rejected transfer applications after first year, initial OCI rejections, and eventually the HB meltdown have all made me a more realistic lawyer.

What law firms and clients want these days is not someone who has never failed but someone who can pick up something in a very difficult situation and try your best to turn it around in a confident and resilient way. Law school will threaten to turn you from a process-based person to a solely-results based one, so you will need to do everything you can to balance the scales the other way.

Embrace the failures – embrace the lessons learned from making mistakes. These will be the toolkit for your tomorrow.

Finally, a bonus piece to consider….. I remember a fellow student sent me this back in the day. The firm (not HB) no longer […]

Read More »
About Us
Will Tao is an Award-Winning Canadian Immigration and Refugee Lawyer, Writer, and Policy Advisor based in Vancouver. Vancouver Immigration Blog is a public legal resource and social commentary.

Let’s Get in Touch

Translate »