The DADM’s Noticeable Silence: Clarifying the Human Role in the Canadian Government’s Hybrid Decision-Making Systems[i]

I wrote this Op-Ed for my Law 432.D course titled ‘Accountable Computer Systems.’ This blog will likely be posted on the course website but as I am presenting on a few topics related, I wanted it to be available to the general public in advance. I do note that after writing this blog, my more in-depth literature review uncovered many more administrative lawyers talking about accountability. However, I still believe we need to properly define accountability and can take lessons from Joshua Kroll’s work to do so.

 

Introduction

Canadian administrative law, through judicial review, examines whether decisions made by Government decision-makers (e.g. government officials, tribunals, and regulators) are reasonable, fair, and lawful.[i]

Administrative law governs the Federal Court’s review of whether an Officer has acted in a reasonable[ii] or procedurally fair[iii] way, for example in the context of Canadian immigration and citizenship law, where an Officer has decided to deny a Jamaican mother’s permanent residence application on humanitarian and compassionate grounds[iv] or strip Canadian citizenship away from a Canadian-born to Russian foreign intelligence operatives charged with espionage in the United States.[v]

Through judicial review and subsequent appellate Court processes, the term accountability has yet to be meaningfully engaged with in Canadian administrative case law.[vi] On the contrary, in computer science accountability is quick becoming a central organizing principle and governance mechanism.[vii] Technical and computer science specialists are designing technological tools based on accountability principles that justify its use and perceived sociolegal impacts.

Accountability will need to be better interrogated within the Canadian administrative law context, especially as Government bodies increasingly render decisions utilizing computer systems (such as AI-driven decision-making systems) [viii] that are becoming subject to judicial review.[ix]

An example of this is the growing litigation around Immigration, Refugees and Citizenship Canada’s (“IRCC”) use of decision-making systems utilizing machine-learning and advanced analytics.[x]

Legal scholarship is just starting to scratch the surface of exploring administrative and judicial accountability and has done so largely as a reaction to AI systems challenging traditional human decision-making processes. In the Canadian administrative law literature I reviewed, the discussion of accountability has not involved defining the term beyond stating it is a desirable system aim.[xi]

So, how will Canadian courts perform judicial review and engage with a principle (accountability) that it hardly knows?

There are a few takeaways from Joshua Kroll’s 2020 article, “Accountability in Computer Systems” that might be good starting points for this collaboration and conversation.

 

Defining Accountability – and the Need to Broaden Judicial Review’s Considerations

Kroll defines “accountability” as a “a relationship that involves reporting information to that entity and in exchange receiving praise, disapproval, or consequences when appropriate.”[xii]

Kroll’s definition is important as it goes beyond thinking of accountability only as a check-and-balance oversight and review system,[xiii] but also one that requires mutual reporting in a variety of positive and negative situations. His definition embraces, rather than sidesteps, the role of normative standards and moral responsibility.[xiv]

This contrasts with administrative judicial review, a process that is usually only engaged when an individual or party is subject to a negative Government decision (often a refusal or denial of a benefit or service, or the finding of wrongdoing against an individual).[xv]

As a general principle that is subject to a few exceptions, judicial review limits the Court’s examination to the ‘application’ record that was before the final human officer when rendering their negative decision.[xvi] Therefore, it is a barrier to utilize judicial review to seek clarity from the Government about the underlying data, triaging systems, and biases that may form the context for the record itself.

I argue that Kroll’s definition of accountability provides room for this missing context and extends accountability to the reporting the experiences of groups or individuals who receive the positive benefits of Government decisions when others do not. The Government currently holds this information as private institutional knowledge, with fear that broader disclosure could lead to scrutiny that might expose fault-lines such as discrimination and Charter[xvii] breaches/non-compliance.[xviii]

Consequentially, I do not see accountability’s language fitting perfectly into our currently existing administrative law context, judicial review processes, and legal tests. Indeed, even the process of engaging with accountability’s definition in law and tools for implementation will challenge the starting point of judicial review’s deference and culture of reasons-based justification[xix] as being sufficient to hold Government to account.

 

Rethinking Transparency in Canadian Administrative Law

Transparency is a cornerstone concept in Canadian administrative law. Like accountability, this term is also not well-defined in operation, beyond the often-repeated phrase of a reasonable decision needing to be “justified, intelligent, and transparent.”[xx] Kroll challenges the equivalency of transparency with accountability. He defines transparency as “the concept that systems and processes should be accessible to those affected either through an understanding of their function, through input into their structure, or both.”[xxi] Kroll argues that transparency is a possible vehicle or instrument for achieving accountability but also one that can be both insufficient and undesirable,[xxii] especially where it can still lead to illegitimate participants or lead actors to alter their behaviour to violate an operative norm.[xxiii]

The shortcomings of transparency as a reviewing criterion in Canadian administrative law are becoming apparent in IRCC’s use of automated decision-making (“ADM”) systems. Judicial reviews to the Federal Court are asking judges to consider the reasonableness, and by extension transparency of decisions made by systems that are non-transparent – such as security screening automation[xxiv] and advanced analytics-based immigration application triaging tools.[xxv]

Consequently, IRCC and the Federal Court have instead defended and deconstructed pro forma template decisions generated by computer systems[xxvi] while ignoring the role of concepts such as bias, itself a concept under-explored and under-theorized in administrative law.[xxvii] Meanwhile, IRCC has denied applicants and Courts access to mechanisms of accountability such as audit trails and the results of the technical and equity experts who are required to review these systems for gender and equity-based bias considerations.[xxviii]

One therefore must ask – even if full technical system transparency were available, would it be desirable for Government decision-makers to be transparent about their ADM systems,[xxix] particularly with outstanding fears of individuals gaming the system,[xxx] or worse yet – perceived external threats to infrastructure or national security in certain applications.[xxxi] Where Baker viscerally exposed an Officer’s discrimination and racism in transparent written text, ADM systems threaten to erase the words from the page and provide only a non-transparent result.

 

Accountability as Destabilizing Canadian Administrative Law

Adding the language of accountability will be destabilizing for administrative judicial review.

Courts often recant in Federal Court cases that it is “not the role of the Court to make its own determinations of fact, to substitute its view of the evidence or the appropriate outcome, or to reweigh the evidence.”[xxxii] The seeking of accountability may ask Courts to go behind and beyond an administrative decision, to function in ways and to ask questions they may not feel comfortable asking, possibly out of fear of overstepping the legislation’s intent.

A liberal conception of the law seeks and gravitates towards taxonomies, neat boxes, clean definitions, and coherent rules for consistency.[xxxiii] On the contrary, accountability acknowledges the existence of essentially contested concepts[xxxiv] and the layers of interpretation needed to parse out various accountability types,[xxxv] and consensus-building. Adding accountability to administrative law will inevitably make law-making become more complex. It may also suggest that judicial review may not be as effective as an ex-ante tool,[xxxvi] and that a more robust, frontline, regulatory regime may be needed for ADMs.

 

Conclusion: The Need for Administrative Law to Develop Accountability Airbags

The use of computer systems to render administrative decisions, more specifically the use of AI which Kroll highlights as engaging many types of accountability,[xxxvii] puts accountability and Canadian administrative law on an inevitable collision course. Much like the design of airbags for a vehicle, there needs to be both technical/legal expertise and public education/awareness needed of both what accountability is, and how it works in practice.

It is also becoming clearer that those impacted and engaging legal systems want the same answerability that Kroll speaks to for computer systems, such as ADMs used in Canadian immigration.[xxxviii] As such, multi-disciplinary experts will need to examine computer science concepts and accountable AI terminology such as explainability[xxxix] or interpretability[xl] alongside their administrative law conceptual counterparts, such as intelligibility[xli] and justification.[xlii]

As this op-ed suggests, there are already points of contention, (but also likely underexplored synergies), around the definition of accountability, the role of transparency, and whether the normative or multi-faceted considerations of computer systems are even desirable in Canadian administrative law.

 

References

[i] Government of Canada, “Definitions” in Canada’s System of Justice. Last Modified: 01 September 2021. Accessible online <https://www.justice.gc.ca/eng/csj-sjc/ccs-ajc/06.html> See also: Legal Aid Ontario, “Judicial Review” (undated). Accessible online: <https://www.legalaid.on.ca/faq/judicial-review/>

[ii] The Supreme Court of Canada in Canada (Minister of Citizenship and Immigration) v. Vavilov, 2019 SCC 65 (CanLII), [2019] 4 SCR 653, <https://canlii.ca/t/j46kb> [“Vavilov”] set out the following about reasonableness review:

[15] In conducting a reasonableness review, a court must consider the outcome of the administrative decision in light of its underlying rationale in order to ensure that the decision as a whole is transparent, intelligible and justified. What distinguishes reasonableness review from correctness review is that the court conducting a reasonableness review must focus on the decision the administrative decision maker actually made, including the justification offered for it, and not on the conclusion the court itself would have reached in the administrative decision maker’s place.

[iii]The question for the Court to determine is whether “the procedure was fair having regard to all of the circumstances” and “whether the applicant knew the case to meet and had a full and fair chance to respond”.  See: Ahmed v. Canada (Citizenship and Immigration), 2023 FC 72 at […]