There is a decision being made about you in this box.
But you're not allowed to look inside — and even if you could, it wouldn't tell you much.
There are countless black boxes, just like this one, making decisions about your online and offline lives.
Some of them are pretty benign, like recommending what movie you should watch next.
But others decide the news you see, who you go on a date with and how much money you can borrow.
They also determine whether you get a job, if you're a potential shoplifter and what route you should take to the shops.
In extreme cases, they have been used to 'predict' teenage pregnancies and cut welfare entitlements for people with disabilities.
These boxes all contain algorithms, the decision-making machines that are creeping into areas of increasing consequence to all of our lives.
It can feel pretty impossible to understand exactly what these algorithms are doing, let alone keep them in check.
However, as you'll see, there are creative and powerful ways to shine a light into these black boxes.
The trouble is, we can only use them if their owners — mostly corporations and governments — will let us.
Centrelink's Robodebt algorithm was locked away inside one of these black boxes.
Hidden from public view, it went about its work, sending out hundreds of thousands of miscalculated debt notices — all based on a flawed assumption.
The algorithm divided the full year’s worth of welfare recipients' income evenly between the 26 fortnights in the year, rather than considering each fortnight individually.
A mundane miscalculation that, when reproduced faithfully and behind closed doors, had consequences that were anything but.
"I was literally crushed. I was in shock," reads one submission to a Senate inquiry. It tells the story of just one of the approximately 433,000 Australians who were subjected to debts created by the algorithm.
"I walked around my house trying to deny the reality of what had happened … I was confused as to how I owed this amount of money. Within weeks, I began receiving calls, texts and letters from a debt-collection agency."
Following a successful legal challenge and extensive media coverage, the government has launched a royal commission to investigate the scheme's failings.
However, even as one algorithm is brought into the light, others continue operating behind closed doors.
The Department of Home Affairs has used algorithms to assist visa processing for more than 20 years. And, with increasing demand for visas since our borders re-opened, this is set to expand further.
A departmental spokesperson confirmed that Home Affairs was considering "a range of emerging technologies" as part of a "modernisation" strategy.
Despite the lessons learned from the Robodebt crisis, these black boxes have, so far, remained firmly shut.
The ABC asked Home Affairs questions about transparency, monitoring, testing and redress mechanisms regarding its algorithmic systems, but these were not addressed in their response.
Visas in a box
To get a glimpse of exactly how using algorithms to assist with visa decisions can go wrong, we only need to look at how the Home Office managed a similar scheme in the United Kingdom.
The inner workings of the UK's visa-processing algorithm had been kept secret from the public for years, locked away inside a black box.
But documents released under freedom of information laws, following a legal challenge, finally allowed us to peek inside.
While they don't give the full picture, they reveal a few key things.
The algorithm sorted visa applicants into three risk groups — high, medium and low — based on a number of factors that included their nationality.
This categorisation had a big impact on the likelihood of an application being refused.
At one processing centre, less than half of applications classed as high risk were approved for visas in 2017, compared to around 97 per cent for low-risk ones.
Applicants in the high-risk category could be subjected to "counter-terrorism checks and DNA testing" by immigration officials before a decision was made on their visa.
This intense scrutiny contributed to the high refusal rates.
Meanwhile, those classed as low risk got by with routine document checks and, therefore, far-lower refusal rates.
This alone wasn't particularly controversial — after all, the UK Home Office was trying to make the best use of its limited resources.
The trouble was that, just like Robodebt, their algorithm had an insidious flaw. Over time, it unfairly amplified biases in the data.
To see how, let's imagine that 200 people, split evenly between two made-up countries — Red and Blue — apply for visas in 2015.
We're going to simulate how things will play out for applicants from these two nations, incorporating details from how the UK Home Office’s algorithm worked.
As the applications from the two nations stream down into the categories, some visas are approved and some are refused.
The rates at which they are refused is dependent on which category they fall into.
To decide how to categorise each application, the algorithm used by the UK Home Office relied on historical "immigration breaches" data.
We've given the Red group a slightly higher rate of historical breaches to simulate the fortunes of two similar — but not identical — nations.
The refusal rates for our two nations (shown at the bottom) reflect this difference in their historical records.
Okay, we're done for 2015.
Of our 200 applications, the Red group had 11 more refusals than the Blue group.
The results for 2015 are pretty close to the historical data that we made up, so it seems like our algorithm is doing its job so far.
Now, we feed these results back into the algorithm.
This is where things start to get ugly. The UK Home Office's algorithm counted merely having a visa refused as a "breach", which led to biases in the historical data being exacerbated.
Fast forward to 2016 and another 200 people from the same two groups apply.
Based on the prior "breaches", our algorithm flags a higher proportion of people from the Red group as high risk than the previous year.
And it's solely the algorithm that's increasing this disparity — there's nothing different about the applicants themselves compared to the year before.
The extra scrutiny placed on Red applications results in 18 more being refused than the Blue one this time around.
Once again, we feed this new information back into our algorithm.
It sees an even greater disparity in the risks in 2017.
In the worldview of the algorithm, the evidence is now clear: People from the Red group are predominantly high risk while those from the Blue group are not.
This time, the Red group sees more than twice the number of refusals compared to the Blue group.
That's a pretty big difference from where we started, isn't it?
As the years rolled on, the data increasingly became a reflection of the algorithm's prejudices.
The algorithm took the differences between nations in the historical data and blew them out of proportion — regardless of whether they were accurate assessments of risk, or had been created by chance, error or discrimination.
So, by 2017, its choices were more of a self-fulfilling prophecy than an accurate reflection of risk in the real world.
Jack Maxwell — lawyer and co-author of Experiments in Automating Immigration Systems — found through his investigations that the UK Home Office's algorithm suffered from a feedback loop much like this one.
And, according to Mr Maxwell, the historical immigration data was flawed too.
By their nature, he said, immigration enforcement statistics were incomplete, and did not "reflect the actual incidence of immigration breaches, so much as the biases of the people reporting those breaches".
Now, there's no indication that the Australian Department of Home Affairs is making, or about to make, the same mistakes as the UK Home Office as it expands and "modernises" its use of algorithms.
However, as long as it keeps its algorithms locked away, we can't be sure.
Fortunately, onerous legal challenges and FOI requests are not the only ways to peer inside.
As we'll see, the tools that can open these black boxes come in a range of shapes and sizes.
Some can help us understand — and challenge — the decisions they make about us, as individuals, while others can illuminate bias and discrimination embedded within a system.
Thinking outside the box
To explain the decisions made by algorithms in a way that humans can understand, leading artificial intelligence researcher Sandra Wachter and her colleagues at the Oxford Internet Institute turned, not to science, but to philosophy.
They went back to basics and "thought outside the box" about what an explanation actually is, and what makes one useful.
"Do people really want to understand the internal logic of an algorithm? Or do they mainly care about why they didn't get the thing that they applied for?" Professor Wachter ponders.
The philosophy textbooks told Professor Wachter that it was more the latter.
"They might want to contest the decision because the criteria that [the algorithm] relied upon is wrong. Or it might be the case that the decision was made correctly, and they want guidance on how to change their behaviour in the future," she says.
Given these goals, simply looking inside these black boxes is not going to tell us what we want to know.
This is because, in practice, most algorithms are a combination of complex variables.
Not even the experts can reliably interpret decisions made by sophisticated algorithms.
So, rather than trying to explain the nitty gritty technical details of how they work, Professor Wachter and her team came up with a deceptively simple alternative.
The idea was to describe "how the world would need to be different, for a different outcome to occur", she explains.
That idea — of imagining alternative worlds — may sound like it belongs in a science fiction writers' room, but the way this potential tool for increasing algorithmic accountability works is really quite simple.
Such a tool would generate a number of "nearby possible worlds" in which your application would have been successful — and tell you how they differ from the real world.
This means you might be told, in plain English, that you'd have been successful had you applied for a different type of visa or requested a shorter stay.
So, you wouldn't need to look inside the box at all to understand how it came to its decision and, therefore, how you could do better next time.
By offering this kind of transparency without opening the black box, Professor Wachter adds, it will be "protecting the privacy of others and with very little risk of revealing trade secrets".
While this "nearby possible worlds" approach is useful and important for understanding specific decisions about one individual, it's not really enough on its own to keep these black boxes in check.
Explanations of individual cases alone will not let us identify systemic issues such as the ones seen in the UK.
Thinking bigger
An individual often will not know how others are faring when they interact with an algorithm, says Paul Henman, a professor of digital sociology and social policy at the University of Queensland.
And even in a system that discriminates against others, many individuals will still receive acceptable or even favourable decisions.
"Because individuals are experiencing these decisions in isolation, they might not see that an entire group is getting a different outcome."
While some algorithms — such as the one used by the UK Home Office — explicitly discriminate based on nationality or other attributes protected by law, discrimination is not always so black and white.
Factors such as the applicant's immigration background, location and even their name are not protected attributes, but can correlate closely with race.
As these structural biases cannot be seen at the level of individual decisions, we need to think bigger.
This is where our second transparency tool — the algorithmic audit — comes in.
An algorithmic audit involves putting the algorithm under the microscope to verify that it meets standards for fairness.
In the case of the UK Home Office, an expert could have checked that people of all nationalities saw comparable outcomes when controlling for other factors.
The same goes for gender, age and other protected attributes.
Results from algorithmic audits can be translated into scores and made public, similar to the health advice that is required on food packaging.
When published, these results can help us to understand what's going on inside the box, without us all needing to be experts.
These tools — and others like them — are not limited to academia anymore. They're being adopted in Australia and around the world.
So, why are we not seeing greater transparency around the algorithms used by corporations and our governments?
The right to reasons
One reason, says former Australian human rights commissioner Ed Santow, is that Australia is lagging other parts of the world on digital rights protections.
In 2021, Professor Santow and the Australian Human Rights Commission made a number of recommendations about how Australia can make automated decision-making "more accountable and protect against harm".
"The cost of inaction [on digital rights] is waiting until the next crisis, the next Robodebt. At the Human Rights Commission, we were saying that we could avoid that next crisis by putting in place stronger fundamental protections," he said.
According to the commission's report, the foundational "right to reasons" would be a legislated right to request an explanation for administrative decisions, regardless of whether they were made by humans or machines.
These protections can be the difference between a problematic algorithm caught early and another crisis identified too late.
Both Robodebt and the UK Home Office algorithm flew under the radar for years before their flaws became apparent, in part due to the lack of transparency around how they operated.
Centrelink sent out erroneous debt notices without equipping those recipients with the tools necessary to challenge or even understand those decisions. Instead, they needed courts and advocates to find justice.
The story is similar in the UK. It took the efforts of Foxglove, a tech advocacy group, and the Joint Council for the Welfare of Immigrants to challenge the Home Office's algorithm in court.
However, it doesn't have to be this way.
Specialised tools like "nearby possible worlds" and algorithmic audits make these explanations more practical than ever to produce.
And the European Union has been blazing a trail in digital rights protections, so there is plenty of precedent for our legislators to learn from.
Having our fates decided by algorithmic black boxes can feel pretty dystopian.
However, if we embrace these tools and legislate the necessary protections, we might at least live in a world where the algorithms have to work in the open.
Credits
- Reporter and developer: Julian Fell
- Designer: Ben Spraggon
- Editor: Matt Liddy
