Photograph: Kylie Cooper/Reuters
The recent panic about a breach of Medicare computer security by an “AI agent” contrasts sharply with other recent cases such as the Telstra and Optus outages that left many Australians unable to reach Triple Zero. In those cases, no one blamed the computers involved. The mistakes were clearly sheeted home to the corporations that operated them.
This wasn’t always the case. When the term “artificial intelligence” was coined some 70 years ago, the first mainframe computers (absurdly primitive by modern standards) were viewed with the same awe and concern as the AI agents of the present day. There were even “algorithms” (though the term wasn’t used in that way at the time) that were supposed to pick ideal dating matches.
Failures were inevitable, and blame-shifting became routine. “The computer made a mistake” was the 1960s equivalent of “your email must have gone to junk”. Gradually, however, we realised that the problem was not with the computer but with incorrect information fed into it or badly written programs invoked as a result.
We need to make a similar adjustment when we discuss AI “agents”. If someone enters a prompt like “find Australian medicine statistics” into a program like ChatGPT or Claude, and the result is a breach of Medicare’s site, the responsibility does not lie with a piece of code. Either the human who entered the prompt or the corporation producing the code made a mistake, and they should be held liable for the resulting damages. If it’s impossible to work out who is at fault, liability should be joint and several – that is, both are liable for the full amount of the same loss, and the cost can be allocated between them.
Fixing the problems of agentic software won’t be easy. The frequency with which early computer programs malfunctioned made “debugging” (a term predating its use in computing) an essential part of information technology. Bugs might be found in the operating system, the program itself or the information fed into it. In one case, the problem was a literal bug: a moth that got caught in the relays. But with enough determination the source of the problem could usually be found and fixed.
Traditional debugging is much more difficult with agentic programs. It may be possible, after the fact, to work out what the program has done. But it’s impossible to inspect the hundreds of billions of parameters in a large agentic model and work out why the agent did it.
In this context, the idea of constraining the program with “guardrails” or “harnesses” is naive in the extreme. The whole point of telling a computer program to perform a task is to get around obstacles to that task. And with no understanding of the internal workings, external constraints will be treated as obstacles.
In most cases, the only solution will be to abandon many of the capacities that are supposed to make agents useful, such as the ability to log in to sites using passwords or to make payments on the user’s behalf.
This will be a huge wrench for corporations that have grown up with the Silicon Valley ethics of “move fast and break things” and “ask for forgiveness, not permission”. They are far more comfortable talking about “hallucinations” and “rogue agents” than about their own responsibility for programs that produce massive errors and real-world damage.
But once AI corporations are made to bear the financial consequences of their reckless negligence, a different kind of calculus will come into play. Instead of thinking, “what cool thing can we make this program do”, the first question will be, “what could go wrong if we let it run”.
Forcing corporations to bear liability for damage caused by their actions will, in all probability, drastically slow the “hyperscaling” rush to produce more and increasingly powerful agentic software. That’s a good thing for the environment as well as the economy.
And none of this will preclude the many benign uses of “AI” software, including massively improved internet search, document summarisation and translation, and software coding. These uses come with the adjustment problems that always arise with new technology, eliminating some jobs while creating others, and so on. But there is no reason to fear that they will accidentally cause a nuclear holocaust or even drain our bank accounts.
• John Quiggin is a professor at the University of Queensland’s School of Economics
