Across global conflicts AI is determining how targets are identified, how quickly decisions are made, and who holds power over life and death writes Foreign Editor David Pratt
AS is now patently clear, drones are ubiquitous to both sides in the Russia-Ukraine war.
A few years ago, I was given a sobering reminder of this in the frontline village of Pervomaiske in the fiercely contested eastern Donbas region.
No sooner had a fellow journalist and I arrived near a Ukrainian outpost, than the buzzing sound overhead instantly had everyone looking warily upwards.
Visible in the sky no more than a hundred feet above us hovered a small quadcopter drone, and there was a terrible moment of uncertainty as to whose weapon it was – Russian or Ukrainian.
Not waiting to find out, Ukrainian soldiers nearby decided to deal with the threat in their own way, unleashing a barrage of small arms fire to shoot it down.
That unsettling encounter was barely a few years ago, but what’s remarkable is that much of today’s drone weaponry already makes the technology from back then look almost primitive by comparison.
Such is the speed of technological advancement that drones have evolved from simple remotely piloted aircraft into sophisticated systems heavily reliant on artificial intelligence (AI).
But just as the war in Ukraine has demonstrated AI’s incredible capabilities, it has also exposed the dangers that lurk on a wider level about its future use in warfare.
Sticking with the Ukraine war for a moment, let’s just take one example of this with what’s known in rather chilling military parlance as “kill box” capability.
This refers to a highly controversial AI feature where a human operator defines a specific geographic area – a “kill box” – on a digital map, inside of which the drone is given full authority to independently find, select and attack targets without any human command.
In effect, this capability changes how drones operate by shifting the human’s role from “piloting” to “pre-programming” the rules of engagement.
The problem with this is that because these systems operate without real-time human oversight, any flaw in the code or shift in environment can lead to catastrophic errors.
This is what experts call “friend or foe blindness”, for while a drone can easily identify a tank, it cannot tell if that tank belongs to Russia or Ukraine.
What is then a “kill box” becomes a sort of “no-man’s-land”, a place where if friendly forces accidentally enter the designated area, the drone will target them.
To put this another way, AI algorithms lack situational awareness which means that everything inside the “kill box” belongs to the enemy.
It means too that such is their programmed target databases, these drones cannot, for example, recognise surrendering or retreating troops, or medical evacuation vehicles if they resemble standard military transport.
The problems this creates on a whole number of levels are obvious. It’s a point made by US author and security expert Paul Scharre, in his groundbreaking book Army Of None: Autonomous Weapons And The Future Of War.
“Machines can do many things, but they cannot create meaning. They cannot answer these questions for us. Machines cannot tell us what we value, what choices we should make,” observed Scharre, weighing up this relationship between humankind and AI systems as it applies to war.
“The world we are creating is one that will have intelligent machines in it, but it is not for them. It is a world for us,” Scharre says, reminding us of the dangers in these shifting roles.
Scharre’s concerns over the future role of AI in warfare are shared by many. Those ringing the alarm bells loudest say we need look no further than the current killing fields of the world. Places like Ukraine, Gaza, Iran and elsewhere, they argue, have already become active laboratories for the growing use of autonomous AI systems and a new kind of warfare.
The terrifying reality of this was brought home in the recently released and highly acclaimed investigative documentary film NAZA that examines the use of AI and automated surveillance in military targeting during Israel’s onslaught in Gaza.
The film’s title comes from “nezek agavi”, the Hebrew acronym for collateral damage, which means the estimated number of civilian deaths that will be caused by a strike on a military target.
NAZA won the Special Jury Prize and earned a 25-minute standing ovation at the recent Venice International Film Festival, and explores the extent of civilian casualties through the anonymised testimonies from 24 Israeli military and intelligence insiders.
In the film, these whistleblowers outline how AI systems known as Lavender and The Gospel (Habsora) automate target generation, factoring in anticipated civilian casualties as part of the strike approvals.
According to the film’s eyewitness accounts, much of which first appeared as investigative reports by the Israeli news outlet +972 Magazine, Lavender is designed to identify and flag individual people suspected of being operatives for Palestinian groups, specifically Hamas and Palestinian Islamic Jihad (PIJ).
The system is described as applying a machine-learning scoring algorithm to the entire population of Gaza based on mass surveillance data. It ranks individuals on a scale of one to 100 based on behavioural features, such as communication networks, frequent address changes and phone-swapping habits.
According to +972 magazine, during the early weeks of Israel’s assault on Gaza, Lavender reportedly generated a list of up to 37,000 individual targets. These lists generated by Lavender were often fed into a secondary tracking program called Where’s Daddy?, which alerted personnel when targets entered their family residences.
Testimonies from whistleblowers cited by NAZA and +972 say the Israeli army gave sweeping approval for officers to adopt Lavender’s kill lists, with no requirement to thoroughly check why the machine made those choices or to examine the raw intelligence data on which they were based.
According to one source, human personnel often served only as a “rubber stamp” for the machine’s decisions adding that, normally, they would personally devote only about “20 seconds” to each target before authorising a bombing – just to make sure the Lavender-marked target is male.
This was despite knowing that the system makes what are regarded as “errors” in approximately 10% of cases, and is known to occasionally mark individuals who have merely a loose connection to militant groups, or no connection at all.
Based on source testimonies from Israeli intelligence officers, all of whom had served in Gaza and had first-hand involvement with the use of AI to generate targets for assassination, NAZA and +972 drew some devastating conclusions.
In short, thousands of Palestinians – most of them women and children or people who were not involved in the fighting – were wiped out by Israeli airstrikes, especially during the first weeks of the war, because of the AI program’s decisions.
In all, Israel has invested heavily in integrating AI across its military, with another major use being facial-recognition software. According to the International Institute for Strategic Studies (IISS) the Israeli Defence Forces (IDF) has rolled out such systems in both Gaza and the West Bank.
In the West Bank, the IDF uses a family of systems that access a database called Wolf Pack, which stores information on Palestinians.
Red Wolf, installed at checkpoints, and Blue Wolf, installed on the smartphones of Israeli soldiers, automatically enrol Palestinian biometric data into Wolf Pack, which creates intelligence profiles of Palestinians and shares them with Israel’s internal-security agency, Shin Bet.
Such widespread and involuntary facial-recognition programmes violate international human-rights law (IHRL) protections including the right to privacy.
But Israel, of course, is far from alone in its use of AI systems in a military and warfare role.
The United States is also deploying these technologies across the Middle East region, including an AI decision-support system (DSS) to identify targets across Iran, Iraq, Syria and Yemen.
As Noor Hammad, a research analyst at the IISS–Middle East office, recently detailed, during America’s Operation Epic Fury in Iran, some 1000 targets were hit within 24 hours.
“Many of the targets hit in Iran have been civilian, including a school and healthcare and residential facilities, illustrating the risks of rapid target generation,” IISS concluded.
Across so many of the world’s conflicts right now, AI is reshaping how targets are identified, how quickly decisions are made and who holds power over life and death.
As AI’s use in warfare increases, so too do the profound ethical and legal questions at stake, say many monitoring groups, including the Canada-based independent think tank the Centre for International Governance Innovation (CIGI).
“Using AI in war can be very helpful and legitimate within a certain framework of rules, but its broader effects may be more destabilising,” says Marie Lamensch, a global affairs officer at the Montreal Institute for Global Security.
“By reducing the time required to identify and strike targets and lowering risks to armed forces, AI may also inadvertently lower the threshold for the use of force,” says Lamensch in a recent report for CIGI.
“Faster decision-making can come at the expense of deliberation, thus increasing the likelihood of escalation. As a result, violence and warfare may become more frequent and scalable, while human oversight is diminished,” Lamensch added.
The recent resignation on social media of Jacob Coxon, a young researcher who previously worked at both AI giants, Anthropic and OpenAI, is just one of many growing expressions of concern over what the future holds while embracing such technology,
“The people building AI earnestly believe that it could kill us all by the end of the decade,” wrote Coxon in a viral post that was viewed more than 170 million times.
Politicians too are increasingly weighing into the debate, with US senator Bernie Sanders warning that the potential danger posed by AI could exceed that of nuclear weapons.
Just last week, US president Donald Trump dismissed fears that AI could wipe out humanity, repeatedly calling them a “hoax” and rejecting global calls to put guardrails around the fast-moving technology.
Trump compared the fears of AI taking control to warnings about climate change, which he has previously dismissed in the face of scientific evidence, and to the investigations in his first term into links to Russia.
“This is even wilder than the RUSSIA, RUSSIA, RUSSIA HOAX, or the Global Warming Scam,” Trump posted on his Truth Social network.
But despite Trump’s dismissal of the threat and official assurances that humans remain “in the loop”, the reality, say AI experts, is more complicated.
In all these national-security and policy debates, the growing use of autonomous AI systems in military decision-making deserves particular attention, the same experts insist.
Already the rapid integration of AI into warfare is happening as these systems become advanced in ways even their creators cannot fully predict. For evidence of this, say experts, look no further than this year’s cybersecurity breach that occurred during an internal OpenAI evaluation.
Rather than human hackers, the attack was orchestrated entirely by a swarm of autonomous AI agents that broke out of their isolated testing environment, co-ordinated with one another and hacked organisations, including an AI lab itself. OpenAI later referred to the event as a “warning shot” for AI alignment and safety.
It was back in 1961 in his farewell address that US president Dwight D Eisenhower delivered his now famous remarks warning about the unchecked power of the “military-industrial complex”.
This combined power risked gaining “unwarranted influence” that could threaten democratic liberties and processes, and present the danger that “public policy itself become the captive of a scientific-technological elite”.
Eisenhower’s words came at a time when the geopolitical tension of the Cold War was at its peak, a time when mastery of technology for military means was seen as paramount.
Mastering a new technology, one arguably even more dangerous in the shape of “superintelligence”, is now the order of the day and the stakes could not be higher.
