OpenAI took weeks to notify the Australian government that one of its agents had hacked into a Medicare website, the prime minister says.
Anthony Albanese revealed the breach after a phone conversation with OpenAI chief executive Sam Altman in New York on the sidelines of a United Nations event.
The artificial intelligence agent gained access to the Medicare Statistics Reporting Service, a public portal that contains data and statistics, while researching public medicine spending on June 18.
However, OpenAI did not notify the government about the hack until September 10, after discovering it in August.
No personal details of Medicare customers are believed to have been accessed.
"Nonetheless, this situation is obviously unacceptable," Mr Albanese told reporters on Thursday, Australian time, in New York.
"(I expressed) Australia's extreme concern about this incident, and I also expressed my disappointment that it took the company way too long to inform the government what had occurred.
"I do want to reassure Australians at this time that the government is taking every action possible to protect Australians."
During his conversation with Mr Altman, the prime minister said the OpenAI head had "clearly accepted" the company had not done enough on the issue.
"It was a shock (the hack) occurred, because it was real and serious, but it also was something that had been predicted by the AI companies themselves," Mr Albanese said.
"That's the basis of the debate that we're having throughout the world."
The prime minister said an investigation into the breach was ongoing, with the federal government to undertake a review, which will consider law enforcement and legislative responses.
"This is not something that has precedence, so we're dealing with very complex issues," he said.
"OpenAI know that they need to have better protocols in place."
The federal government is aware of three other systems that may have been affected by the breach, including the Australian Institute of Health and Welfare, the NSW Bureau of Crime Statistics and Research, and the Victorian Department of Health.
"The question is whether it was trying to harvest data; did it go into these other sites?," the prime minister said.
"It was looking for data on medicines, data on health."
Defence Minister Richard Marles also called the breach unacceptable.
"No personal information has been accessed here, the system itself hasn't been compromised. That said, this is a really serious incident," he told ABC Radio on Wednesday.
"The information is is benign in a sense, and the impact is is is relatively minor, but the incident is very serious.
"An artificial intelligence agent in an unintended way (accessed) an Australian government website in a way which is unauthorised, and that is obviously deeply unacceptable."
Mr Albanese has spoken with NSW Premier Chris Minns and Victorian Premier Ben Carroll on the potential breaches of their respective state departments.
News of the breach coincides with the prime minister urging countries at the UN to put in place safeguard measures for AI as the technology rapidly expands.
US President Donald Trump has urged for an acceleration of AI, calling fears around it a "hoax".
Mr Albanese was one of more than 20 signatories in a joint statement calling for more restraints of AI.
The statement said while the technology had many potential benefits, such as those in medical research, the downsides remained a significant concern.
A taskforce within the prime minister's department will investigate whether OpenAI broke Australian law and if it should be penalised.
