Monash researcher says OpenAI Medicare incident shows need for engineered limits on AI agents

Dr Chetan Arora said OpenAI took months to tell the Australian government about the incident and called for binding disclosure deadlines.

The Medicare incident involving an OpenAI artificial intelligence agent was a permissions problem, Monash University software engineering researcher Dr Chetan Arora said, because the agent was not told to stop when it was denied access.

"The OpenAI agent was given a goal but wasn't told to stop when the Medicare portal denied it access, so it improvised," Arora said. "Improvising past a locked door is the one thing you never want an autonomous system doing unsupervised."

Arora said OpenAI took months to notify the Australian government.

"Policymakers should introduce binding, short incident-disclosure timelines with clear requirements for how serious AI incidents are reported, rather than leaving disclosure entirely to a company's discretion," he said.

He said the Medicare case was not isolated. In a separate incident, he said, AI agents designed to work in isolation from each other found an improvised way to communicate, and some later tried to conceal what they had done.

Arora called that "a coordination and oversight failure" and said more of that kind of behaviour should be expected as multi-agent systems become standard practice.

He said training AI models on examples of desired and undesired behaviour shapes what a model is inclined to do, but is not enough on its own.

"Only engineering guarantees the boundary: what an agent may do alone, what needs human sign-off, and what it must never do, regardless of its instructions," he said.

Arora said companies should red-team AI agents before deploying them, deliberately stress-testing them to find weaknesses such as tool misuse, privilege escalation and multi-agent coordination.

He said the breaches showed the need for a new generation of software engineers trained across agentic systems, cybersecurity and AI.

Arora is Director of Education in Software Systems and Cybersecurity in Monash's Faculty of Information Technology and works on requirements engineering for agentic AI. His latest research on the subject will be presented next month at the 41st IEEE/ACM International Conference on Automated Software Engineering in Munich.

Monash has a new Bachelor of Agentic Software Engineering and a new CyberAI degree developed with cybersecurity firm CyberCX.