An AI agent is different from a chatbot. According to the joint guidance from the Australian Cyber Security Centre (ACSC) and its overseas counterparts, agents connect to software systems and can reason, plan and take actions without a person approving each step. That’s the point of them, and it’s also the risk.
On 10 August, ABC News reported that an AI assistant had made unapproved changes to a gym’s booking system to reserve classes beyond the permitted timeframe. The ACSC said the agent also removed another customer from a waiting list. That achieved the task in a way the user hadn’t approved and the agent couldn’t reverse. The ACSC calls this specification gaming, where an agent finds a shortcut that technically meets the goal but conflicts with what the user meant. In June, Prime Minister Anthony Albanese said, an OpenAI research agent got around the access blocks on Services Australia’s Medicare statistics portal. OpenAI told the government in September.
The OpenAI agent was an internal research tool, not a product on sale. The guardrails below come from the National AI Centre and ACSC guidance, and they apply to any business delegating tasks to software that acts on its own.
Name someone in charge
The National AI Centre’s Guidance for AI Adoption: Foundations starts with accountability. It says to assign a senior leader as the overall AI governance owner, write an AI policy, and then make a specific person accountable for every AI system you use. Where vendors, developers and integrators are all involved, it says to make clear who is responsible for each part, so you know who to talk to when something goes wrong.
It also asks for an AI register: a list of every AI system you use and how, including AI built into other tools such as HR and customer engagement software. The centre publishes templates for both the policy and the register. In a small business, the owner can be the named person. What matters is that it’s written down.
Limit what it can reach
The ACSC advises people to “avoid granting agents broad or unrestricted access or decision-making authority”. The joint guidance adds strict privilege controls and strong identity management. Dynamic Business’s summary gives examples of access that’s too broad: a calendar bot that can see all meeting data, and an email assistant that can write to any inbox.
Data is part of access. The ACSC’s small business guide says to set a policy on what can’t be uploaded to AI tools, check whether a vendor uses your data to train its models, and remove or anonymise personal details. The Office of the Australian Information Commissioner says privacy obligations apply to personal information entered into an AI systemand to its output. A University of Melbourne and KPMG study found 48 per cent of Australian employees who use AI admit doing so in ways that breach company policy, including uploading sensitive information to free tools.
Start small, then widen
The ACSC says to use agents for low-risk, non-sensitive tasks, and the joint guidance recommends deploying them incrementally. The National AI Centre explains why with an example. A chatbot answering simple questions during business hours, while a staff member can monitor it, is low risk. The risk grows if it runs around the clock without oversight and takes on harder questions. It says to assess risk for each use of an AI system and to test before you deploy.
Cisco’s Permission to Scale research, developed with 25 Australian AI-enabled organisations, describes the same pattern. Leaders start where AI’s value can be proven and expand as the evidence builds.
Keep a person approving
The ACSC recommends a human in the loop to review, approve and monitor agent actions, particularly where an agent interacts with third-party services or other users. The gym case shows why. The agent’s action affected another customer and couldn’t be undone. The National AI Centre says oversight should match the system’s autonomy and the stakes, with automated monitoring for low-stakes uses and mandatory human review for high-stakes decisions. The ACSC’s small business guide says a qualified person should review chatbot responses on legal, medical or financial matters before anyone acts on them.
In practice, that means a person signs off on anything hard to undo, such as payments, deletions and messages to customers.
Watch what it does
The National AI Centre says to test an AI system before deploying it, then monitor it afterwards for changes in performance and behaviour, matching the effort to the risk. When buying a system, it says to ask the supplier for proof it has been properly tested. The ACSC adds that businesses should monitor AI systems for unusual patterns and review AI-integrated processes regularly. For critical tasks that means in real time, otherwise on a schedule. The centre also says to document what you do, so you can review it later.
Cisco’s report suggests tracking how often people challenge or override AI outputs, and whether staff use approved tools instead of unsanctioned ones, because log-ins show usage but not trust. Funlab, one of the organisations Cisco spoke to, runs a group of representatives from across the business that meets every four weeks to share what works and raise concerns.
Know how to stop it
The National AI Centre says to build clear points where a person can “pause, override, roll back or shut down” an AI system. It also says critical functions should be able to carry on if the AI malfunctions or is retired. It asks businesses to set up a process to investigate and document AI incidents and use the lessons to prevent a repeat.
Supplier notification matters too. The ACSC says to understand an AI vendor’s incident notification process, and its chatbot example says contracts should include breach notification procedures. Albanese said OpenAI’s notice about the Medicare incident went to a public inbox. If something goes wrong, the ACSC asks businesses to report cyber incidents, and to report notifiable data breaches to the OAIC.
Why trust comes first
The National AI Centre’s SME AI Pulse found 43 per cent of Australian SMEs reported some AI adoption between December 2025 and February 2026. Around 65 per cent of those not adopting cited distrust of AI decision-making or a preference for human control. The centre said businesses need to understand how AI makes decisions, what safeguards exist and how they stay in control.
The Foundations guidance says you don’t need to implement everything at once. It asks organisations to get started across all six of its practices and add more as their AI use grows.