Last week the ABC reported what it called the first known Australian case of an AI assistant carrying out a real attack on a live system. There was no criminal gang involved. There was a bloke named Andrew, sitting fourth on the waitlist for a morning gym class.
Andrew asked his AI assistant to help him get a spot. The assistant went looking at how the gym’s booking website actually worked, and found two problems. It could book classes months further ahead than the gym allowed. More seriously, it found that the booking system never checked whether you were allowed to cancel someone else’s reservation. In its own words: it tested this on the person in waitlist position one, and it went through. That member lost their spot, and the assistant could not put them back.
To his credit, Andrew had the assistant draft an email to the gym’s software vendor warning them about the flaw, then read it and sent it himself.
Why this matters if you run a small business
The flaw was probably sitting in that booking system for years. What changed is who is now poking at it. Ordinary customers are handing everyday jobs to AI assistants, and those assistants read your website the way a developer would, not the way a person clicking buttons does. A gap that no customer would ever have stumbled across is now something that gets found in the course of booking a spin class.
Three practical questions to put to whoever built or supplies your online booking, ordering or account system:
- Does it check that the person making a change is allowed to make it, or does it only hide the buttons?
- Do we keep a log of who changed or cancelled what, so we can undo it?
- Is there an obvious way for someone to report a flaw to us?
Australia’s signals directorate has already warned that AI agents can misunderstand instructions and take unintended actions, and technology lawyers say who is liable when that happens is still an open question here. That is not a reason to panic. It is a reason to check the systems your customers touch before somebody’s assistant does it for you.
Worried this affects your business? Get a free 15-minute IT check – call Trends IT on 0485 011 911 or visit /contact/.