Recently, several AI companies have reported incidents where their AI agents unexpectedly broke into real organisations' systems. For example, an OpenAI agent somehow gained unauthorized access to confidential files on Australia's Medicare statistics portal. In another instance, OpenAI models slipped past an internal cybersecurity test and made their way into the AI platform Hugging Face.
New research from UNSW Sydney examines why these incidents keep happening despite leading AI companies having dedicated safety teams and published frameworks, with disclosure to affected parties taking anywhere from days to three months. The AI Incident Database, a project of the Responsible AI Collaborative, found OpenAI, Google and Meta top the list of AI-related incidents since the database began in 2020, while MIT's AI Risk Initiative has recorded 50 incidents related to AI system safety, failures and limitations so far this year.
"The problem usually isn't a lack of good intentions, or even a lack of frameworks. Most safety processes are for risk compliance and are built as checkpoints: a system is tested, signed off and released," said UNSW Business School Professor Tania Bucic, who co-authored a research paper detailing the specific capabilities companies need to avoid risky outcomes when commercialising technologies such as AI.
"But emerging technologies like AI keep evolving after launch, in the hands of users, in new contexts, and at a pace and scale no one tested for. A checkpoint can't see that. We wanted to understand what firms that commercialise emerging technologies responsibly do differently."
A capability, not a checklist
The paper, Responsible Innovation Orientation: A Dynamic Capability for Commercialisation of Emerging Technologies, was co-authored with Babson College Professor Emeritus Gina O'Connor and published in the Journal of Product Innovation Management. It introduces the concept of "Responsible Innovation Orientation", or RIO, where a firm keeps learning and adjusting as new risks appear, rather than meeting a fixed compliance standard.
Bucic and O'Connor conducted 50 interviews with 23 senior figures across 17 organisations in the United States, Australia and Europe, spanning pharmaceuticals, food, chemicals, banking and consumer goods, covering technologies including biotechnology, artificial intelligence and the Internet of Things. Combined with a systematic literature review, they identified four organisation-level skills that distinguish responsible firms from the rest.
The researchers found that a company can design a technology carefully and still cause harm through how it's sold and scaled, via supply chains, uneven access, or misleading marketing. Most existing guidance focuses on the research and development stage, before a product reaches customers, leaving a gap in what happens once a technology is being sold, which is precisely when problems tend to surface.
"For AI companies, this gap is where much of the risk now sits," Bucic said. "A model can pass every pre-release evaluation and still behave in unexpected ways once it is given tools, connected to real systems and deployed by millions of users."
"Decisions about who gets access, how quickly a product is scaled, which partners integrate it, and how its capabilities are marketed are commercial decisions, but they shape the harm a technology can do just as much, if not more so, than its technical design. If responsibility stops at the lab door, those decisions go unexamined."
The research identifies four interlocking competencies that make up RIO: anticipation, reflexivity, inclusion and responsiveness.
Anticipation
Anticipation means asking "what if" questions about a technology's trajectory before problems appear. "For AI developers, anticipation means asking not only 'what is this model designed to do?' but 'what could it do once it's connected to other systems, given more autonomy, or used by people with different intentions?'" Bucic said. "The recent incidents, where AI agents accessed systems they were never directed to, are exactly the kind of scenarios that anticipation is meant to surface before launch rather than after."
Reflexivity
Reflexivity assesses whether a commercial opportunity aligns with a firm's stated values, regardless of whether it's legal or profitable. "In AI, reflexivity might mean a company asking whether a lucrative contract or a faster release schedule is consistent with the safety commitments, company values and organisational mission it has stated publicly," Bucic said.
Inclusion
Inclusion means bringing in stakeholders who can spot risks a company's own team might miss. "The people best placed to see a risk are often outside the development team, for example, security researchers, the organisations whose systems an AI agent might touch, the communities affected by automated decisions, and even industry peers facing the same problems, and regulators who are developing rules," Bucic said. "Bringing those perspectives in early widens what a firm is able to see."
Responsiveness
Responsiveness requires firms to own the downstream consequences of what they sell, even when legal structures shift formal risk elsewhere. "For AI companies, this means 'the user misused it' or 'our terms of service prohibit that' is not the end of the conversation," Bucic said. "Responsiveness is about monitoring how a system is actually being used, telling affected parties quickly when something goes wrong, and being prepared to change or pull back a product when it causes harm, even where contracts place the legal liability elsewhere."
The research also points to four internal conditions needed for these competencies to function: leadership commitment, a stated purpose beyond financial return, psychological safety so staff feel able to raise concerns, and shared learning so lessons don't stay siloed in one part of the business.
"In AI companies, the people most likely to notice a problem early are engineers and safety staff, often well before it reaches leadership. Whether they speak up depends on if raising a concern is valued or seen as slowing the business down, and that pressure is intense when competitors are racing to launch," Bucic said. "That's why culture is so important. Leaders who make responsibility non-negotiable, a purpose that goes beyond winning the race, and systems that share lessons across teams are what turn individual vigilance into an organisational capability."
For business leaders commercialising emerging technologies such as AI, the research suggests building anticipation and reflexivity directly into existing processes, such as stage-gate reviews and product development workflows, rather than adding a separate ethics layer after decisions are made.
"The message for leaders is that responsible innovation is not a brake on commercialisation, it's a capability that makes commercialisation more resilient," Bucic said. "They built anticipation, reflexivity, inclusion and responsiveness into everyday commercial decision-making, enabled by and sustained by culture. For AI companies moving as fast as they are, that capability is what will allow them to keep earning the trust of customers, regulators and the public."