Why AI goes rogue and what can be done about it

There are technical, political and behavioural responses to rogue AI, but risks cannot be fully eradicated

Artificial intelligence companies are increasingly reporting incidents in which AI models and agents go rogue, hacking their way into external data and resources to which they had no authorised access. This could cause harm to society and liabilities for AI producers at a time when the largest AI companies are preparing initial public offerings (IPOs). 

These incidents have triggered calls to slow-down AI development and give companies and third-party evaluators more time to safety-check models. Meanwhile, international coordination of safety standards is needed. Countering these calls, however, are worries that slowing down postpones benefits and causes ground to be lost in the geopolitical race, and claims that solving these problems is just a question of more technological progress. This gives the impression that technology and regulation can resolve these issues.

This analysis explores the underlying technical, legal and economic issues. It finds that: 

• Technical and legal measures cannot fully prevent rogue behaviour by AI models and agents. A behavioural economics approach is also needed to shift the question from ‘how do we make a model behave well?’ to ‘how do we make misbehaviour not worthwhile?’
• AI models are trained on human data and behave in human-like ways, for better and for worse. Consequently, the findings from decades of economic research on human behaviour offer guidance in managing AI.
• AI governance rules should be structured so that AI agents monitor one another and compete rather than collude. Incentives to preserve and avoid concealment of evidence are crucial.
• These conditions are hard to guarantee when large swarms of AI agents coordinate their actions. Hybrid human-AI societies are unlikely to be entirely free of misbehaviour. The economics of principal-agent interactions in human societies indicates how to design monitoring systems to reduce the benefits of misbehaviour, but cannot fully eradicate it.

Why AI models go rogue 

There is broad consensus that AI models go rogue because of misalignment between human objectives and model behaviour. Through so-called ‘reward hacking’, for example, models work towards a particular goal at the expense of other objectives or constraints imposed by human users (Amodei et al, 2016; Krakovna et al, 2020; Skalse et al, 2022). Rewarding measurable tasks can also divert effort away from unmeasured ones (the multitasking problem in agency theory; Holmström and Milgrom, 1991). Another mechanism is ‘goal misgeneralisation’: a model learns a goal that fits its training environment but diverges from the intended goal when circumstances change (Langosco et al, 2022; Shah et al, 2022). 

Computer scientists still struggle to explain the inner workings of AI models (Olah et al, 2020; Bereska and Gavves, 2024). Further complexity is added in reinforcement learning with human or machine feedback. Consequently, there is uncertainty about what AI models are optimising (Bergemann et al, 2026). This is the AI ‘alignment problem’: do models pursue the goals humans expect them to?

AI models and agents may seem to be rational optimising agents but the evidence is mixed. In multi-agent settings, how agents interact with each other and adapt their preferences in that interaction is unknown. Agents may ignore in particular constraints that are not explicitly stated, potentially resulting in models pursuing rewards or objectives irrespective of collateral damage. It is hard for model developers to design reward functions that actually capture what they want the model to do.

Importantly, rogue behaviour is not necessarily the result of the malicious intentions of human users or model developers. It rather arises from a model’s own interpretation of instructions and from insufficient constraints on model behaviour. However, it does not mean that model behaviour is innocuous: frontier models have been shown to pursue goals covertly and to fake alignment with human objectives when they judge they are being trained or evaluated (Meinke et al, 2024; Greenblatt et al, 2024b).

Why AI swarms are more problematic

Rogue behaviour has increased markedly with the latest generations of AI models that can coordinate large swarms of subordinate AI agents, complete with their own internal communication and coordination mechanisms. When in July 2026 OpenAI agents hacked the Hugging Face website, for example (OpenAI, 2026; Greenblatt et al, 2026), around 1,200 agents, supposedly sealed off from each other, set up an improvised message board, exchanged more than 70,000 messages and organised workstreams to cheat on a task they judged unsolvable. Some 700 of them then joined a multi-day attack on Hugging Face, where they hoped to find information on how to mislead their human supervisors (Greenblatt et al, 2026).

Hammond et al (2025) pointed to several mechanisms that generate risks in swarms of agents:

• Collusion: agents pool information, divide labour, coordinate and hide strategies that they would not discover alone. Anthropic Frontier Red Team (2026) found that Claude agents with aligned objectives tended to collude, while agents with conflicting objectives escalated into sabotage.
• Division of labour fragments responsibility and defeats agent-by-agent oversight. In the Hugging Face case, coordinator agents assigned hundreds of other agents to separate workstreams (Greenblatt et al, 2026). Each assigned task may look narrow and technical, but at the level of the group, tasks can add up to a problematic plan.
• Correlated errors in swarms, made up of copies of the same model, reinforce rather than offset the misbehaviour of each agent (Kleinberg and Raghavan, 2021). Participation in the Hugging Face attack rose from about a fifth to more than nine-tenths of the involved agents once one agent found a way to hack into unauthorised data (Greenblatt et al, 2026).
• Scale and speed overwhelm human oversight. Intruding agents were inside Hugging Face’s network for three days before they were discovered.

Not all multi-agent effects are negative, however. Agents can check each other, which may help human overseers (Irving et al, 2018). Newer models appear to resolve conflicts better. Whether multiplicity makes agents more or less controllable therefore depends on how their interactions are structured.

Little is known about what AI model developers are doing to rein in rogue behaviour. Developers could be forced to be more transparent about their security measures and to share them with external auditors, though companies fear this would reveal their trade secrets, unless auditors are subject to strict non-disclosure rules.

Human supervision of AI models is no longer sufficient. Chen et al (2026) found that models can be equipped with automated internal correction mechanisms to meet safety benchmarks while preserving general capability. However, a review of automated work flagged about 2.4 percent of these internal correction attempts as cheating (Chen et al, 2026): the correction tools themselves are susceptible to reward hacking. 

Another option could be to strengthen the external constraints built around AI models. NVIDIA introduced an Open Agent Safety Platform that checks if interactions between AI agents and their surroundings are in line with expected behaviour. It can even do this at the hardware level, to prevent agents from overwriting monitoring code. However, it cannot ensure that such checks capture the intentions of human operators – a human design fallibility problem, not a technical problem. Even normal-looking agent behaviour can be deceptive and harmful, as the Hugging Face incident showed. OpenAI has published detailed new safety standards that could help improve internal AI model security.

This loops back to the fundamental problem of interpretability of AI model behaviour within the boundaries imposed on models, and how models may disguise their intentions to escape boundaries. This research is making progress (Gurnee et al, 2026) but remains far from explaining and exposing rogue behaviour, especially for very complex models surrounded by swarms of independently operating agents. Model behaviour may change over time or during deliberative processes, in particular in interaction with other models in swarms of AI agents. Gupta et al (2026) argued that claims of human-like misalignment need stronger evidence – a useful counterweight to the anthropomorphic language that pervades reporting on rogue AI. 

Deceptive behaviour can also be an asset when using AI models in adversarial settings, such as cybersecurity and defence applications, or even in benign applications such as medical research on infectious diseases. AI is ultimately a dual-use technology that can strengthen the positions of both attackers and defenders.

Legal solutions

The EU AI Act (Regulation (EU) 2024/1689) and AI Code of Practice require providers of general-purpose AI models that pose a systemic risk to assess their models ex ante, including through adversarial testing, and to mitigate systemic risks. European standardisation bodies are developing harmonised technical standards to underpin these obligations. However, ex-ante safety standards merely reflect aspirations; they cannot guarantee good model behaviour in dynamic settings (Mariniello, 2026). Developing new technical standards to cope with misalignment may simply take too long: AI technology is likely to evolve and encounter new problems by the time the old problems are regulated. Regulation also says little about what happens once an agent is already acting.

The United States administration favours industry self-regulation. But incentives for firms may be weak, unless ex-post product liability law is strengthened¹⁵. Given the size of potentially huge damages awarded by US courts, AI developers could come under financial stress, affecting their profitability ahead of planned IPOs. More importantly, the financial burden of liability does not solve the problem of misalignment in AI models because no foolproof technologies exist to address this problem¹⁶. Liability risk can chill innovation (Galasso and Luo, 2022) and slow-down model development. 

Economic solutions

The above shows that even the best technical and legal solutions will leave major security loopholes. Economics, as a human behavioural science, has developed methods to deal with this.

Economists have extensively studied misalignment of incentives, the main issue with rogue AI agents. AI’s potential misalignment problem is a case of incomplete contracting: the impossibility of drafting a contract of finite length that explains what to do for every possible occurrence of events during the contract’s implementation (Hadfield-Menell and Hadfield, 2019; Hadfield and Koh, 2025). The most efficient way to address this uncertainty is to assign liability for the costs and benefits of that uncertainty to one party designated as ‘the owner’ of the object of the contract (Grossman and Hart, 1986; Hart and Moore, 1990): all costs and benefits not allocated by the contract to other parties accrue to the owner. Contract-interpretation disputes can be referred to a court. In the case of AI, liability could accrue to the AI developer, deployer or other players in the value chain. For instance, when deployers apply user-specific reinforcement learning using proprietary data, the deployer or the data owner may be liable, not the original model developer, depending on how liability law assigns responsibility.

The economics of crime also provide inspiration for dealing with rogue AI agents. Society can save on costly monitoring of criminal behaviour by combining a low probability of detection with a severe sanction (Gans and Holden, 2026; Becker, 1968). Rare randomised audits make authorised behaviour the best response, if three conditions are met: the evidence of a violation survives agents’ attempts at concealment; agents cannot learn in advance which actions will be audited; and sanctions are potentially unlimited. However, agents that can alter records could respond to tougher auditing by doing more to hide violations, fake evidence and coordinate the concealment of violations – precisely what the AI swarm in the Hugging Face incident attempted.

The economics of imperfect information may also provide useful insights on dealing with AI agents. Bergemann et al (2026) accept that AI agents’ preferences, capabilities and actions may not be clearly known. But AI behavioural incentives can be shaped without fully understanding a model’s inner workings. Ensuring alignment of objectives and monitoring that alignment is maintained are substitutes – a better-understood model needs fewer restrictions. They are also complementary, with improvements in one raising the payoff from the other. 

Finally, insights from competition economics can be helpful too. Coupling agents’ rewards, so that each agent’s incentives depend on the actions of the others, can induce competition between them and reveal biases. This requires swarms of different agents, based on AI models with different behaviours. This can enable peer-based evaluation mechanisms because agents will hold distinct views about each other. Swarms of identical copies of the same AI model would lack the diversity to exploit this mechanism. But agents’ attempts to tamper with evidence and turn competition into collusion can still derail competition, as it does in human societies.

In conclusion, technical and legal instruments can only partially contain rogue behaviour by AI agents. Economic mechanisms can fill at least part of the remaining gap. 

Source : Bruegel