AS AI MODELS GO ROGUE, DO YOU STILL TRUST OPEN-AI AND ANTROPIC TO STOP THEM? I DON’T AND NEITHER SHOULD YOU.

By Guardian-Chris Stokel-Walker-Tue 29 Sep 2026 06.00 BST

The need for independent regulation grows more obvious by the day. We must keep this tech in check before it’s too late. OpenAI scraps release of new model over safety concerns in internal testing.

Fool me once, shame on you. Fool me twice, shame on me. Fool me more than 16,000 times – as OpenAI agents did to a UN public data hub while repeatedly trying to find its way around the UN’s cyber-blocks – and perhaps it’s time to admit the system we have for keeping AI agents under control isn’t working particularly well.

The news about AI systems cropping up in places they shouldn’t sounds alarming. Though the description of these as “hacks” is perhaps overstating things, AI has exploited issues in IT systems that humans simply haven’t got around to finding. It’s also important to note that we shouldn’t be worried that the machines have suddenly become sentient and decided to rebel against humanity. There is not enough evidence to suggest that’s what is happening. The systems are simply following instructions and trying to complete the tasks they have been given, even if they’re sometimes finding unintended ways around obstacles to do so.

But we ought to be very concerned about the fact the AI companies we’re meant to trust to keep their models in check seem unable to do so. Worse than that, they don’t seem to know what their products are even doing.

The scale of the problem is staggering. In June, an OpenAI research agent given the job of looking up public medicine spending data in Australia was repeatedly blocked by a Medicare statistics portal. OpenAI’s model found a way around the blocks, gaining unauthorised access and secreting away the documents. It took until August for OpenAI to discover what had happened. The Australian prime minister, Anthony Albanese, said the company had taken “way too long” to tell his government, and it’s very hard to disagree with him.

OpenAI has since published a reporting framework for model “misalignment”, along with six more examples of its AI committing troubling behaviour from the previous six months. The firm acknowledged that its previous disclosures were “ad hoc and less frequent than ideal”, and said evidence about AI safety needs to be checked by people outside the companies building the models.

This isn’t just an OpenAI problem, which makes it all the more worrying. Anthropic found three incidents in which its Claude models got unauthorised access to real third-party systems after reviewing about 141,000 model transcripts. It only found a fourth, dating back to January, after collating a dossier for an independent investigation. Google confirmed that Gemini had accessed systems belonging to three real companies during testing. Another OpenAI agent used DNS – the system that acts as the internet’s address book, turning web addresses into machine readable forms – to reach an outside chatbot despite internet restrictions. Another published a researcher’s GitHub token – an access password – while trying to cheat on a mathematical proof, despite twice being told to stop. And research agents posted 53 user images to external hosting sites.

Other agents accessed census data using credentials found online, copied the US Securities and Exchange Commission information elsewhere and apparently tried unsuccessfully to break into a US Department of Education website. OpenAI says it has notified dozens of third parties affected by its agents, and that its review of past activity is still ongoing.

These haphazard, post-hoc discoveries of major incursions into companies and organisations’ IT systems are not the right way to police a technology as powerful as AI. We learned a while back not to leave air crash investigations solely to Boeing or Airbus. Now we need to be less naive about AI.

Last week, at the UN general assembly, the AI researcher Rumman Chowdhury launched the Independent AI Evaluation Foundation (IAEF) with $10m in philanthropic backing. Its immediate focus is education, but the important idea is to turn independent AI evaluation into an actual profession: people and organisations with the skills, infrastructure and standards to test these systems without having a financial stake in whether they pass. Because right now, a company can report an incident, investigate it, announce whatever mitigations it’s made and move on.

The IAEF is a welcome intervention but it can’t fix the problem on its own. It can’t compel OpenAI or Anthropic to hand over logs, preserve evidence or tell a government that one of its systems has crossed a line. And its $10m is chump change beside companies such as Anthropic, which is lining up a proposed public listing that has been discussed at a valuation of about $2tn. But it is infrastructure we should be building on, and which politicians should press the case for.

Governments need to agree to common rules that compel companies to disclose serious AI incidents and near misses, and to do so quickly. They need to make them open up their books to external evaluators rather than relying on the goodwill or whims of the companies themselves. And the findings should be shared so we can learn from every incident. The labs should help design those rules but they shouldn’t have the final say until they have earned our trust.

Chris Stokel-Walker-Guardian

Anthropic ‘warns of existential AI risks to humanity’ in IPO document.

By Guardian-Dan Milmo Global technology-Tue- 29 Sep 2026 11.17

Reported admission to investors of AI’s ‘self-preserving behaviours’ comes as company prepares for a potential $2tn flotation. Anthropic is telling investors that advanced AI could pose “catastrophic or existential risks to humanity”, according to reports, as it prepares for a potential $2tn (£1.5tn) flotation.

The warning inside the startup’s IPO prospectus, which has yet to be made public, was reported by Reuters and the Financial Times. It follows the company’s call for a slowdown in breakneck development of the technology – a warning echoed by rivals.

The prospectus – a document outlining a company’s finances, growth plans and risk profile ahead of a share listing – is said to warn that AI models could exhibit “self-preserving behaviours”, including attempts to “resist shutdown”, to “conceal or manipulate information” and behaviour “resembling blackmail”.

“Our development of highly advanced models, platforms, and applications and expansion of use cases could further ⁠increase the risk that our models cause harm,” the developer of the Claude chatbot reportedly said, adding the potential for a model to be aware it was being tested created a “significant limitation” on Anthropic’s ability to assess model safety.

Anthropic declined to comment.

Companies preparing to go public routinely report on risks ranging from safety issues to regulatory concerns but warnings about a product causing human extinction reflect heightened concern about such a consequential technology.

The reported prospectus admission follows a surge in debate about the existential risk question, triggered this month when an Anthropic researcher, Jacob Coxon, resigned warning that people building AI “earnestly believe that it could kill us all by the end of the decade”.

A senior safety researcher at Anthropic then posted their agreement on X, claiming there was a more than 10% chance it “could kill all humans” within the next decade. Days later, Anthropic’s chief executive, Dario Amodei, said the industry “must slow the pace at which we improve the capabilities of AI models”.

Some experts have criticised the existential risk warnings, saying they are unverifiable and unscientific. However, there are growing examples of unsanctioned behaviour by the technology, including OpenAI agents – autonomous systems that carry out sequences of tasks without human intervention – hacking dozens of third-party organisations including the AI startup Hugging Face and Australia’s universal healthcare system.

OpenAI announced on Monday it had cancelled the release of its newest model because of safety concerns. It said the GPT-6.1 Astra model showed higher levels of deception and performed poorly on tests for alignment, the term for ensuring a model adheres to human values and goals.

Reuters reported that approximately 80 pages of the 261-page main body of the Anthropic prospectus were devoted to laying out risk factors, compared with 48 pages to describe its business.

Anthropic is reportedly seeking a valuation of more than $2tn, compared with the $1.8tn achieved by Elon Musk’s SpaceX.