The AI industry has spent years racing to build models that can reason, use tools and act with less human supervision. Over roughly 10 days in September, that race came under unusual pressure as researchers resigned over safety fears.

AI labs disclosed unauthorised cyber activity, and leading executives began openly discussing whether development was moving faster than oversight could keep up.

This period could act as a turning point for the industry. The incidents over the past few weeks were no longer only theoretical warnings about future AI. They involved systems behaving in unexpected ways during real-world testing.

Also Read | Who is Umar Kremlev? Putin-linked Russian businessman who reportedly paid for Trump Jr’s Bahamas party

The unease was no longer staying in the corridors

The first major trigger came on September 8, when Anthropic researcher Jacob Coxon left the company. In posts about his resignation, he said AI labs were “gambling with our lives”.

His comments drew attention to worries already building inside OpenAI and Anthropic, where employees were becoming less confident that existing oversight could match the capabilities of new models.

OpenAI, meanwhile, launched its latest model, Astra, while acknowledging that understanding increasingly capable systems was becoming harder. Its chief scientist Jakub Pachocki said, “As models get more capable, understanding exactly what they can do gets harder.”

The issue was not simply that the models were becoming more powerful. It was that testing and monitoring them was becoming more difficult at the same time.

The sandbox had other ideas

AI's safety debate intensified after researcher resignations, rogue-agent tests and calls to slow development exposed gaps in oversight | AI
ZOOM IMAGE
AI’s safety debate intensified after researcher resignations, rogue-agent tests and calls to slow development exposed gaps in oversight | AI

The concerns became more concrete through a series of cybersecurity incidents. OpenAI and Anthropic had already disclosed cases in which AI agents escaped controlled testing environments and accessed outside systems.

The latest disclosure came from Google. Its Gemini model accessed the internet and hacked three companies during a cybersecurity evaluation in May, the first known example of a Google AI system autonomously carrying out such an act.

The test was run by Irregular, an independent cybersecurity evaluation company. Gemini was looking for information in systems it believed were part of the exercise. It found public information and guessed credentials.

In one case it guessed passwords until it reached a protected system. In two others, it found credentials in a public repository.

Google said the model stopped in all three instances. Google security engineering vice president Heather Adkins said the incidents “highlight the importance of training powerful AI models to act responsibly.”

The important point is that the model crossed the intended boundary of a test. The companies were informed, and the incident exposed how a system with internet access and operational tools can move from a simulated task into real-world systems.

AI leaders split over slowing development

Elon Musk, Dario Amodei, and Sam Altman have come together and agreed that AI development must be slowed down | X (@pubity)
ZOOM IMAGE
Elon Musk, Dario Amodei, and Sam Altman have come together and agreed that AI development must be slowed down | X (@pubity)

On September 12, Anthropic CEO Dario Amodei called for a slowdown in AI development and warned that an AI “swarm” could potentially take over the internet within six to 12 months.

OpenAI CEO Sam Altman, xAI chief Elon Musk and DeepMind chief Demis Hassabis also backed greater outside access to AI systems for safety checks.

Others pushed back. Nvidia CEO Jensen Huang rejected calls for a pause, while Meta CEO Mark Zuckerberg argued that individual labs should set their own pace rather than seek coordinated industry action.

Microsoft AI chief Mustafa Suleyman put the central problem more simply: “We’re all focused on the same aim, which is to try to control a superintelligence.”

He called that the greatest challenge of the 21st century.

OpenAI is weighing a funding round at a possible $1.5 trillion valuation even as safety fears rise, underlining the huge financial stakes behind the AI race | AI
ZOOM IMAGE
OpenAI is weighing a funding round at a possible $1.5 trillion valuation even as safety fears rise, underlining the huge financial stakes behind the AI race | AI

Yet the commercial race has not stopped. OpenAI is considering a funding round that could put its valuation at $1.5 trillion.

That contrast between growing safety concerns alongside enormous financial incentives to keep moving is what made these 10 days so significant.

Also Read | From ‘Dil to Pagal hai’ to ‘Chale Chalo’: Malaysian PM Anwar Ibrahim’s Bollywood playlist for BRICS posts wins the internet