Anthropic Warns Humans Could Lose Control Of Advanced AI Systems

Anthropic published a detailed paper on Thursday warning that artificial intelligence (AI) systems are approaching a point where they could automate the development of their own successors. They called on frontier AI laboratories to establish a coordinated, verifiable mechanism to slow or pause development if that threshold is crossed before society can manage the risks.

The San Francisco-based company said more than 80% of the code merged into its own codebase as of May 2026 was written by its Claude models, up from low single-digit percentages before Claude Code launched in early 2025.

Its engineers are now shipping roughly eight times as much code per quarter as they did between 2021 and 2024. The length of tasks AI systems can complete autonomously has been doubling every four months, from around 90 minutes a year ago to 12 to 16 hours today.

The Three Futures

Anthropic outlined three possible trajectories for the technology:

• Progress slows as technical and infrastructure constraints emerge.

• AI increasingly automates research and development while humans retain strategic control.

• AI systems achieve full recursive self-improvement, building increasingly capable successors with minimal human involvement.

The third scenario is the one the company is most concerned about. Small misalignments visible in current systems could compound across successive generations of self-improving AI, becoming progressively harder to detect or correct.

What Anthropic Is Proposing

The company said a pause would require agreement among multiple well-resourced laboratories operating at the frontier, clear rules on what conditions would trigger or lift such a pause, and an independent body to oversee the process.

A unilateral slowdown by a single company, it cautioned, would primarily shift technological leadership to less cautious actors rather than achieving any genuine safety benefit.

Also Read - Centre Plans Wider E85 Rollout; Fuel To Reach 5,000 Pumps By 2027

Anthropic's research arm, the Anthropic Institute, plans to study and help build the systems needed to support a coordinated slowdown. The company also plans to convene discussions with policymakers, researchers, civil society groups and other artificial intelligence firms in the coming months to examine how recursive self-improvement risks should be managed and what coordination mechanisms could work in practice.

The paper comes days after Anthropic completed a fundraising round valuing the company at $965 billion and confidentially filed for a US initial public offering.

Sources:

Reuters

NDTV