Warnings that advanced artificial intelligence could pose an existential threat to humanity are becoming increasingly prominent as leading AI companies push for stronger safeguards and tighter controls on frontier development. At the same time, questions are emerging over the evidence supporting predictions of runaway AI and the enormous financial resources required to develop increasingly powerful systems.
Industry estimates and analysis cited by Byline Times suggest OpenAI, Anthropic and Google parent Alphabet could collectively face research-computing costs of about $10.7 trillion between 2026 and 2030. Even under optimistic growth assumptions, revenues could cover only part of that expenditure, potentially leaving a funding gap of roughly $7 trillion to be addressed through reserves, financing, increased income or lower costs.
AI Companies Raise Concerns Over Existential Risks
Debate intensified following the resignation of AI safety researcher Jacob Coxon from Anthropic, the company behind Claude. Coxon, who previously worked at OpenAI, warned about efforts to develop AI systems capable of improving themselves and potentially becoming substantially more capable than humans.
Other Anthropic researchers have also publicly expressed concerns. Evan Hubinger estimated the probability of human extinction during the next decade at more than 10%, while colleague Samuel Marks warned about the possibility of extinction “within a few years”.
Anthropic chief executive Dario Amodei has separately argued that rapidly advancing AI could become capable of “taking over the entire internet” within six to 12 months.
Amodei has called for US Government support or antitrust exemptions that would allow leading AI developers to coordinate on safety standards and development limits. He has also advocated independent monitoring by the nonprofit Model Evaluation & Threat Research (METR), alongside restrictions on Chinese access to advanced semiconductors and chipmaking equipment.
Calls for greater caution over frontier AI development have also received support from prominent industry executives, including OpenAI chief executive Sam Altman, Google DeepMind chief executive Demis Hassabis and xAI chief executive Elon Musk.
Evidence for Runaway AI Remains Limited
At the centre of the debate is the concept of “recursive self-improvement”, or RSI. AI systems are already being used to assist researchers in developing better AI models. A more extreme scenario would involve increasingly capable systems improving their successors with progressively less human involvement, potentially creating a rapidly accelerating cycle.
However, existing experiments have yet to demonstrate such a self-sustaining intelligence explosion.
Daanish Masood, chief executive of AI company CX-1 and former co-lead of the United Nations Innovation Cell, said the distinction between demonstrated capabilities and predictions was important.
“The AI risks Dario details are serious enough to justify enforceable safeguards,” he told Byline Times. “I think we need to distinguish what’s been demonstrated from what is being predicted.”
He added:
“AI is already helping to develop AI, but that alone does not establish the runaway, self-sustaining improvement process that is implied by the rhetoric or the term RSI. We also don’t need to declare or pretend that a takeoff has arrived to take the risks seriously.”
Experiments Test AI Self-Improvement
AI research company Weco tested part of the recursive-improvement process in July, with a system producing seven improved versions of an AI research programme over eight days without human intervention. Some of the resulting improvements also transferred to tasks beyond those originally used for development.
The experiment nevertheless stopped short of demonstrating runaway improvement. When one of the enhanced programmes was placed in charge of another development cycle, its eventual performance was no better than that achieved when the original system managed the process.
The improved programme appeared to reach the result more quickly, but researchers said the evidence was insufficient to establish a reliable speed advantage. Weco itself said the system was not close to the runaway acceleration commonly described as an “intelligence explosion”.
METR conducted another experiment in July in which AI agents were given up to five days and $10,000 to improve software used for training AI systems. The work forms part of wider efforts to establish how effectively current AI can automate the research and engineering required to build more capable models.
Huge Computing Costs Challenge the AI Industry
Alongside the safety debate sits an increasingly significant economic issue: the cost of developing frontier AI.
Training and operating advanced models requires enormous computing capacity, specialist chips, data centres and electricity. As companies attempt to build increasingly sophisticated systems, infrastructure requirements are expanding rapidly.
The analysis cited by Byline Times estimates that OpenAI, Anthropic and Alphabet could collectively require approximately $10.7 trillion in research-computing expenditure from 2026 to 2030.
If realised, expenditure on that scale would represent a substantial share of global economic output. The analysis suggests that projected business growth alone might cover approximately one-third of the estimated bill, leaving around $7 trillion that would have to come from additional financing, company reserves, higher revenues or significant reductions in computing costs.
For Britain and other economies seeking to develop competitive domestic AI industries, the scale of investment required also raises broader questions about market concentration. Smaller companies and research organisations may struggle to match the computing resources available to the largest technology groups.
Safety Rules Could Reshape Competition
The debate therefore extends beyond whether advanced AI presents serious long-term risks. It also concerns who establishes safety standards and how those rules affect competition.
Two experts with experience developing AI programmes at the United Nations and a major global bank told Byline Times that measures intended to reduce AI risks could have wider commercial consequences. Restrictions on development, computing infrastructure or access to advanced technology could potentially make it more difficult for smaller competitors and developers of cheaper models to challenge established companies.
That does not mean AI safeguards are unnecessary. Researchers across the sector continue to investigate potential risks associated with increasingly autonomous and capable systems. However, the available experimental evidence has not yet demonstrated the runaway, self-sustaining improvement process underpinning some of the most severe extinction scenarios.
As governments, including the UK, consider how to regulate advanced AI, policymakers face the challenge of balancing genuine safety concerns with competition, innovation and the extraordinary economic demands of frontier model development. The debate is likely to intensify as both AI capabilities and the cost of building them continue to rise.
Thomas Hawthorne is a contributor to Nintendo-power.com, covering a broad range of topics including news, business, technology, entertainment, lifestyle, and current affairs. He focuses on delivering clear, accurate, and accessible reporting that helps readers stay informed about important developments and emerging trends. With a reader-first approach, Thomas aims to provide useful context, balanced insights, and engaging stories that reflect the issues, events, and interests shaping everyday life.