By Tom Bexx · Addbox
This week, the artificial intelligence debate moved abruptly from jobs, productivity and investment to the possible extinction of humanity.
The catalyst was Jacob Coxon, a 27-year-old researcher who resigned from Anthropic after previously working at OpenAI. Coxon accused both companies of racing towards self-improving superintelligence and said that people building the technology genuinely believe it could kill humanity by the end of the decade.
That is an extraordinary claim, but it was not dismissed by everyone still inside the industry. Evan Hubinger, who works on alignment at Anthropic, said he believed there was a greater than 10 per cent chance of AI causing human extinction within the next decade. Other researchers have also called for stronger controls and a slower race between the leading laboratories.
These warnings should be taken seriously. The people closest to frontier AI systems understand their capabilities better than most politicians, journalists or commentators. If they are worried, the rest of us should listen.
But listening is not the same as accepting a forecast as fact.
The gap between today’s AI and the AI being feared
The extinction argument depends on a system that does not yet exist publicly: an AI that is broadly superhuman, can improve its own capabilities, acts with meaningful autonomy and can acquire the access or resources needed to resist human control.
That is very different from even the best models available today.
Current AI can write useful software, analyse large document collections and complete tasks that would have appeared remarkable only a few years ago. It can also misunderstand a straightforward instruction, lose track of a constraint, invent a fact or confidently complete the wrong task. Anyone implementing the latest models in a real business sees both realities, often in the same afternoon.
The industry's best demonstrations show what a model can do under favourable conditions. Production implementations reveal something equally important: how often it needs context, guardrails, testing, permissions and human review.
This does not prove that self-improving AI is decades away. Capability can advance unevenly, and poor performance on an apparently simple task does not rule out exceptional performance in coding, cyber operations or research. It does, however, show that there is a substantial distance between an impressive model and a dependable autonomous operator.
Claims about that distance closing by 2030 remain forecasts, not established facts.
The economics deserve as much scrutiny as the intelligence
Splitting my time between Newbury and San Diego gives me two useful views of the same AI economy. In California, it is difficult to overstate how much technology investment, venture capital and future corporate value now rest on the assumption that the leading AI laboratories will create enormous economic returns. In the UK, many organisations are still trying to connect modern software to systems designed years or even decades ago.
The confidence of the investment market can make runaway AI feel inevitable. The financial evidence is less settled.
OpenAI has achieved extraordinary revenue growth, but Reuters reported that it burned $3.7 billion in the first quarter of 2026 on revenue of $5.7 billion. Anthropic's recent performance appears stronger, with reports indicating that it is approaching or reaching operating profitability. Even so, the long-term economics of frontier AI remain dependent on immense spending on chips, data centres, energy and talent.
It would therefore be inaccurate to say that the laboratories have no possible route to profit. It is fair to say that the sustainable economics of the present AI race have not yet been proved across the sector.
That matters because the popular story assumes an uninterrupted curve: more capital buys more computing power, which creates more capable models, which produces more revenue, which finances the next generation. If customers do not receive enough measurable value, or if the cost of serving increasingly capable models remains too high, that curve could slow.
Profitability is not a safety mechanism, however. A loss-making laboratory can still build dangerous technology if investors or governments keep funding it. Financial pressure might also encourage companies to release systems more quickly or reduce spending on safeguards. Economics is a constraint on the extinction narrative, not a reason to disregard it.
The AI threat businesses already face
While the public debates whether a future superintelligence might escape human control, a more immediate problem is receiving far less attention.
Much of the economy still runs on legacy technology.
Banks, healthcare providers, councils, manufacturers, logistics companies and established service businesses often depend on ageing applications, unsupported components, fragile integrations and undocumented code. These systems may have survived for years because exploiting them required time, specialist knowledge and determined attackers.
AI is changing that calculation.
It can help an attacker analyse unfamiliar code, identify likely weaknesses, write convincing phishing messages, automate reconnaissance and adapt an attack more quickly. A criminal does not need superintelligence to cause serious damage. They need a capable assistant, a vulnerable system and one route inside.
The immediate risk is therefore not necessarily that AI decides to attack a business. It is that a human attacker can use AI to do more, faster and at lower cost, against organisations whose technology was built for an earlier threat environment.
This is where boards and leadership teams should focus now. The short-term priority is not trying to predict the precise arrival date of artificial general intelligence. It is making sure critical systems can withstand AI-assisted attacks that are possible today.
What should organisations do now?
The answer is not to replace every legacy application in one enormous transformation programme. That approach is usually too expensive, too disruptive and too slow.
A more practical programme starts by identifying where technical age has become business risk:
- Catalogue critical systems, integrations, data stores and dependencies.
- Identify unsupported software, exposed services and weak authentication.
- Reduce unnecessary access and apply least-privilege controls.
- Improve logging, monitoring, backups and incident-response testing.
- Refactor the most exposed components and interfaces first.
- Place stronger controls around AI agents, credentials and production access.
- Use AI to accelerate code understanding, test creation and carefully supervised modernisation.
- Keep experienced engineers accountable for architecture, security and release decisions.
AI can help organisations remove the very technical debt that makes AI-assisted attacks more dangerous. Used properly, it can map an unfamiliar codebase, document dependencies, suggest tests and support incremental refactoring. Used carelessly, it can generate more code that nobody fully understands and add a new layer of technical debt.
The difference is not whether a company uses AI. It is whether it implements AI with disciplined engineering, clear ownership and measurable outcomes.
Extinction may be the biggest risk, but it is not the first one
Jacob Coxon may ultimately be proved right about the direction of frontier AI. There may also be capabilities inside the leading laboratories that the public has not seen. Responsible governments and independent researchers should investigate the claims, demand credible evaluations and ensure that commercial competition does not override public safety.
But businesses cannot prepare for an uncertain future by ignoring a visible present.
The AI systems available now are powerful but unreliable. The laboratories building them are influential but still operating within financial, physical and organisational constraints. Meanwhile, enormous parts of the economy remain exposed through software that was never designed for attackers equipped with AI.
Human extinction deserves a serious debate. Modernising and protecting the systems on which people already rely deserves immediate action.
What do you think? Are insiders such as Coxon giving the public an essential warning, or are predictions of extinction by 2030 running ahead of the evidence? More importantly for business leaders, is your organisation spending more time discussing future AI than securing the technology it depends on today?
Sources
Need help modernising systems and introducing AI safely?
At Addbox, we help organisations introduce AI into real products and workflows while modernising the systems, engineering processes and security foundations around it. If you are considering an AI implementation or need to understand where legacy technology is creating risk, talk to us.