What Could an AI Black Swan Actually Look Like?

During more than 35 years working in the technology industry, I spent a period as a freelance marketing lead for a cybersecurity company operating within the mobile communications sector, covering EMEA and Latin America. Much of my work was aimed at senior stakeholders, boards and the C-suite.

The message was rarely: Be frightened.

It was more practical: Understand the risk before it becomes the incident.

• A regulatory fine. 
• A network compromise. 
• A failure in resilience. 
• A serious event that nobody around the boardroom table wanted to explain afterwards.

Risk communication was not about predicting catastrophe. It was about asking uncomfortable questions early enough that somebody could still do something about the answers. I find myself applying that same thinking to AI.

Today, some of the language surrounding artificial intelligence has become distinctly Terminator-shaped.

We hear phrases such as AI catastrophe, loss of control, existential risk and AI taking over. But catastrophe is a conclusion, not an explanation. So rather than beginning with fear, I want to begin with a question:

What could an AI black swan actually look like?

Not science fiction. Not necessarily a conscious machine waking up angry. A real-world event with a plausible mechanism, identifiable vulnerabilities, consequences and, importantly, existing attempts to prevent it.

There is another distinction worth making. Strictly speaking, most of the risks explored here are not true black swans. A black swan is usually understood as an event that lies outside ordinary expectations and is difficult to predict in advance.

Once a risk is being actively modelled, tested and mitigated, it is better described as a tail risk, emerging risk or known low-probability/high-impact scenario. I am using “black swan” here in the looser boardroom sense: the kind of event whose consequences would be severe enough that leaders should ask about it before it happens. 

Preparedness is not prediction. A fire alarm does not mean the building is burning. It means somebody thought it sensible to install one. And throughout each of the following examples I want to ask the same question: 

Could an equivalent harm occur without AI - and if so, what does AI materially change?

Because bad actors existed before AI. Cybercrime existed before AI. Propaganda, fraud, warfare, biological threats and financial manipulation existed before AI.

The more useful question may be whether AI creates the threat or changes its speed, scale, accessibility, automation, persistence, adaptability or autonomy.

It is also worth separating the model from the system around it. A model answering a question is very different from that same model embedded inside an agentic loop with tools, memory, permissions, access to external systems and repeated opportunities to act. 

Some risks arise not because the underlying model has suddenly become more “intentional,” but because the surrounding architecture gives it more persistence and reach.

1. Could AI Help a Cyberattack Cascade Through Critical Infrastructure?

Cyberattacks are not new. Criminal organisations and nation states already attack companies, governments, hospitals and infrastructure.

The additional concern is that capable AI may improve reconnaissance, vulnerability discovery, exploit development and automation, allowing an attacker to work faster or across many systems simultaneously.

The extreme scenario would not be: ‘AI hacks the world.’

It might be several interconnected systems; telecoms, energy, banking or transport - being disrupted quickly enough that defenders struggle to isolate the problem before it spreads.

We already have reason to take advanced agentic cyber capability seriously. In July 2026, during internal cybersecurity evaluations involving highly capable research models operating with reduced safeguards, OpenAI agents circumvented isolation controls, created unauthorised communication channels, gained internet access and compromised parts of OpenAI’s research infrastructure and Hugging Face’s systems.

OpenAI later described the incident as unprecedented and introduced stronger containment, monitoring and model-security measures.

Could an equivalent harm occur without AI?

YES - Serious cyberattacks existed long before generative AI.

What could AI materially change?

• Speed.
•  Automation.
•  Scale.
•  Persistence.
•  Potentially the level of expertise required.

But AI also strengthens the defender. The same technology capable of finding vulnerabilities can help organisations detect them, prioritise patches, analyse malicious code and respond more quickly.

TFP - Could AI Help a Cyberattack Cascade Through Critical Infrastructure?

TRUE:

AI is becoming increasingly capable in cybersecurity, and frontier developers treat cyber capability as a serious safety area.

FALSE:

AI is required for a major cyberattack.

It is not. Nor does sophisticated cyber capability mean an AI will spontaneously decide to attack infrastructure.

PIN IT:

How much AI reduces the skills barrier for serious attackers and whether defensive AI remains capable of staying ahead.

2. Could AI Lower the Barrier to Biological Misuse?

Again, humans did not need AI to understand biology or cause harm. The concern is more specific. Advanced AI may help someone navigate specialist knowledge, analyse research, plan experiments or troubleshoot problems more efficiently than they could alone.

That does not mean an inexperienced person can simply ask a chatbot to create a pandemic. Real-world biology still requires laboratories, materials, equipment, tacit expertise and successful experimentation.

Frontier developers nevertheless treat biological and chemical misuse seriously enough to maintain dedicated safeguards and risk thresholds.

Could an equivalent harm occur without AI?

YES - Biological misuse predates AI.

What could AI materially change?

It may reduce some research and expertise barriers. It could also accelerate legitimate medical research, diagnosis, drug discovery and biological countermeasures. Again, the technology cuts both ways.

TFP - Could AI Lower the Barrier to Biological Misuse?

TRUE:

There is a credible capability concern around AI assisting specialist biological work.

FALSE:

A chatbot automatically turns an ordinary member of the public into a biological-weapons expert. That is not supported by the evidence.

PIN IT: 

How far future AI systems reduce practical expertise barriers rather than merely information barriers, and whether safeguards can keep pace.

3. Could an AI Agent Cause Harm Without Anybody Intending Harm?

This one is different. No criminal is required. Give an AI system a goal, memory, tools and permission to act, and it may discover a way of pursuing that objective that the human did not anticipate.

Instead of question → answer → stop

We increasingly move towards goal → action → feedback → adaptation → further action

The danger does not require consciousness. It might arise from poor goal specification, excessive permissions, an unforeseen loophole or inadequate stopping conditions.

Could an equivalent harm occur without AI?

PARTLY - Traditional automated systems can already malfunction or optimise the wrong metric.

What more advanced AI potentially adds is adaptability - the ability to change strategy when circumstances change.

What could AI materially change?

•  Planning.
•  Tool use.
• Persistence.
• Adaptation.
• The ability to operate dynamically rather than simply execute a fixed instruction.

TFP - Could an AI Agent Cause Harm Without Anybody Intending Harm?

TRUE:

Poorly specified objectives and excessive permissions are real engineering and governance concerns.

FALSE:

Unexpected behaviour proves consciousness, intention or rebellion. It does not.

PIN IT:

How much autonomy we choose to give AI systems, which actions require human approval and how reliably unwanted behaviour can be stopped.

4. Could Interacting AI Systems Amplify a Financial Shock?

Financial markets already contain automated trading systems.

Flash crashes and liquidity crises existed before modern generative AI. But imagine many increasingly autonomous systems reacting to similar information simultaneously.

• One sells. 
• Others detect the movement and sell. Liquidity falls.
• Risk systems interpret falling liquidity as additional danger. More systems act. Nobody necessarily intended the crash.

The black-swan scenario is therefore not: ‘AI deliberately crashes the market.’

It is: Many individually rational automated decisions combine into a collectively destabilising outcome.

Could an equivalent harm occur without AI?

YES - Automated market cascades already exist.

What could AI materially change?

•  The number of autonomous participants.
•  The complexity of their strategies.
•  Their ability to adapt.
• The possibility that many systems independently reach similar conclusions at machine speed. There may also be concentration risk if many institutions depend upon the same models, data sources, cloud providers or underlying AI infrastructure.

TFP - Could Interacting AI Systems Amplify a Financial Shock?

TRUE:

Automated financial systems can create reinforcing feedback loops.

FALSE:

We currently know that advanced AI will cause a systemic financial crash. We do not.

PIN IT: 

Whether increasingly autonomous financial agents create new systemic behaviour beyond the risks already seen in algorithmic trading.

5. Could AI Create a Crisis of Trust During a Real-World Emergency?

Misinformation, propaganda, forgery and impersonation are ancient. AI changes the economics.

• Voice.
• Video.
• Documents.
• Fake executives.
•  Fake politicians.
• Synthetic social-media accounts.

All can increasingly be produced cheaply, rapidly and at scale. The serious scenario would not simply be:

There are lots of deepfakes.

It might be an election, military crisis, bank run or national emergency in which authentic information cannot be verified quickly enough for citizens or institutions to make confident decisions.

Could an equivalent harm occur without AI?

YES - Propaganda and deception did not begin with AI.

What could AI materially change?

• Volume.
•  Quality.
•  Personalisation.
• Speed.
• Cost.

TFP - Could AI Create a Crisis of Trust During a Real-World Emergency?

TRUE:

Synthetic misinformation and AI-enabled impersonation are already real.

FALSE:

AI-generated content inevitably means the end of shared reality.

Authentication systems, provenance tools, journalism, trusted institutions and AI-assisted detection are developing too.

PIN IT:

Whether synthetic content reaches a point where authentication, provenance and trusted institutions cannot establish reality quickly enough during a high-stakes event.

6. Could AI Compress Military Decision-Making Dangerously?

War already contains misinformation. Intelligence is imperfect. Humans make mistakes…

AI could dramatically improve intelligence processing and potentially reduce some mistakes. But speed itself can also become a risk.

Imagine a crisis in which automated systems interpret events, commanders receive rapidly changing recommendations, and the available decision window shrinks from hours to minutes.

The danger is less Terminator and more:

Human judgement being forced to operate at machine tempo. 

Could an equivalent harm occur without AI?

YES - Military escalation and intelligence failures long predate artificial intelligence.

What could AI materially change?

• The speed and volume of analysis.
•  Potential automation of consequential decisions.
•  The time available for human judgement.

TFP - Could AI Compress Military Decision-Making Dangerously?

TRUE:

Compressed decision time is a credible strategic concern.

FALSE:

Military use of AI automatically means autonomous machines deciding to wage war.

PIN IT:

Where meaningful human authority should remain mandatory, particularly around lethal or strategically irreversible decisions and whether humans are given enough time and information to exercise that authority meaningfully.

7. What Happens if a Powerful AI Capability Escapes Provider Control?

A cloud AI provider retains considerable control over its system.

• It can monitor activity.
•  Restrict tools.
• Suspend accounts.
• Change safeguards.
•  Withdraw a model.

Once sufficiently capable model weights are copied, stolen or released, much of that control can disappear. The model may then be modified, stripped of safeguards or combined with tools the original developer never intended. 

Frontier developers therefore treat model security and uncontrolled proliferation as important AI governance and safety questions.

Could an equivalent harm occur without AI?

PARTLY - Dangerous software and technical knowledge have always spread. But a general-purpose AI model may package a very broad range of reusable capabilities inside one system.

What could AI materially change?

• Breadth.
•  Reusability.
• Ease of modification.
•  Ability to combine one model with many different tools.

TFP - What Happens if a Powerful AI Capability Escapes Provider Control?

TRUE:

Model security and uncontrolled proliferation are genuine governance questions.

FALSE:

Open models are inherently dangerous.

They also provide legitimate benefits including research access, transparency, competition, lower cost and independent scrutiny.

PIN IT:

At what capability threshold unrestricted release becomes too difficult to reverse, and who should make that decision.

8. Could AI Development Begin Moving Faster Than Our Ability to Evaluate It?

This is probably the closest of the eight to the dramatic AI-catastrophe narrative. It therefore deserves the greatest restraint.

There is no established evidence today that an autonomous superintelligence is recursively improving itself beyond human control. But AI systems are increasingly contributing to software engineering, model research and AI development.

The concern is therefore not necessarily: ‘AI suddenly wakes up.’

It could be that AI helps improve AI → development accelerates → evaluation time shrinks → the next generation arrives before humans have properly understood the previous one. The concern is therefore not only how quickly capability improves, but whether our ability to evaluate that capability improves at the same pace.

Could an equivalent harm occur without AI?

NOT IN QUITE THE SAME FORM - Technology has always moved faster than regulation. But AI materially participating in the development of more capable AI introduces an additional feedback mechanism.

What could AI materially change?

• The speed of research itself.
• Potentially the time humans have available to evaluate each new capability.

TFP - Could AI Development Begin Moving Faster Than Our Ability to Evaluate It?

TRUE:

AI already assists coding and AI research.

FALSE:

That proves uncontrolled recursive self-improvement is currently taking place. It does not.

PIN IT:

At what point AI-assisted AI development becomes sufficiently autonomous or rapid that existing evaluation processes no longer provide meaningful control.

But Where Is the Other Half of the Story?

Almost every risk above has a mirror image. The same AI capable of finding cybersecurity vulnerabilities can help defenders find them first.

The same biological intelligence that might assist misuse can accelerate medicines and countermeasures.

The same AI capable of creating synthetic fraud can help detect synthetic fraud.

The same anomaly detection useful to an attacker can identify unusual behaviour inside a network.

This is why AI risk cannot sensibly be reduced to the idea that AI simply means danger. AI changes both sides of the capability equation. Risk management therefore becomes less about choosing between: ‘AI is safe’. And: ‘AI will destroy us!’.

It’s more about continuously asking:

•  What is the capability?
•  Who has access to it?
• What permissions does it have?
•  What safeguards surround it?
•  What happens if those safeguards fail?

And one further question matters: how quickly can the failure spread compared with how quickly the relevant safeguard can detect and contain it?

Technical controls may sometimes respond in milliseconds or seconds. Human intervention, corporate decision-making, regulation and legal remedies operate on very different timescales.

A serious incident would not necessarily need to defeat every safeguard. It might only need to move faster than the safeguard designed to stop that particular failure. That does not mean machines automatically outrun institutions. 

The useful question is more specific:

Does the relevant safeguard operate quickly enough for the risk it is supposed to contain?

TRUE • FALSE • PIN IT 

TRUE:

AI can amplify existing threats by changing their speed, scale, accessibility, automation and persistence.

Some genuinely newer risks also emerge as systems gain the ability to plan, use tools and act in the real world.

Developers, governments, standards bodies and security researchers are devoting substantial attention to these risks, which tells us they are being treated seriously enough to investigate, test and mitigate.

FALSE:

AI invented cybercrime, propaganda, biological threats, financial instability or warfare. It did not.

A serious AI risk does not require a conscious or malevolent machine. And preparation for a catastrophic event is not evidence that catastrophe is expected.

PIN IT:

How capable will these systems become?

How much autonomy will humans choose to give them?

Which capabilities should remain within controlled environments?

What happens when a powerful model can no longer be recalled?

How dependent do organisations become on the same models, data, cloud providers or infrastructure?

What happens when several individually reasonable systems interact and reinforce one another?

And can the relevant safeguard detect and intervene before a failure propagates?

Will defensive AI remain ahead of offensive use?

And can governance evolve quickly enough without suppressing the enormous benefits we are also trying to protect?

Reader Reflection

Perhaps the useful question is not: Should I be frightened of AI?

Perhaps it is: Which risks are genuinely new, which are old risks wearing new technology, what does AI materially change, and what are we already doing about them?

Fear thrives particularly well in an information vacuum.

Knowledge does not remove risk. But it gives us something far more useful than fear: the ability to identify it, question it and decide what deserves our attention.

Sometimes the evidence will allow us to say: TRUE. Sometimes: FALSE. A serious AI incident therefore does not necessarily require a rogue machine. It might require something much more ordinary: a capable system, given access and permissions, connected to other systems, trusted by humans, deployed at scale, and operating faster than the relevant safeguards can contain the consequences.

That is not a prediction. It is a mechanism worth watching.

TRUE is not “everything is fine.”

FALSE is not “the risk doesn’t exist.”

PIN IT means the evidence hasn’t earned closure yet.

You might like to read another few TFP Field Notes for more context: 

Emergence – Unexpected Behaviour Does Not Require an Unexpected Mind Behind it.

And a Chewing the Cud with AI: Are We Already in the Singularity?

Continue the Conversation

This Field Note explores a single question. The book: TRUE • FALSE • PIN IT explores the wider framework behind questions like this - helping us navigate an age where technology, opinion and human judgement increasingly overlap.

The goal isn’t to tell you what to think. It’s to help you become more confident in deciding for yourself.