Could AI Risk Be Less About Intelligence and More About the Pipe? 

We spend a lot of time asking how intelligent AI might become.

• How capable.
• How autonomous.
• How powerful.
• How quickly it might improve.

But I’ve been thinking about a different question. What if the more important systemic-risk question is not simply:

How smart is the AI?

What if it is: 

What can it reach, what is connected to it, what depends upon it, and what happens if something goes wrong?

That changes the picture. Because an AI capability does not automatically become an AI consequence.

Something must connect the two.

That is the pipe - Find the pipe

A very simple fear narrative might look like: AI capability → catastrophe

But that skips almost everything in between.

A more useful chain might be:

Capability → Access → Permissions → Connection → Dependency → Feedback → Propagation → Safeguard Failure → Consequence

Every arrow matters. Capability does not automatically create access. 

Access does not automatically create permission. Permission does not automatically create connectivity. Connectivity does not automatically create dependency.

Dependency does not automatically create systemic failure. And systemic failure does not automatically become catastrophe.

So perhaps one of the most useful questions we can ask is: Where is the pipe?

How does the capability actually travel from what the AI can do to what happens in the real world?

Cyber Makes this Easy to See

Take a cyberattack. Saying: “AI could hack critical infrastructure” is not really enough.

There is a chain.

An attacker still needs some combination of: access, credentials, network reach, vulnerabilities, permissions, persistence, lateral movement, and a failure of defensive systems.

AI may reduce the time, expertise or coordination required at some stages. That matters. But AI does not simply erase every layer between attacker and target.

• Authentication still exists.
• Segmentation still exists.
• Monitoring still exists.
• Isolation still exists.
• Recovery still exists.

The better question becomes:

Could AI materially reduce the time, cost or expertise required to move through enough of that chain that existing defences can no longer contain the attack?

That is much more useful than: Can AI hack everything?

Biology Shows the Same Pattern

AI might assist someone with scientific knowledge.

It may help interpret research.

It may help with design, planning or troubleshooting.

But biology still happens in the physical world.

There is another pipe:

Knowledge → Design → Materials → Equipment → Laboratory → Experimentation → Validation → Production → Distribution

The further a scenario moves from information into physical execution, the more dependencies appear. And dependencies can become: chokepoints, controls, barriers, or intervention points.

So again: Capability is not consequence.

Agentic AI is different

This is where the risk becomes more interesting. A normal chatbot interaction may look like: 

Question → Answer → Stop

An agentic system may look more like: 

Goal → Action → Feedback → Adaptation → Further Action.

Add: tools, memory, permissions, external systems, persistence. And other agents, and the system becomes much more capable of doing rather than simply advising.

The concern still does not require consciousness. It does not require rebellion. It may simply require: a capable system, with access, given enough permission, connected to enough things, operating for long enough, inside a feedback loop.

That is a different risk architecture from a passive chatbot. But again, it is still an architecture. Not destiny.

Finance Introduces Common-mode Failure

This one fascinated me. Imagine hundreds of financial systems that appear independent. But what if many of them rely on the same: data, models, cloud infrastructure, risk assumptions, market signals, or communications network? Then apparent diversification may conceal concentration.

A simplified chain might look like:

Common data → Similar signals → Similar decisions → Correlated action → Market movement → Risk response → More correlated action

That is a feedback loop. The systems do not have to coordinate deliberately. They only have to depend on similar inputs and react in similar ways.

This gives us another useful question:

 How independent are independent systems really? And perhaps the next question is: how independent are their backups?

A backup is not genuinely independent if it ultimately depends on the same cloud, network, data source, software layer or authentication system.

There is another question too:

 How easily can the system switch?

A system may depend heavily on one component and still remain resilient if it has genuine alternatives, spare capacity, tested failover and functioning manual procedures. Dependency becomes more concerning when switching is slow, difficult or impossible.

Information Risk is Also About Velocity

Deepfakes are not interesting simply because fake content exists.

Humans have always lied.

The deeper issue is speed. 

What happens when: generation + distribution + personalisation move faster than: authentication + verification + institutional response?

That creates a velocity problem. And this matters because a safeguard may exist but still arrive too late.

A serious failure does not necessarily have to defeat every safeguard. It may only have to move faster than the safeguard designed to stop that particular failure.

That feels important. Because we often talk about AI safety as though there is one protective wall. There isn’t.

It may also help to distinguish between different scales of failure.

An AI can produce a bad output. An AI-enabled system can cause an operational problem. And an operational problem can, in some circumstances, propagate into something systemic. Those are three different claims, requiring three different levels of evidence.

There are many different layers: technical controls, monitoring, human intervention, corporate governance, regulation, law, recovery systems.

And they all operate at different speeds. So, the useful question becomes: Does the relevant safeguard operate quickly enough for the risk it is supposed to contain?  In other words, perhaps we should not only find the pipe. We should measure it: how quickly can a failure move through it, how wide is the potential blast radius, and where can it realistically be interrupted?

Military Systems Make this Obvious

In military decision-making, speed can improve analysis. But speed can also compress judgement.

The chain might be: Information → AI analysis → Recommendation → Human decision → Action → Adversary reaction → New information

The concern is not necessarily that AI makes the decision. It may be that the human technically remains “in the loop” while having too little time to meaningfully challenge the recommendation.

That gives us another distinction:

• Speed of information is not the same as speed of understanding.
• Open models change where control lives

A cloud provider can monitor access:

• Restrict tools.
• Suspend accounts.
• Change safeguards.
• Withdraw a model.

But once sufficiently capable model weights are copied, released or stolen, control becomes more diffuse.

The pipe changes.

Model capability → Replication → Redistribution → Modification → New deployment → New tools → New permissions → New consequences

This does not mean open models are inherently dangerous. It means:

Control of the original system is not the same as control of every future deployment of that system.

That is a governance problem. Not necessarily a science-fiction escape story.

What about AI improving AI?

This is where people often jump very quickly to the Singularity. But again, the pipe matters. AI may assist coding, research, model design, testing, analysis.

That could accelerate development. But large-scale AI development still sits inside a very physical system:

Chips → Compute → Memory → Networking → Electricity → Data Centres → Cooling → Software → Engineers → Capital → Supply Chains

AI does not float above infrastructure. It lives inside it. That does not prove acceleration will be slow.

It does not prove governance will keep up. It simply means the rate, scale and timing of future development still depend on real systems. And those systems can create intervention points.

Intervention Windows

This may be one of the more hopeful parts of the picture.

The transition from: Capability to:

Systemic dependency does not necessarily happen instantly. There are build-and-connect stages. Before consequential systems are connected, there may be opportunities for: testing, monitoring, security requirements, segmentation, human approval, licensing, liability, redundancy, recovery planning.

But that should not become false reassurance. An intervention window only matters if someone uses it.

Governance may still move too slowly.

Competitive pressure may still encourage deployment before safeguards are ready.

Infrastructure can create intervention points. It can also create vulnerabilities.

Perhaps Dependency Matters More than Intelligence Alone

This is where the question becomes really interesting. Maybe the most important systemic-risk question is not: How intelligent does AI become?

Maybe it is: 

How deeply does increasingly capable AI become embedded in systems that society cannot easily operate without?  And who controls those dependencies - the model, the compute, the cloud, the network, the power, the data and the permissions?

Because once you have:

Capability → Integration → Dependency → Concentration → Feedback → Propagation

The risk is no longer only technological. It becomes systemic. AI does not need to become a conscious superintelligence to create serious disruption. It may simply become deeply embedded in interconnected human infrastructure.

That is a very different proposition from: AI wakes up and takes over.

TRUE • FALSE • PIN IT

TRUE:

AI capability can materially change existing risks through speed, scale, automation, persistence, adaptability and reach.

Agentic systems introduce additional control questions because they may plan, act, receive feedback and adapt.

As AI becomes more deeply integrated into financial, informational, technological and physical systems, dependency and common infrastructure become increasingly important parts of the risk equation.

FALSE:

A future capability automatically becomes a future consequence. It does not.

A scenario is not yet a mechanism.

A mechanism is not yet a transmission pathway.

And a transmission pathway is not yet a demonstrated catastrophe.

And a local failure is not yet a systemic cascade.

The existence of a powerful AI system does not prove that every required bridge between capability and consequence exists.

PIN IT

How quickly will AI become embedded in critical systems?

How concentrated will those systems become?

How independent are their supposed backups?

Can failures spread across domains?

Will safeguards be established before consequential connections are made?

Can detection, diagnosis and intervention keep pace with propagation?

And will competitive pressure push deployment faster than governance can respond?

From the TFP Notebook

Perhaps the most useful rule is simply this: Do not confuse a future capability with a future consequence.

And another: Do not give a date to a mechanism that has not been demonstrated.

If somebody says: “AI catastrophe by 2030.”

Perhaps the first question should not be: True or false?

Perhaps it should be:

• What capability?
• What access?
• What permissions?
• What connection?
• What dependency?
• What feedback loop?
• What propagation pathway?
• What safeguard fails?
• And what evidence establishes the timeline?

That does not dismiss risk. It does not amplify it either. It simply asks us to locate the mechanism.

Curiosity Says: Find the pipe.

TFP says: Prove the joins. 

And if the mechanism connecting AI capability to catastrophic consequence has not yet been demonstrated? PIN IT and keep watching what the evidence actually does.

TRUE • FALSE • PIN IT

Pause. Question. Pin It. Then Decide.

Continue the Conversation

This Field Note explores a single question. The book: TRUE • FALSE • PIN IT explores the wider framework behind questions like this - helping us navigate an age where technology, opinion and human judgement increasingly overlap.

The goal isn’t to tell you what to think. It’s to help you become more confident in deciding for yourself.