Fear as a Moat: The Politics of P(doom)

I’m deeply skeptical of the entire RSI / “P(doom)” narrative, and I think the more important story may be why this narrative has become so prominent in the first place.

My concern is that much of AI doomerism functions less like an organic reaction to technical evidence and more like a political and regulatory strategy. The major frontier AI companies are operating in a nascent industry with enormous sovereign risk. They know governments will eventually decide who can train models, what compute can be used, what models can be released and potentially who is even allowed to participate in the industry.

If you are one of the incumbent companies, convincing governments that AI is extraordinarily dangerous creates a very convenient conclusion: only a handful of extremely well-capitalized, government-approved institutions should be trusted to develop it.

That is almost a textbook recipe for regulatory capture.

Anthropic is particularly interesting in this respect because of its deep connections to the Effective Altruist and AI-safety communities, which have spent years building institutions around catastrophic AI risk. Those networks have also been extraordinarily influential in funding AI organizations, research groups and policy efforts.

What makes me suspicious is the strange inversion of normal business incentives. These companies are simultaneously telling investors that they are building one of the most valuable technologies in history while telling governments and the public that the technology they themselves are building could potentially destroy civilization.

I can’t think of many historical examples of an industry deliberately marketing its own product as an existential threat while simultaneously seeking hundreds of billions of dollars of capital to build more of it.

The analogy I keep coming back to is the early internet.

Imagine if, instead of developing as a relatively open protocol, the internet had initially been controlled by three or four corporations. Now imagine those companies spent their time emphasizing that the internet would enable child exploitation, terrorism, hate groups, economic disruption, mass unemployment and social instability.

All of those risks would have contained some truth.

But imagine the policy conclusion was therefore that only those three or four companies should be allowed to operate the internet.

We would probably regard that today as an extraordinary attempt to capture one of the most important general-purpose technologies ever created.

That is what worries me about AI.

The genuinely dystopian outcome may not be an AI spontaneously becoming recursively intelligent and exterminating humanity. It may be something much more mundane: governments and several corporations successfully convincing society that intelligence itself is too dangerous to be decentralized.

Then the most powerful computational technology ever developed becomes permissioned infrastructure.

This is also why I’m skeptical of the recurring wave of AI “whistleblower” stories. I’m not saying every individual involved is insincere. But I think journalists should examine much more carefully how these stories originate, who amplifies them, which institutions fund the surrounding ecosystem and what regulatory outcomes those institutions are advocating.

There is an enormous difference between saying, “AI presents real risks that deserve serious engineering and governance,” and saying, “AI is so uniquely dangerous that development must effectively be concentrated in the hands of a few approved organizations.”

The first is sensible.

The second conveniently produces an oligopoly.

And in my view, that possibility deserves at least as much scrutiny as RSI itself.