Fear as a Moat #3: Who Pays When AI Breaks Things?

There is another layer to the AI regulation debate that I think deserves more attention: liability.

In the first two parts of Fear as a Moat, I focused on the strange incentives created when the companies building frontier AI are also some of the loudest voices warning that the technology could become catastrophically dangerous. The obvious question was whether fear of AI could become a regulatory moat.

But there is another question underneath that one.

Who pays when AI actually breaks something?

The early internet had to confront a version of this problem almost immediately.

When Congress passed Section 230 of the Communications Decency Act in 1996, it created one of the foundational legal protections of the internet. The basic idea was that an online platform generally should not be treated as the publisher or speaker of information created by somebody else.

The immediate legal background is important. In Stratton Oakmont v. Prodigy, a New York court treated Prodigy more like a publisher because it moderated its message boards. That created a bizarre incentive: an online service that tried to moderate harmful material could potentially expose itself to more liability than one that simply refused to moderate anything.

Section 230 helped reverse that incentive. Congressional history of the law makes clear that lawmakers were concerned about discouraging online services from moderating their platforms while the commercial internet was still developing.

But Section 230 is often described too broadly.

It is not a universal immunity card for technology companies. It primarily solves an intermediary problem: if I write something defamatory on a message board, the company operating the message board generally should not automatically become the legal publisher of my statement simply because it provided the infrastructure.

That distinction becomes extremely important with AI.

Facebook hosts the thing you wrote. ChatGPT writes the thing. And increasingly, AI agents may actually do the thing.

That is a very different liability problem.

If an AI system generates defamatory information, writes malicious code, makes an autonomous financial transaction, manipulates somebody into doing something harmful or eventually operates machinery in the physical world, the argument that the technology company was merely a passive intermediary becomes much harder to sustain.

The infrastructure is increasingly becoming an actor.

That is why I think social media may be the canary in the coal mine for frontier AI companies.

For years, Section 230 gave social platforms extremely strong protection against liability arising from third-party speech. But plaintiffs and regulators increasingly learned to attack the problem from another direction: instead of arguing that a platform published something harmful, they began arguing that the platform itself was a harmful product.

In August 2026, Meta agreed to a proposed roughly $17 billion multistate settlement resolving claims that included allegations that Facebook and Instagram were deliberately designed with features that encouraged compulsive use by children while Meta misrepresented the associated risks. Meta did not admit those allegations, but the shift in legal theory is important.

The argument was no longer primarily about what one user said to another.

It was about what Meta built.

AI companies begin much closer to that second category.

And this is where the liability question starts becoming interesting.

There are basically two different ways society can govern a dangerous technology.

One is ex post liability. You are allowed to build your product, but if it harms somebody because it was defective, negligently designed or irresponsibly deployed, you can be sued afterward.

The other is ex ante regulation. Before you deploy the product, the government establishes rules for how it must be designed, tested, audited and monitored. The company demonstrates that it complied with those requirements before or while operating.

These approaches can coexist, but from the perspective of a company's risk department they are very different.

Open-ended tort liability is difficult to predict. Maybe your system causes an accident, somebody uses it to commit a crime or an autonomous agent makes a decision nobody expected. Maybe years later a jury decides that your safeguards were inadequate.

A regulatory compliance regime is much easier to model. The questions become whether you performed the required evaluation, hired the required auditor, filed the required reports and followed the recognized safety framework.

Those are much more legible risks for a large corporation.

And I think this leads to a much more interesting version of the Section 230 analogy.

Maybe frontier AI companies are not actually looking for "Section 230 for AI." At least publicly, that does not appear to be what either OpenAI or Anthropic is consistently asking for.

Maybe the modern equivalent is something subtler: converting an open-ended liability problem into a standardized regulatory compliance problem.

That would not necessarily make the companies immune.

But it could make the risk much more predictable.

If compliance with a government-defined safety regime eventually becomes strong evidence that a company behaved responsibly, the existence of that regime could become enormously valuable when lawsuits arrive later.

This is not purely theoretical.

Congress is already considering almost the opposite model. The bipartisan AI LEAD Act would explicitly classify AI systems as products and create a federal cause of action allowing people harmed by AI systems to bring product-liability claims against developers and deployers.

The logic is straightforward.

If a defective physical product hurts someone, the manufacturer can potentially be sued.

Why should an AI product be fundamentally different?

So there is already a real policy fight over where responsibility should live. Should AI companies face ordinary product liability when their systems cause harm? Should the primary mechanism for accountability become a specialized frontier-AI regulatory framework? Or should both systems operate simultaneously?

The frontier labs are clearly interested in the regulatory-framework side of that equation.

Anthropic has advocated third-party testing of frontier systems for years. It argues that sufficiently capable AI models require external evaluation and that policy should build independent testing capacity around them.

Anthropic has also been unusually explicit about the risks of regulatory capture. In that same 2024 policy paper, the company acknowledged that a well-capitalized AI company could capture regulation and warned that compliance structures with high fixed costs can inherently advantage larger businesses.

That is an important counterpoint to my broader argument.

Anthropic is not publicly saying, “Please create regulation that protects Anthropic from competition.” Its stated position is almost the opposite: regulation should be targeted at demonstrated risks, should avoid unnecessarily burdening smaller developers and should not simply ban open-weight models.

Anthropic reiterated that position in 2026, saying it does not support banning open-weight AI as a category and describing non-dangerous open models as a public good.

But incentives are still worth examining separately from stated intent.

A federal system built around expensive evaluations, security programs, audits and regulatory reporting may have competitive effects regardless of whether those effects were the purpose of the law.

The same is true of liability.

OpenAI's position provides an especially interesting case study because it has moved around.

In early 2026, OpenAI testified in favor of Illinois Senate Bill 3444. That bill contained a genuine liability safe harbor for frontier developers. Under the proposal, a qualifying developer could avoid liability for certain catastrophic harms if it had published required safety and transparency materials and had not intentionally or recklessly caused the harm.

That was almost exactly the mechanism I initially imagined when thinking about this article.

Comply with the safety framework and receive protection from liability.

But the story did not end there.

Anthropic opposed the safe harbor, arguing that the liability shield was too broad. OpenAI then changed course and later said it did not support the SB 3444 safe harbor, shifting its support toward Illinois SB 315 instead.

Illinois ultimately enacted SB 315, which retained frontier-model safety, transparency and auditing requirements without the same liability protection.

That sequence is extremely revealing because it shows that these questions are not theoretical.

The policy architecture is being negotiated right now: what counts as compliance, who performs the audit and what happens if a company follows every rule and something still goes catastrophically wrong.

Does regulatory compliance reduce liability? Does it eliminate liability? Should it have no effect at all?

Those are enormously consequential questions.

OpenAI's current position is also more nuanced than the early Illinois episode suggests. In June 2026, OpenAI published a federal frontier-safety blueprint calling for a durable national framework for increasingly capable AI systems.

By September, OpenAI was explicitly calling for mandatory capability-based national AI safety regulation and arguing that industry standards should complement rather than replace federal safeguards.

OpenAI is therefore very clearly pro-regulation at the frontier.

But its current federal blueprint does not advocate blanket immunity from severe-harm liability.

That distinction matters.

So I don't think the interesting theory is that OpenAI and Anthropic secretly got together and decided to recreate Section 230 for themselves. There isn't evidence for that.

The more interesting theory is structural.

Frontier AI creates potentially enormous and poorly understood liability. The companies building it know that, governments know that and plaintiffs' lawyers know that. Everyone is now negotiating what legal system will sit between the technology and the harm.

The frontier labs have a rational economic interest in making that system predictable.

That does not require bad intent.

Every company wants predictable risk.

If you are investing tens of billions of dollars into a new technological infrastructure layer, “we might be responsible for an unlimited universe of harms caused by increasingly autonomous systems under fifty different state tort regimes” is an extremely uncomfortable line item.

A national framework is much easier to model: one set of standards, one reporting architecture, recognized auditors, defined safety thresholds and potentially federal preemption of overlapping state requirements.

That is not Section 230.

But from a business-risk perspective, it may accomplish part of the same underlying objective: defining where the company's responsibility begins and where it ends.

And this is where the first two parts of Fear as a Moat connect back into the story.

If you believe frontier AI may cause catastrophic harm, then centralized regulation begins to sound extremely reasonable. You need safety frameworks, auditors, incident reporting and government oversight. Maybe you need international agreements.

Perhaps all of those things are actually necessary.

But every layer of institutionalized safety also answers another question: what constitutes responsible behavior by the company building the model?

Once government formally answers that question, liability changes.

A company that can walk into court and say it followed the federally mandated safety framework, passed the independent evaluation, disclosed known risks and complied with every reporting requirement is in a very different position from a company operating in an undefined technological frontier where a jury gets to decide after the fact whether it acted reasonably.

That does not mean the company automatically wins.

It does not mean victims lose their right to sue. And OpenAI's current policy explicitly rejects a blanket safe harbor for severe harms.

But it does mean the architecture of regulation can reshape the architecture of liability.

That is the part I think deserves much more scrutiny.

Section 230 solved one of the early internet's central liability problems by separating the platform from the people using it.

AI makes that separation much harder.

The platform is becoming the product, and the product is becoming the agent.

So when frontier labs tell us that these systems may become extraordinarily powerful, autonomous and potentially dangerous, I think we should listen.

But we should also ask the next question.

If that happens, who owns the risk?

The user? The developer? The company deploying the agent? The auditor who certified it? The government that approved the framework?

Or the public?

The fight over AI regulation may ultimately be as much about answering that question as it is about preventing P(doom).

And if the companies building the most powerful technology in history are helping design the rules that determine when they are legally responsible for what that technology does, that deserves at least as much scrutiny as the technology itself.