Fear as a Moat #2: Meet the Players

Following up on my last post about AI doomerism and regulatory capture, I think there is a specific funding and institutional network here that deserves much more scrutiny.

Start with Dustin Moskovitz, the Facebook co-founder and billionaire who was an early investor in Anthropic’s 2021 Series A. Moskovitz and his wife Cari Tuna founded Good Ventures, whose partnership with GiveWell created the project that eventually became Open Philanthropy and, in 2025, Coefficient Giving. Good Ventures historically served as Coefficient’s dominant funding partner, although Coefficient now works with a growing number of other donors.

Holden Karnofsky was a co-founder of GiveWell and the former CEO of Open Philanthropy. He left Open Philanthropy in 2024 and joined Anthropic in 2025, where Anthropic confirmed he was working on its Responsible Scaling Policy and safety planning for advanced AI systems. Karnofsky is married to Daniela Amodei, Anthropic’s co-founder and president. Karnofsky himself has publicly discussed both the marriage and the financial conflict of interest created by their Anthropic equity.

This matters because Coefficient Giving has become a major financial engine behind the modern AI-safety ecosystem.

Coefficient says that in 2024 alone it committed $168 million across technical AI safety, security and field-building. That more than doubled to $351 million in 2025, and as of September 2026 Coefficient says it is on track to commit more than $1 billion this year across that portfolio.

It has spent years funding research organizations, governance programs, educational pipelines and people focused specifically on catastrophic risks from advanced AI. It has funded organizations including the Center for AI Safety, BlueDot Impact and the Future of Life Institute. It has also funded programs specifically intended to move researchers, government employees and other professionals into AI safety and governance work.

So when I say I think the AI doomer narrative is being manufactured, I don’t mean that everyone involved is lying or that AI has no serious risks.

I mean that an enormous amount of money has been deliberately spent over roughly a decade building institutions, researchers, career pipelines, policy organizations and communications infrastructure focused on catastrophic risks from advanced AI.

This is not speculation about Coefficient’s strategy. Coefficient itself describes its early AI work as beginning in 2015, when it focused on “building institutions and supporting talent pipelines” for AI safety and security. Its current strategy includes developing technical and policy safeguards, improving societal visibility into AI risks and investing in the talent and institutional capacity needed to address them.

Its programs have funded graduate study, internships and career transitions into AI-safety work. Its AI governance grantmaking explicitly includes fellowships, mentorship, workshops and educational courses designed to train people for roles in government, industry and other institutions. Its broader capacity-building portfolio has also included media production and communications work.

Whatever one thinks about the merits of the underlying safety arguments, this history means the modern AI-safety ecosystem did not simply appear spontaneously.

It was funded.

It was institutionalized.

It was professionalized.

And increasingly, its ideas are moving into public policy.

The Future of Life Institute, which Open Philanthropy began funding for advanced-AI risk work in 2015, just held its “Pro-Human Assembly” in Washington featuring figures as politically different as Bernie Sanders and Steve Bannon. Contemporary reporting described the gathering as bringing progressives, conservatives and AI-safety advocates together around calls for greater human oversight and stronger AI guardrails.

At the same time, Congress now has the FRONTIER Act, H.R. 9925, a bipartisan bill introduced in July 2026 that would establish federal oversight for the development and deployment of frontier AI. It has been introduced and referred to House committees but has not been enacted.

The bill establishes tiered obligations based on the size of frontier developers, including risk-management frameworks, incident reporting and independent audits. Very large frontier developers would be required to retain federally licensed Independent Verification Organizations, or IVOs, to conduct recurring assessments of their risk-management and governance systems.

The bill would also preempt state and local laws that impose certain new substantive obligations on AI developers in three covered areas: catastrophic-risk transparency, third-party auditing or independent verification and incident reporting.

Think about the institutional structure this creates.

People within this network invest in frontier AI companies.

The philanthropic organizations within the network fund research into potentially catastrophic frontier-AI risks.

They fund educational and career pipelines that move people into AI safety and governance.

They fund policy research and programs designed to train people working in government and industry.

Those ideas are increasingly reflected in proposed public policy.

And under proposals such as the FRONTIER Act, compliance for the largest frontier developers would involve specialized third-party auditors and federally licensed independent verification organizations.

One example of this evaluator ecosystem is METR. METR began as ARC Evals, which was incubated inside the Alignment Research Center before spinning out as an independent nonprofit. Before the spinout, ARC Evals had already performed independent evaluations of GPT-4 in partnership with OpenAI and Claude in partnership with Anthropic.

Coefficient does not currently fund METR directly, and that distinction is important. But Coefficient has publicly recommended METR to outside donors, describing its work on Frontier AI Safety Policies as “one of the biggest success stories of frontier AI governance to date,” while explicitly noting that METR is not a Coefficient grantee. Anthropic has also used METR as an independent reviewer of its own safety work.

So the exact financial relationships vary, and it would be wrong to collapse all of these organizations into a single coordinated entity.

But the documented relationships show substantial overlap between investors, philanthropists, frontier AI labs, safety researchers, governance organizations and external evaluators.

Anthropic in particular connects several parts of this network. Moskovitz participated in its initial financing. Daniela Amodei is its president and co-founder. Holden Karnofsky now works there on safety policy and planning. METR has independently evaluated its models and reviewed its safety work.

Again, maybe all of these people are completely sincere. In fact, I assume most of them are.

That isn’t really the point.

The regulatory-capture concern does not require a conspiracy.

It requires examining incentives.

A regulatory framework with substantial fixed costs for safety programs, audits, reporting and third-party verification could advantage the largest and best-capitalized frontier labs relative to smaller competitors and open-model developers.

That does not mean the requirements are necessarily unjustified. It means the competitive effects of those requirements deserve scrutiny alongside their potential safety benefits.

Under that scenario, regulation does not stop AI.

It could create a moat around the companies already best equipped to comply with it.

And the more I look at the institutional history of AI safety, the more I think this possibility deserves serious investigation.

Because the same broader ecosystem has spent years funding research into the threat, building the talent pipeline studying the threat, financing organizations communicating and governing the threat and supporting the institutional infrastructure proposed to evaluate the technology. Coefficient’s own descriptions of its strategy make much of that field-building explicit.

That does not prove the threat is fake.

But it should make us extremely interested in who gets power under the proposed solution.

My concern remains the same as in my last post: the most dystopian outcome from AI may not be recursive self-improvement or some machine deciding to exterminate humanity.

It may be something much more ordinary.

A handful of corporations and governments successfully convince the public that intelligence itself is too dangerous to be open.

And once that premise is accepted, the people who already control the models, compute, data and much of the safety infrastructure become the obvious candidates to administer the system built around it.