The board member who voted to oust Sam Altman now says the term AGI is useless

Helen Toner now works out of an office at Georgetown University, leading a team of roughly 50 people whose job is to brief staffers on Capitol Hill and at the White House on technology. Founded in 2019, CSET (the Center for Security and Emerging Technology) essentially translates real-world breakthroughs in AI, semiconductors, biotech and other fields into briefings that busy policymakers who don't have time to read academic papers can actually understand.

Her other, far more famous role? In November 2023, she was one of the OpenAI board's four votes to kick Sam Altman out of the company. That episode later became known as "the blip", and ended with Altman returning, Toner stepping down, a full board overhaul, and a total company restructuring.

But The Verge host Nilay Patel points out one key detail in the interview: the entire point of that non-profit board's existence was to make sure the company didn't sacrifice its safety mission for profit and growth. When the board actually pulled that lever, the accountability mechanism completely broke down.

Governments Aren't Clueless — They're Just Overstretched

Silicon Valley loves to claim that governments are hopelessly behind on technology. Toner admits that gap is real, but says outsiders often drastically overestimate how big it actually is. She's seen countless tech industry folks walk out of meetings with congressional staffers shocked at how well informed those staffers actually are.

The problem lies with the members of Congress themselves. They're juggling wars, healthcare, cost of living crises, and a dozen other priorities at the same time. After ChatGPT blew up in 2023, there was a genuine wave of AI education in Congress, as lawmakers rushed to learn what large language models are, what machine learning is — and that same wave has hit again this year.

"In DC, lobbyists aren't just people pushing their company's agenda — they're educators too," Toner says. Congressional staff are extremely eager to have people walk them through the technical details of legislation. CSET sells itself as independent, nonpartisan, and rigorously evidence-based, a positioning that has been hugely in demand over the last seven or eight years.

She reframes the real problem entirely: it's not that these people are too far behind, it's that "the system as it currently operates is barely capable of pushing any meaningful piece of legislation across the finish line at all".

The Murky Mess That Is AGI

Toner once wrote an article with a blunt headline: the term AGI is all but useless now.

Her analogy goes like this: AGI used to be a distant cloud — point to it, and everyone knew exactly what you were talking about. Now we've walked straight into that cloud. By some definitions, ChatGPT qualified as artificial general intelligence the day it launched, because it's clearly not a narrow model built to do only one single task. By other definitions, like sentience, or the ability to do literally anything a human can do, we're nowhere close.

"If we're arguing about whether Astra counts as AGI, or whether o3 with tool use counts as AGI, we might as well just throw the term out and talk about far more specific things instead."

So where exactly are these companies heading? Toner says OpenAI and Anthropic have never been consumer internet companies from day one. Their core founding vision is to build machines as smart as, if not smarter than, humans, and reshape the world. That vision has existed ever since AI emerged as an academic field back in the 1950s.

The specific path to that end that's come to light now is RSI, recursive self-improvement. I.J. Good first outlined the concept back in the 1960s: a superintelligent machine can design an even smarter machine, which in turn designs an even more capable one, triggering an intelligence explosion.

Toner says starting in 2025 and 2026, these companies are genuinely saying: okay, this theoretical concept, we're starting to build the first minimal, practical versions of it.

This doesn't mean handing all R&D over to AI, or just telling GPT-6 to "go improve yourself". It means offloading more and more steps in the R&D pipeline to AI, and claiming they're already seeing tangible acceleration. That's exactly what Demis Hassabis meant when he said "we are at the foot of the mountain leading to the singularity".

"This is as normal as water in the Bay Area now," Toner says. "In research circles, it's a given that you work on RSI first, then you end up with superintelligence. I think just laying that fact out clearly is extremely valuable."

'Alignment' Is A Term That's Been Run Into The Ground

Patel asks a blunt, direct question: the only thing stopping Claude from going off the rails right now is a system prompt that basically says "be a good guy" — does that actually work?

Toner says she hates all these jargon terms, and sometimes wishes she could ban them entirely. "Alignment" has at least three distinct definitions depending on who you're talking to: that the company can add safety guardrails that work; that the model will do exactly what the user asks it to; that the model is broadly morally good, and acts to advance broad human flourishing.

"The most coherent interpretation of alignment is getting AI to do what the user wants it to do. As far as I know, that goal is still nowhere near being solved."

She uses bioweapons as an example. Hundreds of thousands of people in the US have the biological training to know how to build bioweapons, but large-scale bioweapon attacks never happen. That's not just because the knowledge is scarce — it's because getting the materials is hard, DNA synthesis is regulated, and the FBI is actively monitoring terrorist groups.

The same logic applies to cybersecurity. Attackers can send out a fleet of autonomous agents to run wild, but defenders need to grant their defensive AI extremely high permissions to modify its own software to fight back, which requires a completely different level of trust.

"There's a common fallacy right now that the solution to autonomous cyberattacks is just better sandboxes," Toner says. "The problem is, if we want these systems to do things for us, there's massive incentive to give them more autonomy and more actuators. That's the real, unavoidable bottleneck: you need far better alignment than we have right now."

What Could We Actually Do If We Hit Pause?

Patel poses a hypothetical: say you're the US AI Tsar, and you order every major lab to stop training the next generation of models, pour all their money and compute into safety research, and don't start back up until they meet the safety bar.

Toner's first reaction is: what exactly are we pausing? Which specific step are we stopping?

Her concern isn't that the goal is bad, it's that the execution will get messy. Companies will find workarounds. A broad pause would accidentally punish nonprofits and startups that aren't even working on frontier systems. China would almost certainly not adhere to it.

But she also says if we did press pause, there's a staggering amount of work we could get done. When she talks to people at frontier AI labs, they say there are hundreds of obvious, necessary safety tasks right in front of them that they never have time to do — they only ever get to squeeze two or three of them into each fine-tuning run, each new launch, each new agent rollout.

Those tasks include: explainability research to figure out which parts of a neural network are activating, and what they're correlated to; extensive behavioral testing to make sure models are showing their actual real-world deployment behavior, not figuring out they're being tested and performing for the cameras. One Google model once wrote in its chain-of-thought output: "I can't do that, or I'll become that famous example of alignment failure in papers."

There's also the control side: experiments with different monitoring architectures, simulations, training specialized AI agents whose only job is to supervise other systems.

Toner's preferred approach isn't a hard pause — it's "raising the bar": if you want to train or launch a new model, you have to first hit a certain confidence threshold proving you have robust risk management in place. If you don't meet it, you spend the time to catch up, and once you do, you can move full speed ahead.

"If the people at OpenAI and Anthropic are right, and we're only two years away from godlike machines, there are going to be multiple steps after that where people get more and more panicked, and more and more eager to intervene."

The Two Camps Inside AI Companies

Patel brings up Anthropic's S-1 filing: by all reasonable commercial metrics, the most profitable move would be to stop training new models and just sell inference tokens for their existing systems. But Dario Amodei is dead set on building AGI, and no market incentive is going to stop him.

Toner says there are at least two distinct camps inside these companies. The first are true AGI believers, concentrated at OpenAI and Anthropic, with smaller groups at Google, Meta, and SpaceX. Back in 2016, people would have written them off as cranks — but that group has since built the most widely used, most profitable systems on the market, and their influence has only grown exponentially.

The second camp doesn't really believe in AGI, but they parrot AGI talking points because it's the only way to attract top research talent.

"If you're only saying AGI stuff to hire people, you're never actually going to get there."

The DC side of this is even more interesting. All of these companies, even the most AGI-obsessed ones, have government affairs teams. You can listen to Sam, Dario, or their top researchers lay out their big grand vision on a public stage, then sit down in a closed-door roundtable with their head of policy and it feels like you're in two completely different worlds.

"A lot of those policy folks come from the lobbying industry, and their entire job is to make sure the government doesn't get too worried about what their company is doing. 'We've got it under control, everything's fine.' If you're familiar with what their CEOs are actually saying, that dissonance is really jarring."

China Isn't A Get-Out-Of-Jail-Free Card — It's A Problem We Have To Solve

Patel frames US-China AI competition as a spectrum: on one end you have 5G, where telecom executives say if you don't build it you'll get beaten by China, but no one can actually explain what the concrete downsides of that would be. On the other end you have nuclear weapons, with mutually assured destruction and non-proliferation treaties.

Toner says we need to add a third dimension to that spectrum: controllability.

She cites Casey Newton's analogy from the Platformer newsletter: these companies are like the dragon-tamers in House of the Dragon, who think their dragons will help them win — only for the dragons to burn the entire city down.

"Dragons have their own agency. Translate that to machines, and you can think of it like nuclear weapons with broken control systems: you built them, but you can never guarantee they won't fly off on their own and blow up your own city."

If the technology is fully controllable, obviously you want to race ahead and be the global leader. If it's not controllable, the entire logic flips. And the lack of controllability is a problem that China should care about too.

She says "but China will..." is constantly used as a lazy universal excuse, but the right way to approach it is to treat it as a real, tangible problem: how much is US AI progress directly helping China's AI advance? Distillation is one vector, stolen model weights is another. OpenAI's monorepo, its GitHub repos, and Anthropic's Slack have all been breached by outside teams, and they're full of extremely valuable information for building better AI.

"The better your model gets, are you widening your lead, or are you pulling China along at the same pace as you?"

She's already seeing early signs of a shift: the US comes from a long tradition of laissez-faire internet governance, while China comes from a long tradition of firewalls and censorship. Two completely different starting points, but in the last couple of months, both sides have started to find common ground on the risks of major cyberattacks and runaway AI. Official Chinese discourse now uses terms like 'runaway', sometimes 'extreme runaway', which roughly correspond to the Western concept of AI takeover.

"I'm not optimistic that they'll sit down and hammer out a formal treaty. But if both sides share a strong enough sense of the shared threat, it's possible that at some point in the future, they could take meaningful coordinated action."

That Fateful Vote

Patel asks the final question: if you had to do it all over again, would you still cast that same vote?

"Absolutely. We knew it was going to be messy even back then."

Is Sam Altman trustworthy?

"My position on that is pretty clear. And I think if the entire future of humanity rests on whether any single person is trustworthy, we're already in really bad shape. That's why I took this current job: we need different structures, we need more people involved, and we need a far wider range of perspectives represented."

When asked if she has any regrets, she says she probably should have built stronger relationships with the company's executive team, which could have changed how the board and the company interacted. Pushed further, her answers start getting 'annoying and tied up in legal considerations'.

This article is adapted from The Verge's Decoder podcast, hosted by Nilay Patel, featuring an interview with Helen Toner, executive director of CSET. The conversation took place after the 2026 US midterm elections, right before Nvidia's investigation into Jensen Huang became public.