The OpenAI board member who fired Sam Altman says we still can’t trust AI CEOs

Today, I’m talking with Helen Toner, the executive director of the Center for Security and Emerging Technology at Georgetown University. This episode is part two of our series on AI in real life. Helen is a well-known figure in the AI safety and policy community, though she is perhaps best known for having served on the board of OpenAI, when she was one of four board members who voted to fire Sam Altman. That incident, now sometimes referred to as “the blip,” ended with Sam coming back as CEO, Helen resigning, and a new board of directors. OpenAI has since restructured entirely. But it’s worth remembering what Helen and the board were supposed to do: OpenAI’s nonprofit board existed specifically to make sure the company didn’t betray its mission to build safe AI by prioritizing profit and growth over safety. When the board actually tried to pull the lever, that accountability mechanism broke, and here we are. Verge subscribers, don’t forget you get exclusive access to ad-free Decoder wherever you get your podcasts. Head here. Not a subscriber? You can sign up here. Now, the question of AI safety is everywhere. This summer, AI models from OpenAI and Anthropic broke out of their testing environments and hacked real companies. The CEOs of these frontier labs have since called for a slowdown and asked Washington for help, but the White House’s answer was a voluntary accord — one that relies in part on the kind of board accountability that failed at OpenAI. Anthropic is also about to go public, which will put a whole new set of market pressures on OpenAI’s biggest competitor. Helen has seen up close what happens when safety commitments run headlong into commercial incentives. And as she put it to me, if our hopes for the future depend on any one person being trustworthy, we’re in big trouble. So we talked about what we might do instead: what the labs are actually trying to build, whether AI alignment is a problem that can be solved, and what a smarter approach to AI regulation might look like. That includes how to think about China, which is the issue that looms large in every policy conversation about AI right now — especially now after the Jensen Huang affair. Just for clarity, Helen and I talked before Nvidia opened its investigation and Jensen ultimately resigned, so you won’t hear us talk about that directly. Before we get going, do me a favor and subscribe to Decoder on YouTube, where we put out new episodes every Monday and Thursday. Okay: Helen Toner, the executive director of CSET, on the future of AI safety. Here we go. This interview has been lightly edited for length and clarity. Helen Toner, you’re the executive director for the Center for Security and Emerging Technology at Georgetown University. Welcome to Decoder. Thanks so much. Great to be here. I have wanted to talk to you for 10 million years. I am so excited you’re here. You’ve been involved in a lot of AI safety and ethics ideas for a long time. You were obviously on the board of OpenAI during what everyone has come to call “the blip.” Now, you’re talking a lot about AI safety, about which I have 10 million questions. But let’s just start at the start. You run the Center for Security and Emerging Technology, people call it CSET. What is CSET? It’s awesome to be here. I’m really excited to talk to you as well. CSET is a team inside Georgetown, but we’re really not academics. We’re about 50 people and have been going since 2019. Basically in 2019, a group of us decided that there was a lot happening and a lot of emerging technologies that were becoming very relevant in the national security space, and the security space more broadly,. And some policymakers who had very important jobs did not have the time or the expertise to get across what was going on with those emerging technologies. Already at the top of our list in 2019 was AI. Basically our mission is to do analysis that is really rigorous, but also actionable, for policymakers and analysis that is both grounded in the technology and grounded in real national security expertise and understanding and issues. We produce long reports. We go do short briefings on those long reports to busy policymakers. We have a big data team that helps us do our research. We do all kinds of cool stuff. Can I just ask about the dynamics of that for one minute? You and I are talking the day after the midterm elections in the United States of America, and there was a big blue wave. There’s a lot of polling that says the number one issue is corruption. We’re going to throw the bums out. We’re going to start over and get rid of these corrupt guys. The AI industry, and the tech industry, has flooded a bunch of money into politics. And then somewhere next to that is, “Oh, there’s a bunch of well-meaning researchers who are going to show up and tell you how it actually works and what the real risks are.” It just occurs to me that maybe people don’t believe that’s effective anymore, that the money and the corruption and the AI CEO standing next to Trump have completely outweighed the well-meaning, sincere, “here are the actual problems” research that you do. What is that dynamic actually like? It’s a reasonable concern to have, and there are definitely some spaces where money just absolutely overwhelms everything. I’m Australian. I didn’t grow up in the US, and so the amount of money in US elections is pretty mind-blowing to me, but there are limits to what money can do. AI is a great example. In this election, there were just unbelievable amounts of money being thrown at candidates, at messages, at opposition research. There was all kinds of astroturfing on Twitter, and some very strange things trying to get anyone who talked about AI governance, regulation, safety, any of those words thrown out or not elected. It turns out to have really not worked very well. I think the reason that it didn’t work very well is one, this is an issue that voters increasingly care about. And two, also interestingly, it’s an issue that is pretty bipartisan at this point. What we see in our work is that there is a real thirst among policymakers for high-quality information. That’s members of Congress for sure, but they’re also in the executive branch, just a lot of people who have these very busy jobs and are seeing AI playing a larger and larger role in their work and their world. In DC, if you live and work here, you know that lobbyists in DC are not just people who go in and push their company’s position, though they do do that. They’re also educators. A big reason that congressional staff and others want to have lobbyists come talk to them is because they’ll really help them understand the weeds of a given piece of legislation — what does it mean for that industry and so on and so forth. What we find over and over again is we can go in and say, “Hey, we are independent. We’re nonpartisan. We do really rigorous work. We’re just here to help and we can go into the weeds with you as well.” There’s a ton of interest in that and appetite for that. It’s been really encouraging over the past seven to eight years. Are you doing that with the principals, the members of Congress, or are you doing that with the staff? Everyone and anyone. Again, we do a good amount of work with Congress on Capitol Hill, with some members of Congress, with lots of staff. We do hearings and briefings. Also, we do lots in the executive branch, with different agencies — the White House, Department of War, Department of State, Department of Commerce, all across the board at different levels, depending on the topic and what’s happening at any given moment. The reason I’m starting here is I want to have a pretty heady conversation with you, but I know what our audience thinks and it’s just that the government doesn’t get it. There’s no way the government will ever catch up to the tech industry. There’s no way they’ll ever write a regulation. They should just get out of the way. In this election cycle, I’ve seen a number of candidate forum videos where someone was like, “I’ve tried Claude once.” There’s some huge gap, real or perceived. I’m just trying to understand how you see that gap, how big you think that gap is, because as we get into AI safety and regulation, it’s that gap that has to close. There absolutely is a gap. It absolutely is a big gap. It also is possible to overestimate the gap from outside, or if you’re just looking across from California. I lived in the Bay Area for three years, so I definitely am very familiar with this perspective. I know a lot of people from the tech industry who come over and start talking to congressional staff or staffers in executive branch offices. Once they actually start having those conversations, they’re like, “Wow, these people are really smart. The staff do actually know what they’re talking about.” With principals, it is a different issue. Some of them are in the older generations, but they’re also getting smart. It takes time and it takes effort, and these people have so many demands on their time. The cynical thing to say is they’re just busy trying to get elected, but a lot of people in these roles also just have really big other topics that they have to be thinking about. They have to be thinking about wars and healthcare and cost of living and they’re looking at a lot of other stuff from day to day. I saw after ChatGPT, there was a wave in 2023 of, “Oh, maybe Congress is going to get activated on AI, maybe they’re going to actually do something.” It didn’t really turn into anything actually happening, but it did turn into a real moment where a lot of members of Congress and other policymakers started to get smart on AI. It takes some time to learn what a language model is. What even is AI? What is machine learning? They put in some of that work in 2023. I think this year we’re seeing another big wave of interest and people making the time and making the headspace to get smarter on it. I definitely don’t want to say there’s no gap or the government is super smart on AI across the board, but there are some champions who are really smart. I think the staff are generally smarter than people give them credit for. In my mind, it’s much more a question of, “Can the institutions, in the state of functionality that they’re in, get anything significant across the line?” more than, “Are these people just too behind the eight ball?” We’re going to see. There’s a whole bunch of new people. I imagine you have a new cast of characters to go and brief about the technology and what it’s capable of. We’ll see if they have revitalized the institutions. [Laughs] I’m sure it’ll be all fixed overnight. They’re very excited today, but we’ll see how it goes. The other thing that I want to ask you about is the technology itself, the nature of the technology itself, and how you are thinking about it, and the nature of the companies that are making the technology. The reason I’m asking that is I feel like there’s a pretty huge gap in the AI tools and systems most people experience and the way this industry thinks about it. My favorite example of this is we have straight-up asked Anthropic if it thinks Claude is alive and the answer was not no. The answer was a long discursion into the nature of the word “alive.” Well, it depends on what you mean by “alive” to start with. It does. And their answer was, “You need blood.” And I looked at that and I was like, “Who was talking about blood? It wasn’t us. Why are we talking about the requirement of flesh and blood to be alive?” And then some further conversation about the moral status of Claude is yet to be determined. There’s a huge divergence here between the AI overviews most people experience in Google search and “the moral status of Claude is uncertain.” Where are you? How are you situated in that debate? A big part of how I see our role at CSET as an organization, and then also my role as someone who is crossed between these worlds, is trying to translate and voyage to California and then bring back the artifacts and the recountings of my journey. I definitely think that there are a lot of people who have used the free tier of ChatGPT, or they’ve used voice-based models, which generally do worse, or they’ve just used Google AI overviews and they’ve laughed at how stupid Google AI overviews are. They haven’t used agents and they don’t know what frontier models can do. So there is just a basic piece around what the current generation of models can do. There’s a historical piece as well, which I find increasingly relevant as I’m now 10 years into working on AI and national security stuff. If you’re looking at this current moment in time, this snapshot, and saying, “Well, AI seems kind of cool, but it can’t do X,” you have to situate that in the long history of people being like, “Well, AI seems kind of cool, but it can’t do X,” for many, many values of X that have since become totally overcome by events. You can go back and say, “Okay, well, here’s how we were talking about creativity, for example, five or 10 years ago.” Or we were talking about reasoning as this insurmountable thing before you had these new models come out. A huge thing that I am spending time on these days is not just the past and present of AI, but how companies are thinking about what they’re doing, especially the two leading companies, OpenAI and Anthropic, and  this is true at Google, Meta, SpaceX to some extent as well. This idea in the Bay Area that it’s just in the water, very normalized of, “Well, first we’re going to figure out how to do recursive self-improvement (RSI) and then we’re going to do that until we have superintelligence that is AI that vastly outstrips humans.” That’s taken for granted in a lot of circles at this point. I think there’s still a ton of alpha in just taking the fact that that is what they’re trying to do and just explaining what that means. It takes a little while to lay out. “What is RSI? Why would you want to do that? What is superintelligence? What does that mean? Is that realistic?” All of that. But it is something I find myself spending a lot of time on these days, and something that there is increasing willingness to take seriously and to listen to in DC as well. Lay that out for me. I think probably a lot of people listening to this show understand what you’re talking about, but we have a big audience. What do you mean by, “We’re going to do RSI and that will lead to superintelligence?” The starting point here is these companies, especially OpenAI and Anthropic, were not set up to be consumer internet companies. A lot of people encountered this whole space after ChatGPT was launched. So the natural slot in their brain that they put a company like OpenAI or Anthropic into is a consumer tech company, where they have an app and I play with their app. I cannot overstate how far that is from the vision of both companies when they started, and also the vision deep in the hearts of so