Illinois Mandates Third-Party Audits of Frontier Labs and Trump Lifts Export Controls on Anthropic's Fable

A

Foreign.

B

Welcome back to the AI Policy Podcast. I'm Alok Mehta, director of the Wadhwani AI Center.

A

And I'm Nicole Larrera, a researcher with the Center. Today we'll be talking about a new Illinois frontier AI law, fable access being restored, the limited release of OpenAI's newest model, and alleged distillation of anthropic models by Chinese AI lab.

B

This is Nicole's first time on the podcast. She's been an invaluable researcher on the Wadwani AI center team up to this point, and we're really excited to have her. So welcome to the podcast and we're going to have an interesting discussion today.

A

Thanks. Thanks for having me. Well, we've got a lot to cover, so let's just dive right in. On July 6, July, just a couple of days ago, Illinois Governor J.B. pritzker signed the AI Safety Measures act, or SB 315, into law. Now, this is a frontier AI bill that mirrors many of the provisions we've seen in similar state AI laws passed in California and in New York. But it does contain some unique elements, including making Illinois the first state to require independent audits of Frontier AI labs. So, Oleg, what can you tell us about this bill?

B

Yeah, so this is sort of a third in a series of Frontier AI regulation bills that we've seen. So previously there were bills signed in California and New York. And so this bill contains a lot of similar elements to what we've seen in those previous bills. So it's a Frontier AI regulation bill. And so it does target, you know, the training of frontier models and large frontier developers. And so there are many provisions that we've seen that are very similar to provisions we've seen in these other two bills, as well as other state bills that we've seen that are under consideration in various places around the country. So these are things like reporting on critical safety incidents, transparency provisions. So requirements that frontier developers, pilots publish certain material on their website, whistleblower protections for the largest frontier developers. There are specific provisions that, again, mirror what we've seen in previous bills. So this is a requirement for a Frontier AI framework that covers the steps that the developer takes to identify and assess catastrophic risks. And there's a definition for catastrophic risks in which it's something that could cause death or serious injury to more than 50 people or more than a million dollars in property damage. Stricter transparency requirements for these developers in terms of what they're required to publish. But the most interesting thing about this bill is that it is the first of these bills that we've seen, that has an independent audit requirement, which is basically a requirement to retain a third party to assess the compliance that the company is undertaking with its obligations under the law and the various things that it's published about how it approaches assessing catastrophic risks and the steps it takes to address those. And so this requirement, this is the first time we've seen it codified into law. It does have details about sort of what requirements these third parties have to have in place in terms of technical expertise and competence, how they'll report their findings, and then how they make these findings publicly available, including provisions for redacting information that might endanger national security or trend trade secrets. And so this is, I think these are the highlights of the bill. There are, like I mentioned, there are more strict requirements on the largest frontier AI developers, but a lot of the core of this bill is similar to things we've seen in other places, with the exception of this sort of independent audit requirement.

A

Right. And can you speak a bit about how the passing of this bill signals what the government's thinking is the federal government about approaching their own AI initiatives such as the Great American AI act, which you and Matt have talked about in previous episodes?

B

Yeah. So I think that this bill does represent a shift in sort of some of the core thinking about around frontier AI model development. And especially it signals that there is growing traction around this idea of third party audits. So this is an idea that we have seen in draft legislation at the federal level. So the Great American AI act does include provisions related to this around this particular issue. OpenAI and Anthropic have endorsed an auditing market. So they've talked about it in some of their various policy documents. And so I think that this is, this represents an evolution in sort of how the AI industry is thinking about sort of testing for catastrophic risks. And this is sort of a new emerging area of consensus that appears to have traction on both the industry and the government side. There are, I think, some implementation issues that we'll have to figure out. So with independent verification organizations, one of the issues that we'll have to deal with is that the frontier AI market, right. The market for the developers or the largest, most capable models tends to be fairly concentrated. They're three or four major developers. And so I think one of the open questions we have to answer is how, how can we create a sustainable market for organizations that actually perform these functions if there are only going to be a handful of companies that they can work with? And the other thing I think one of the interesting things we'll have to figure out about the state federal dynamic is that what we see happening at the state level is sort of does send a market signal about the durability of these types of approaches. It can provide certainty to organizations that are moving into this space, but it is less certainty than if it was a federal requirement. And so one of the things that might be beneficial is if this happens at the federal level, it's just a much stronger signal that there's support for this, there's a durable market for it, and that people who are making investments in. In standing up these kinds of organizations that they can recoup the investments they're making. And so other people have been writing about this issue as well.

A

Yeah, that all makes sense. Well, we'll keep an eye on that story, but for now, let's move on to our next topic, which is the latest development in the Mythos saga that we've been tracking for quite some time here on the podcast. On June 30, the Department of Commerce lifted export controls on Anthropic's Fawn, Fable 5 and Mythos 5 models, effectively reversing a June 12 directive that banned foreign access to those models. And we covered that directive in our last news roundup with colleague Kate Corin. But alok, could you refresh our memory here? Why was the Trump administration so concerned about these models in the first place?

B

Yeah, we've covered this pretty extensively on the podcast, so I'll try to just touch on the highlights. A lot of this originated with the initial release of Mythos 5 by Anthropic, which was a highly capable model. Especially in the era of cybersecurity, it could really engage in very sophisticated detection of cyber vulnerabilities and exploits. And not only that, was able to figure out how to chain exploits together to create sort of more impactful ways to sort of penetrate a system or, you know, be able to extract information or sort of otherwise sabotage a system. And so since that time, the government, I think, has. The federal government has really shifted its thinking about the risks that AI models might pose. We've seen various iterations and levels of concern around this model that included, you know, a brief discussion of potentially imposing a licensing regime. This. These various issues definitely played into how the administration is approaching things, like the recent executive order it released on cybersecurity. But in terms of what specifically happened around Fable, the sequence of events was roughly that Anthropic released a version of mythos 5 called fable that had additional Safeguards in place, a lot of those revolving around how it answered or refused to answer questions relating to cybersecurity and finding vulnerabilities. And just a few days after that, the government issued this Export Control Directive, basically banning access to the model by foreign nationals. Anthropic, finding that incredibly difficult to implement in practice, ended up sort of blocking access to the models entirely. Part of their issue was that because the directive said that no foreign nationals could have access to Fable or Mythos, that meant that even its foreign national employees couldn't have access. And so it was really difficult for. For them to comply with. And the reason the government took this action really had to do with a report from Amazon researchers who had apparently found a jailbreak in Fable 5 safeguards. So a jailbreak is a way to sort of prompt it so that it can bypass some of the safety sort of practices that Anthropic had put into place. And so what Amazon found was a jailbreak that could allow the model to produce code demonstrating how you could exploit a cyber vulnerability. And it was apparently some communication between Amazon and the federal government that sort of re raised fresh concerns on the part of the government and led them to take this Export Control Directive. And so you mentioned that Kate and I have been looking into this. Kate Corin, who is in our economics program here and a real expert on export controls. The government here used a novel authority that's never been used before. And there's still a lot of outstanding questions about sort of their interpretation of the law, how durable this is. And now with the reversal, we may not get answers on a lot of those questions until maybe a later date.

A

Right. Which brings me to my next question. What exactly led the Trump administration to reverse its initial disruption?

B

Yeah, so it wasn't a particularly quick resolution. It took a couple of weeks. But I think what we saw is that in that time, this was Anthropic's most powerful model. And. And they really, really want people to have access to it. It's critical to their business. It's critical to them making the case that companies and governments and consumers should adopt their models. And so I think, basically, since the time Fable came out, Anthropic's been in communication with the federal government quite a bit trying to find a resolution to this. And so we don't know the exact contours of those discussions. We know that a lot of those discussions were taking place and that

A

the

B

outcome of those was that Anthropic has made some agreements and commitments to address some of the concerns that the government raised. So this includes closer cooperation with the U.S. government, closer information sharing with the U.S. government about potential malicious activity. They've seen more commitment to proactively detecting and addressing security risks. This is all laid out sort of in a letter that the government sent to Anthropic's chief compute officer. And so it's the collective weight of these actions that sort of, I think, led to the government's reversal. I think some of the things to note in addition, are that what Anthropic has said is that they address the specific risk that the Amazon report raised, so that they implemented a new safety classifier that targets and blocks that specific behavior, and that they've made a number of commitments as part of coming to a resolution. I mentioned some of them, but in a blog post, Anthropic wrote that they've committed to providing pre release government access for models sharing safeguards after there's the detection of new jailbreak, standing up dedicated Anthropic teams to work on government priorities and enhance their cooperation with government, working towards a voluntary industry standard around evaluations and tooling and best practices related to addressing national security risks. And so it seems like there was quite a bit of work done by Anthropic to sort of address the concerns raised by the government.

A

Gotcha. And it turns out that Anthropic's not the only criteria lab kind of wrapped up in all this. Several days before the Trump administration reversed its original export control directive, OpenAI also announced that it was limiting the rollout of GPT 5.6, which is its latest model family, at the request of the US Government. So what exactly do we know about that request?

B

Yeah, so I think this is a request. I have to assume that it's informed by what happened around fable that that OpenAI essentially release GPT5.5.6 only to a small set of government approved partners before any wider release. And the government cites some of the, I think, security concerns similar to what drove them to make the decisions they did around Fable and Mythos. And I mean, I think it is important to highlight that this is sort of a new thing that that has never happened before. This is the first time the US government has preemptively asked an AI company to restrict the launch of a model before release. And that is sort of a signal that they're approaching the issue of AI regulation, I think very differently than the approach they came into the administration with, which was largely a focus on deregulatory approaches to AI and sort of making sure that there are as few regulatory obstacles interfering with the rapid development and release of AI. That being said, you know, it's somewhat consistent with voluntary practices we've seen from the AI companies. So both OpenAI and Anthropic, when they released their recent cybersecurity focused models, because they didn't have the same kinds of safeguards in place that a lot of their consumer models have, you know, partly to make them more useful as cybersecurity tools, they also released them to a select set of partners. But they did that voluntarily. They didn't do that at the request of the government. And so I think this is a novel thing to keep in mind, which is now the government is asking them to release this in a limited way and it's not completely done at the voluntary. It's not a completely voluntary decision on the part of the AI labs.

A

Right. And that's definitely a notable shift. Well, I'm sure this won't be the loss that our listeners hear from us on this story. But now kind of switching gears a bit. On June 10, Anthropic sent a letter to the US Senate Committee on Banking, Housing and Urban affairs in which it accused Chinese tech company Alibaba of extracting capabilities for CLAUDE models in what they called the largest known distillation attack on Anthropic to date. So what can you tell us about this letter from Anthropic?

B

Yeah, so maybe we can start with what is distillation? So distillation is a kind of technique you can use that can accelerate the training of models and allow you to train on less data. And the way that works is that you could, you can essentially take sort of a frontier or more powerful AI model. You can query it a bunch, you can see what it outputs, and then you can use those outputs to sort of jumpstart or shortcut your ability to train your own model. And so this is a technique that is sometimes used in the AI industry. Sometimes it is used by Frontier Labs where they will sort of train a smaller model based on their larger model. It could be quite common, we don't know for sure generally in the AI industry. But in this latest exchange, what Anthropic has done is really said that the Chinese tech companies, Alibaba in particular is, is engaging in a sort of distillation at scale. And what they're essentially doing is piggybacking on all the work that Anthropic has done, the billions of dollars of money it's spent on training. And so this is allowing Chinese models to sort of gain capabilities that they wouldn't be able to if they didn't have access to this information from AI models, that they're doing this in ways that are explicitly, you know, against the terms of service. They're doing it at an industrial scale, that they're using thousands of fraudulent accounts to extract millions of data points, and that this is not the first time that this has happened, that this is a part of a pattern that we have seen with, or that Anthropic is seen time and time again from Chinese developers where they engage in sort of these types of distillation campaigns.

A

Right. And as you kind of alluded to, Anthropic has made similar allegations before against Chinese AI labs, as has OpenAI. So how does this latest accusation compare to those previous rewards?

B

I think in a lot of ways it's similar. Maybe the difference here is that we have more data about the scale of this particular attack. So it seems like this is larger than some of the other claims that Anthropic and OpenAI have made about distillation attacks. So Anthropic has raised concerns about Deep SEQ and Moonshot and Minimax and said that those labs have engaged in industrial scale attacks. Again, thousands of fraudulent accounts, millions of exchanges. But it seems like in this case, Alibaba sort of was at a different level. So just a single company having something like roughly 25 to 28 million data points extracted, whereas before, I think they had identified maybe a collective 16 million exchanges over three, three different AI laboratories. And so it seems like the scale at which this is happening is increasing.

A

Gotcha. So obviously these US Frontier labs and policymakers are now aware that this is happening. What steps have been taken to kind of mitigate and prevent, prevent these kinds of distillation attempts from taking place?

B

You know, there are a few things happening. So one is that the companies have their own mechanisms for trying to prevent this kind of attack. So some of that has to do with the way that they sort of restrict access. So for example, anthropic and OpenAI have banned access to their models from China. In some cases, companies banned use by Chinese owned companies. There are various technical measures that the labs are trying to use to detect and prevent this kind of activity and then flag potential fraudulent accounts that are engaging in industrial scale sort of distillation activities and block them. We, we've seen things like companies sort of start to sort of provide less information about the chain of reasoning or the steps of logic that models take, because that can enhance the impact of a distillation attack. And we've also seen companies sometimes sort of reroute models or answer in different ways that they suspect that the query is coming on behalf of an account that is improperly distilling a frontier model. The problem is that that all of these are relatively difficult to implement and scale. We know that there is a large number of ways that Chinese companies and sort of various organizations that provide support to Chinese companies can get around these kinds of things. So they're things like proxy servers and spoofing services, third party accounts and various other things that allow essentially like a black market or secondary market to have emerged of accounts and access to sort of US models that allows you to sort of engage in these distillation type attacks. And another thing is just generally, right, there's a trade off here between utility and protecting against these things. A lot of things that you do when you're distilling are the same kinds of things that you do when you're trying to use a model regularly to provide value and support office functions and help with coding. And so it is very difficult to know when to draw the line between is this sort of legitimate or illegitimate kind of activity. And so this increases the difficulty of sort of dealing with this. The other thing I will say is that this is also something that the government has taken notice of and sort of is thinking about how to address. So this is true at both the congressional level, at the federal level. At the federal level, there was a letter from OSTP and OSTP Director Michael Kratzios outlining the importance of this issue and thinking about various steps that the federal government can take to address this issue. And then in the Senate, there's a bipartisan bill from Bill Haggerty and Andy Kim that would be in the NDAA that would look at whether the US Government can sanction or blacklist or take other actions against Chinese firms that are sort of improperly accessing US AI models. And so we'll just have to see how this develops. It's definitely something that's on the radar for both the administration and Congress.

A

Right. And we'll be keeping an eye on that by paragraph. Single as well. But now moving on to our fourth and final topic. On July 2, the Financial Times reported that OpenAI was in talks with the US government to provide a 5% equity stake for a public wealth fund. So could you tell our audience what exactly is OpenAI proposing here and where do those talks primacy end?

B

Yeah, first thing I think we should note is these talks are very speculative. And so I'd Say that it's very early and we have no sort of serious indication that this is something that will happen anytime soon. But at least the discussion has been started and so we'll have to see where it goes. So what is specifically happening here? Well, Sam Altman and others at OpenAI have suggested that each of the US leading AI developers could allocate 5% of their equity to the US government. And there are various ways to do that. One of the specific suggestions was that it could be something like what we have in Alaska, where they have a permanent fund or a sovereign fund that invests, you know, the wealth that that state derives from oil into the stock market and then pays dividends both to the government and residents. It is unclear exactly who would be included in this provision. Certainly OpenAI suggested it. The it would almost certainly include Anthropic if it was happening across multiple companies, probably Google. We don't know if Meta Xai or other companies might be included. But you know, we're talking real money here. So at least at their latest valuations, Both Anthropic and OpenAI have valuations well north of $800 billion. And so you're looking at potentially $100 billion or more of sort of equity that would, that would go to the government.

A

Right. And this particular proposal might not end up working out, but it is definitely part of this broader push for the US Government to take a stake in AI companies. So could you tell us a little bit more about that broader push and in particular, what are the other proposals that have come out of Guess who?

B

Yeah, so there are various proposals here. I think we can maybe break this down into sort of, sort of a sort of broader set of philosophical approaches to this and then some more short term pragmatic approaches. So at the broader philosophical level, a lot of this has to do with the fact that there is this significant concern that that AI is going to fundamentally change the labor market, put a lot of people out of jobs, maybe lead to higher long term unemployment. We don't really know if any of that is going to happen. But both AI executives and economists have at least flagged that this is a possibility. And so one of the, I think approaches here has to do with if that's the case. Um, almost certainly that would mean that AI companies become very, very valuable. And so shouldn't there be a mechanism to sort of make sure that that wealth is distributed to more than the people who are invested in or that are have significant ownership of these AI companies? Shouldn't that benefit go to A broader set of people. And so they, you know, when you look at what, say, Senator Bernie Sanders has proposed, he proposes something like an American AI sovereign wealth fund that would give the public a 50% ownership in AI companies. And this is, I think, has to be read in this idea of AI is going to transform the world in a really significant way, a way that is perhaps unique in human history. And that. So we need to think about this in a way that is unique as well, which is requiring a level of active government involvement and sort of a stake in these companies in a way that is relatively unique in the modern economy. And then you have the administration. And so, you know, both in this term and in the previous term, the president has really been interested in the idea that the US Government can sort of strike deals with companies where the US Government provides some benefit to companies and that in return there's some sort of revenue sharing or equity arrangement. And generally. Right. This is not in line with a lot of strands of sort of conservative thinking where over the past century or so, we've really seen a move towards deregulation and privatization and the government getting out of the business of operating industries where previously they had perhaps had a de facto monopoly or that they had significant sort of operational stakes and moving those things into the private sector. So things like mining and package delivery and utilities. We've seen a significant level of privatization in all of those areas where the government was previously more involved. But I think President Trump has sort of bucked that trend and thinks a lot more about how perhaps the US Government can sort of share in the wealth creation and the value creation that we see from a lot of industries. And a lot of that value creation comes out of steps the government has taken to sort of facilitate their business. I think I will also say that you can also maybe sort of think perhaps a little cynically about these proposals. So. So maybe these proposals are part of this idea that, you know, companies want to curry favor with the government, that they. That this is perhaps a way for them to sort of make it more difficult for the government to regulate them. Or if you're being really cynical, the idea that you could perhaps set the stage for if the industry goes through a hard time making it easier or making it more important for the government to engage in a. In a bailout of these industries. But I don't think that explains all of it. I think that we should take these proposals, at least in part at face value, and that there are people in the industry who've been thinking about this issue for a long time and that this particular approach is not new, that people in the industry have been considering this particular approach for a long time. And some of this, some of these conversations come out of the idea that the industry has really ballooned in value in the past year or year and a half and that it's now time to consider these issues which had maybe been sort of gestating on a slower burn for a while, much more directly.

A

Gotcha. And kind of in that spirit of taking, taking these proposals at face value, could you walk us through some of the upsides and downsides of these proposals if we were to draw the kind of pros and cons chart, what would be put on there?

B

Yeah. So, you know, I've talked a little about sort of if the government has a stake in AI companies, it just might make it more difficult and complicated for the government to take action to rein in these companies. It might create this strange incentive mechanism where because the government has a stake in a company, it will want to take actions that increase the value of that company. And so that might make it harder to intervene in situations where maybe the government might have. Might have taken steps to sort of address a particular market failure or some other negative impact on either U.S. consumers or U.S. society as a whole. I think we also, you know, I think just mech. The mechanism of making this happen, I think is going to be very difficult. There's just so many open questions about how this would work. Sort of where if this happens, where the money would be invested, Would it all be invested in something like Trump accounts? Would it be invested in something like a sovereign fund? How durable could you make that sovereign fund? So even if you sort of, sort of set this money aside and say, what we're going to do is we're going to preserve the principal and then we're going to invest this money and we're going to provide sustainable level of dividends to the US for perpetuity. What we've seen, and I know this from personal exam, personal experience, because I worked on both appropriations for the U.S. house of Representatives and at the Office of Management and Budget, is that it is so tempting when we, as in the US Government needs to pay for something to sort of look for all the change under the cushions. And it's just going to be really tempting to try to take this money and use it to plug a budget hole. And so figuring out how to do this sustainably is going to be really difficult. Another issue is, you know, I think the proposal was that this would happen for multiple companies at once. So figuring out the coordination problem of like, how do you get this to happen for multiple companies in a way that sort of works? Perhaps you need regulation or legislation. If you need legislation, how exactly would that work? I think there are a lot of mechanics here that are unique and that we've never seen before, at least in this way. And so we'll have to figure out that. And I think there's also one more thing that I want to flag, which is that yes, Anthropic and OpenAI and Google are all US companies, but the impact they're having is global. And a lot of these discussions are around let's provide this money to the US Government and let's provide the benefits of that equity in some way to the US population. And I think that leaves a big open question about what does it mean for the rest of the world and what does it mean, especially if we think about sort of like a one time payout of this equity to the us what does it mean for future generations? Because if AI is going to have the impact that a lot of people think it will, it's going to affect more than just people who are alive now. It's going to affect people who are for many generations to come. And so I think we have these broader sort of philosophical questions to address as well.

A

Yeah, absolutely. Well, I think that's a great place to wrap up. Thank you Alok, for getting us up to speed on what's been happening in the news. And thanks as always to our audience for tuning in.

B

Thanks thanks for listening to this episode of the AI Policy Podcast. If you like what you heard, there's an easy way for you to help us. Please give us a five star review on your favorite podcast platform. Subscribe and tell your friends. It really helps when you spread the word. This podcast was produced by Sarah Baker and Matt Mand. See you next.