Jordan Schneider

China's AI Mythos moment is coming. So what's going to happen next?

Matt Sheehan

Claude?

Jordan Schneider

Mythos, of course, has completely scrambled American AI policy. Trump 2.0 administration, whose AI dominance rhetoric manifested in a let it rip regulatory approach is is shook now. To quote Dean Ball, we currently have a de facto involuntary licensing pre approval regime for frontier models from the administration that promised us not that. Will Beijing also freak out once it faces down the prospect of wide domestic access to a model that can hack everything? We may not have that long until we find out. Drew AI co founder Jetong called his shot telling Elon on X that China will have a model on par to Mythos before the end of the year, even if he's off by a few months. American think tank Iops puts its date at February 2027. A Chinese mythos level model is inevitable and only a few months off. So we are now seeing politics not just shape the hardware ecosystem, but also the model layer. So what is going to happen next?

Matt Sheehan

What does this mean?

Jordan Schneider

What does this turning point mean for the domestic Chinese AI ecosystem? Chinese domestic regulation, open source in China, open source around the world, US China governance efforts as well as broader global efforts around AI safety. To discuss, we have back on Kevin Hsu of Interconnected as well as Matt Sheehan of Carnegie. We're going to start with. I'll start with Matt. So as the preeminent Western follower of all things China AI and regulation, my, and having read all the things you've written, my sense is that what they've done so far doesn't really change, hasn't really changed how these businesses have operated. It's been a little bit of this, a little bit of that. But the first kind of like global real impact we've seen on a model provider was Trump taking Claude Fable offline. So I guess I'm curious, Matt, given sort of your following of the arc of Chinese AI regulation. We've seen ripples of various things and we've seen them kind of tip their hand as to the sorts of things that they're worried about. But you know, we're not talking Jack Ma, we're not talking Didi, we're not talking EdTech, right? It's still been very sort of like, all right, live and live and let live. Is, is this stable? Like, do you what, what's your sense on once they face down a model that can really hack through, you know, big chunks of the cyber world, what the Chinese regulators are going to end up doing?

Matt Sheehan

So first is like a couple of qualifiers on like They've been super chill about this so far. I think, you know, there was a period of time when they really did impact the company specifically like 2023, sort of the year after ChatGPT. You know, companies were holding up their model releases for three months, for six months basically until regulators first had the time to put together a generative AI regulation and then had time to sort of work through these initial approvals that were essentially licenses at the time and have become a little bit more like registration now. So that was the time when it was really like genuinely slowing things down. Since then it has transformed more into just like a burdensome compliance regime. So it's not like a hard no on model releases, but, but it is a bunch of testing that you have to do. You have to submit the results and it's, you know, it's like it's a burden on the companies. It's a lot of employee hours that are dedicated to that. But the overwhelming majority of that time is spent on content stuff, on political content, on sort of social issues, basically making sure that their broad Internet censorship, you know, information management, content, content moderation regime works for AI. So the question is essentially, yeah, when the main threat in the CCP's eyes or when a new threat equivalent to those content threats comes online and that threat is from sort of AI enabled cyber attacks, then what do they do? I think they have kind of built the infrastructure that allows them to handle this about as smoothly as a country can. You know, like we basically watched from I guess April through now, July, as the Trump administration just kind of like stumbled and fumbled through a bunch of responses with a lot of the question being like, who's in charge here? Like who does this go to the ac? Does this go or not? Ac, what do we call it now? Center for AI Standards and Innovation. Does this go through Commerce? Does this go through the NSA and China? Of course, like bureaucratic shuffling and competition is inevitable and permanent. But they do have like a channel that this goes through. You have to, before you release a model you have to submit sort of a battery of tests to the cac. You don't have to submit the model weights. At least historically you have not had to submit the model weights, but you need to give them like API access. You have to give them accounts to test it with. And what I think they're going to do is they're going to first like add on a bunch of cyber related tests to those pre deployment testing that have traditionally been about content. They're going to add on a Bunch of sort of filtering mechanisms or filtering requirements related to it. Like we saw some of this yesterday actually when the CAC has been running a, like a rectification campaign on AI applications. And one of the actions that came out of that was one company, it was minimax, had to add on a bunch of safeguards against generating malicious code and they're going to sort of tack all those things on. And I guess it depends on like the size of the threat. If it's like from sort of, you know, where they are now and suddenly they're at Mythos level capability. I think they're going to do a, essentially a government project glasswing and then a company project glasswing and then a wider release in some of the limited documents that they've released so far, not documents the government has released. Sort of influential thinkers over there have put out on this. The first thing they're worried about is like the safety of the government systems, the safety of critical infrastructure systems. How do we essentially create a fast track to patch all these things first? And so I think, you know, once we really jump to that level of capability, they're going to have to prioritize sort of government and like public infrastructure. But then I think they'll do a slow rollout towards eventually maybe open full, like open weight releases haven't seemed reasonable.

Kevin Hsu

I think that all sounds both reasonable. But I think what's important that Matt said that's worth laying out is that there's actually a lot of problems. Prior art. There's a lot more prior art in China in the governing system to regulating AI stuff in general. Right. Like we go back to like deep synthesis content. This was pre chat GPT. This is back in I think 2017, 2018 land. Right.

Matt Sheehan

And synthesis is 2022, but 2020, right? Right, yeah, 2021 at least.

Kevin Hsu

Right. And then we have, you know, all these other stuff that Matt has mentioned about controlling AI in various of ways already in ways that our government has not done. Like China has all this mandatory restrictions on. You have to label AI generated content in all the social media apps, for example, that is already out there. Everybody knows whatever they're seeing on Douyin, not TikTok, but Douyin or Xiaohong Zhu. Is AI generated or not? Right. That is already out there. So there's a bit more effects of generative AI in China already now. Does the cybersecurity vulnerability completely upends any of that in a way that is quite much more dramatically unfolded in the United States? My base case is I don't think so. Right. As much as a powerful Mythos level model can discover and form cybersecurity attacks, it is equally capable of discovering and patch up the defense side of the same thing. Right. That's all there is to it. I think even before Mythos level models, AI has been pretty good at finding bugs. That was never something that AI coding tools were not good at doing. I think the Mythos fable level is the long horizon task capability where an AI model can now autonomously execute 10, 20, 40, 50 steps worth of a project for hours and weeks long end, which is actually what you need to exploit a cybersecurity vulnerability. Because it's not like you look at one code base, you found a bug, you inject something in there and. And then you have a cybersecurity attack. All of a sudden it's really mismatching multiple databases, multiple files, multiple accounts, seeing weird patterns where you can put one thing and go to another. These are all very complicated things that apparently Mythos level models are capable of doing. Same capability can be used to patch up the same vulnerability. And I think what will end up happening is maybe just a delayed release of some sort, but where the patching has to happen for all the major banks, for other regulated industry, for all the public infrastructures before the model itself gets released into the wild that serves hopefully, you know, non adversarial, normal commercial, civilian purposes. So I think it will be just a little bit more measured and live within the existing infrastructure of the Chinese governance system, in which in ways that I think is still a TBD right now. Who do you call to get the permission to release a model inside the United States government? Is it Commerce? Is it this institute that we stood up? Is it Treasury? I honestly have no idea. It's probably everybody and the White House Chief of staff's office as well. So that's really the. The rules are confusing at this point, but I also think we will have to get there. So in a way, this is maybe the one case where China does not look at the United States for tips and, you know, tricks on how to regulate a emerging trend of some kind in which China has been looking to the west in general to learn about regulating antitrust, learn about regulating your stock market and all sorts of other things. But this is at the frontier where China may be just going to be actually in de novo territory when it comes to regulation.

Jordan Schneider

Well, I mean, Matt just said they're going to do Glasswing, right? Which is like not de novo territory that's following down.

Kevin Hsu

Well, That's, I mean, I don't know if that's gonna happen or not. Right. Like, I think that's, that's how we always feel like they will do something that we will do a month ago because that's what we did.

Jordan Schneider

All right, so let's, let's talk through the, like the. Not. All right, so let's do glasswing scenario and then let's do not Glasswing scenario. So glasswing is like, all right, glm, CHIRPU or ALI or whoever sends CAC an email. They're saying, look, we got this new model, it can hack the shit out of everything. We probably shouldn't just put it up on open router. And then maybe it's just the government. Maybe it's the government plus SOEs. Maybe it's the government plus SOEs plus your 20 favorite companies in China, they all get access to it for three months at some point. You have this like awkward thing where the company says, we can release it now. And the regulators are like, oh, no way. Like, we don't want like a giant sort of like cyber attack on our own things by open models. But like that gets resolved at some point and then two, three, four months later the model gets released and we sort of have a, we have a mental model for that. Like the, the crazier one I think is some model developer just like really wants to compete and releases it. And right now, because there aren't, there isn't the current regulatory kind of dynamics, like this is a. There's enough competition in the ecosystem that you just press publish on a closed model or even an open model that has like really good, kind of like really a dramatic step up in the current capability that you can get out of models in China. So let's live in that world, Matt. Like what GLM 5.7 drops and it's just like way better. What happens next?

Matt Sheehan

So let me say why I don't think that will happen is like, yeah, letter of the law, letter of the departmental regulation. You submit these filings, you just have to complete these certain tests. Those tests don't include cyber. And so once you've submitted those and the CAC sort of accepts those, that's their licensing form, then you can go. But it's not just that they run through this very formal process. Like CAC is constantly touching base with all of these companies. I don't know if these are still going on, but at least as of a year or two ago, they were having weekly meetings where they called in the GR people And some policy people, government relations people and policy people from the companies to just like talk about what's new in AI policy and regulation. So when I was talking to someone, this was around the time of the Hollywood writers strikes and they were like, yeah, the CSE called, you know, called us all in the other week and they wanted to talk about like labor and the Hollywood writers strikes and like what should, you know, what do we think about this in relation to China?

Jordan Schneider

Want to chat?

Matt Sheehan

Yeah, I just want to chat your

Jordan Schneider

friendly neighborhood regulators, you know.

Matt Sheehan

Yeah, I mean that's been cac. CAC has attempted this type of kind of rebrand repositioning from like bad cop to more like bad cop. That's like you do something and then I will tell you or punish you to like let me help you comply. And part of that is to hold these regular touch, you know, touch points with the companies. So I think in that environment it would be very surprising if a Chinese company was like, let's just not give them a heads up and then let's just drop this and you know, technically we've met all our requirements so let's just release it and let it go. That you know, that'd be very, very bold and like self destructive behavior by a Chinese tech company. I think. So I guess that's my, I mean

Kevin Hsu

that doesn't even happen today, right? With much less capable models. Like I don't think any Chinese companies has ever not released, not given the regulator a heads up or the registration process that we talked about before it's

Jordan Schneider

released into the, I mean but there's dede, right? They were told not to list and they listed.

Kevin Hsu

Well, that's a totally separate topic. I mean let's not conflate a New York listing to a model release to,

Matt Sheehan

to, to bolster the case for this. I think one thing that we do, I think we have some insight into, although I want to get a little more clarity from the companies, is that they don't have to run the full battery of tests for every sort of iterative model release. They do not do a new registration for each new model release. So like if you go into the model filing system and you look up like Deep Seq, how many models has Deep Seq registered? Maybe pick Ali and it'll be much, much less than the sort of total number of models that they're Hosting on, on GitHub or on hugging Face. Excuse me. Okay.

Jordan Schneider

I think we're, I think we're missing the theme a little bit. So the, the, the what we have now right. Is we've, we've crossed the threshold from like regulators just kind of keeping an eye on it to regulators understanding that there are like big national level impacts if they get this wrong. And that's not like, that's not like a few people harming themselves, but you know, hospital systems going offline, financial institutions being, you know, potentially shaken to their core. We've seen Besson always freak out and call the bank, you know, call the heads of the bank saying you guys really have to pay attention to this. So I guess like both of you seem to on a sort of like status quo base case. But I guess my, my, my sense is that like you had a status quo base case in the US which was proven like radically wrong about how the US government was going to relate to these models going forward. So what like what is, are, are there? What let's live in the worlds in which the Chinese government decides it really needs to change its approach as opposed to just like putting a few more you know, cyber questions on its like registration form for these models where you know we've been having this debate for a long time. Right. Is China AGI pill, You know, is, is, is the politburo cagi pilled? And it's like they're not like people cannot be until they are when it just starts to stare you in the face how powerful and impactful these things are.

Kevin Hsu

Right.

Matt Sheehan

I think the Glass wing scenario would be not base case. Like if they were to do a first government only, then SOEs, then big companies and then wide release that would be very different than their current status quo. So I guess I think that, I think that it is going to prompt a significant change in that.

Kevin Hsu

I also think the status quo is different in the United States versus the status quo in China when it comes to the level of existing regulation. When it comes to AI, that's where our base case is. Not starting from the same starting point actually. Right. I think one thing that we need to keep in mind about what how glasswing actually came about, which is why I think a same repeat will not happen in China most likely is that glasswing was started by a private company.

Jordan Schneider

Yeah.

Kevin Hsu

It was not initiated by a government agency telling a AI model company that made a really powerful model that hey, maybe we should have a close pre review release process. Let me give heads up to the banks and the critical infrastructure and then we release it to the public. Anthropic chose the members of glasswing, not the United States government. And that's why the US government has been kind of scrambling to respond and now that's been our posture, right? To not over regulate and frankly to commit to not having a licensing regime to the AI model world up to this point. And now we sort of have a weird de facto one. But that's why we were scrambling to begin with is because a private company outside the government's control started it. I think a similar pre approved or pre release situation that Matt was talking about if it were to happen in China will most definitely be be started by a government agency and the list will be pre approved by the government agency before anything else happens. That is a, I think it to me by nature a very different response to how glasswing manifested itself to begin with.

Jordan Schneider

So. So who's in and who's out? Like what companies are on the bubble?

Kevin Hsu

You're talking about like the one getting

Jordan Schneider

the Chinese glass wing. Like who gets it first, who gets it second? Like what is it?

Kevin Hsu

Well, I mean all the government agencies will get it right. All the state grid companies will get it. The petroleum companies or the strategic new resources frankly all the companies that are non tech and have really bad probably buggy infrastructure. And that's the thing that you know we're worried about here too. It's like why aren't the state of Tennessee getting glasswing access? Because frankly that is not a very good IT system that's going to get hacked very, very easily. Right. But that's. I don't think the list will look that different probably.

Matt Sheehan

I guess.

Jordan Schneider

Yeah, maybe it's more for the China, for the, for the private companies like who gets it and who doesn't.

Matt Sheehan

Yeah, it'd be especially interesting for the. Say it, say it happens, say it comes out of your pool then yeah, I think it's, it's government ministries, it is central so ease. Then maybe it's low then maybe it's provincial governments, local so stuff like that. But what, what is what happens to Ali in the case that dripu creates at first I think is a pretty big question. That might be one where the government steps in and is like you guys have to work with them on this. I mean Al probably wouldn't be that far behind in either case but like a good amount of government and like wider useful infrastructure is run on like Alibaba cloud and so the idea that that would be left insecure I think is probably not very likely. Maybe if you take kind of like maybe a more extreme reaction than this type of like government glasswing thing would basically be if like at some level the security Organs step in and are like, thank you for bringing us to this point Drip who Thank you for bringing us to this point Deepseek but like, we'll take it from here. This is now at the level of a sort of a national priority that has to be brought in some way fully in house. I think that would be a really bad idea because they, these groups just don't know how to like run these companies and they don't know how to train the models at the same level. But I, it's not entirely out of the question. I think a lot of this depends on essentially like how, how high is the government's confidence relative to like how much it needs the companies. Like when the government is feeling very confident and they don't really need the companies, they tend to crack down a lot. I think that was essentially like the tech crackdown era. They felt free to like impose very heavy burdens on the companies. Like, we're doing great in China, our capabilities are great. Post ChatGPT government's feeling much more insecure and they're like, oh dang, like maybe there was a cost to that whole tech crackdown. We actually have to give these companies a little bit more leash and let them run with it. I think one of the interesting changes over the last year, two years has been like in the wake of deep seek, the government once again feeling good, like, okay, maybe China is actually, you know, back at or close to the frontier on capabilities and we can start getting like more hands on with our companies again in with the most extreme version of that being, you know, some version of a, of a government takeover. I don't think that's likely and I don't think it would be smart. But I think if they sort of get a little bit too high on their own supply of confidence in their capabilities, then I think, you know, that's the extreme version of that wither open

Jordan Schneider

source in this world.

Kevin Hsu

Kevin it's kind of interesting that within the last just two or three days, both the founders of Minimax and Drupal made pretty high profile pronouncements about open source, among other things, right? The Minimax CEO is committed to, I guess, donating or committing 1% of the Minimax market cap to supporting open source. I think what I've heard is that they're going to start some sort of a foundation of their own with that seed money to promote open source in perpetuity. Drupal's founder also wrote a internal memo that was quote unquote leaked onto Xiaohongshu that Got translated and I just read it yesterday as well. And he also commits to supporting open source as part of, you know, their rather grandiose mission statement of bringing AI to all of humanity, et cetera, et cetera. And obviously people have read the inklings that China may stop releasing open source model because of their capability increases. So we have a lot of mixed signals. I tend to probably stick closer towards the signal given by the people who are actually making the models who are all relatively well connected or at least well, you know, they have a chain of communication with the relevant regulators in China for them to be able to operate, which is that I think their commitment to releasing open source models for the most part right now as part of the strategy is not going to run counter to anything that the government may do. We'll see how long that lasts. But I think the commitment from some of these labs in China is still very strong for open source because frankly it has gotten them this far right in a way that is non trivial as far as the progress of glm, progress of Deep seq obviously. And you do have a population of model makers in China that is increasingly closed source as well for their own commercial needs. So it's not a national strategy that was dictated from the top to begin with, which is a common, I think misconception for a lot of people. There is a distribution of opinion when it comes to why open source, is it good or not for my company, for humanity, so on and so forth.

Jordan Schneider

So let's just for some context on the article that Kevin was referring to. Last week we had a Reuters report saying that Chinese authorities have held meetings with top tech firms over the past months about potentially restricting oversees access to China's most advanced AI models, including those yet to be released. At those meetings, participants discussed putting limits on the most advanced Chinese AI models, both closed source and more open versions. Officials talked about making a leak or theft of proprietary AI an offense over the National Security Law. The scope is still being discussed but like I think there's a way of interpreting the sort of moves by the dripu and the Minimax CEO as like trying to hold their grand, hold their ground a little bit and say hey wait, no guys like this is, this has worked really well for us. We actually think it's a positive thing. But clearly, I mean it's hard to read into an article like this. But like all you can say is like there are currents within the Chinese government ecosystem that are concerned about wide access to increasingly powerful models, which is not something we've really Necessarily seen signaled even before that. Sorry, that's right.

Kevin Hsu

No, and I think just to add on to the Reuters article that you mentioned Jordan, which got a lot of play I think on the same day, or maybe close to the same day, there was a Chinese official who actually gave a speech at this like UN air gathering in Geneva as well where this person committed to open source as you know, a very important part of China's like overall posture towards global AI in which it's from a, you know, official voice. So you have leaked sources that went to Reuters and then you have an official person giving a public speech that commits to open source. And I think we probably will get the best signal of any of this stuff in a few weeks when Shanghai hosts a world AI conference where Xi Jinping himself is slated to speak. My anticipation is that he will say at least something about open source or open source AI and we will see if that happens or not. And that will be the clearest signal from a country level perspective as far as where does open source stand in the grand scheme of things.

Matt Sheehan

Can I couple comments on the article and stuff like that? Few interesting points. One, I'm pretty sure from memory that the Reuters piece says that the relevant like regulators that were in the room were Ministry of Commerce and ndrc. Ministry of Commerce in relation to like export controls. That's very interesting because they are not, you know, Ministry of Commerce is not the main AI regulator. That's cac. They are not like the people who are testing models and they are have been pretty out of the loop on AI stuff other than interfering in the Manus deal and other sort of export control related things. I think one version of this is they create a ability to export controller to control the export of something like model weights. This is what they did with algorithms, recommendation algorithms in advance of the TikTok deal. So when it became clear that the US was getting ready to ban TikTok and was trying to force a sale China, I'm pretty sure this would be. The Ministry of Commerce essentially put it, wrote it into their controllable items list. And I'm getting the details wrong here that stuff like algorithms can be controlled in that way. So essentially they gave themselves like a veto over a future TikTok deal which they did like hold on to until the end. That was an open question. It's like is the central government going to approve this? So that might be one reading of these events because Commerce really, they're not the security people. They could be essentially creating a veto over future manus deals. They could be trying to find a way to get in the game when it comes to like model releases, although I don't think they would necessarily succeed at that. I personally am skeptical that they are going to impose some kind of like sort of wide, wide controls on open source. Not just because of how beneficial it has been to the companies, but this has been a very beneficial political, global political narrative for the CCP in relation to AI. There are global AI governance initiative in 2023 came out right before the first AI safety summit at Bletchley. And they've said like from the very beginning, like China stands for global access, stands for, you know, not using technological hegemony to exclude countries, stuff like that, trying to very clearly position themselves as the opposite of the US US wants to control this, it wants to prevent you from getting access to this. And we are the country of ocean openness. You know, in an extreme situation with China's like real core interests are threatened, they're not going to opt for nice diplomatic language over core interests. But I think it does. They do feel the benefit of that. And so I think if they were to start to impose some level of slowing down or restrictions on open source, it would be done in a way that was not a sort of a total shut off and would allow them to maintain this line of hey, yeah, no, we still are the open country, we still are sharing with you. We're just doing some internal security checks first and then we will share something along those lines.

Jordan Schneider

Here's where I think you guys are wrong. In the world in which AI keeps getting more powerful and is on some sort of exponential. Right. I find it hard to imagine like a scenario where even if it's where the Chinese government is like helping to enable random Chinese gangs or just like criminal enterprises, what have you, the world who wants to hack China to use Chinese technology for that purpose. And, and if we're six months later, 12 months later, 18 months later, and the world. And so China isn't China's project glasswing doesn't harden the entire Chinese society from hacks that it's open weight, from hacks that its companies open weight models can do. I think you start to get a balance between like, yeah, the rhetoric, the company's interest, but then you really start to get the security state being like, okay, that's fine, we're having like 100x scams. We're having all of these like domestic disturbances that are caused by people using AI or even AI itself because it's powerful enough to Just like if it's open, wait, then you can set things off and run. So I hear both of your arguments that there would be real sort of diplomatic costs, there would be economic costs, but I think it is worthwhile to take the like, okay, this gets really powerful really quickly and starts to have these like emerging capabilities that we can't just like after two months of an access window, feel comfortable we've hardened ourselves from. Do those delays turn into six months? Like how do you even make that argument if you keep coming up with like more and more more things? It just seems more murky to me. And the arguments as the technology keeps getting more powerful seem to be stronger and stronger from a like, I don't know, zhongnanhai brain hat to just slow this thing down.

Kevin Hsu

But I think that conflates model capability trajectory, which I agree with you, is probably on a much higher curve than any of us could ever visualize or even feel on a daily basis. But you're conflating that with whether open or closed source release even matters in that scenario, right? Like if the model is capable enough to do all these damages that you're talking about, and we're probably seeing evidence of that already pre approval glass wing type situation to hide harden your infrastructure, then allowing the model to release, whether as a commercial product behind APIs or as open weights on the hugging face is actually not that different because these models are so massive that I think we tend to over we kind of spin our brain into a rabbit hole sometimes thinking that as soon as the weights are out there and these weights are supposedly going to have cyber capabilities, that some random person will just be able to hack the state in no time. The amount of infrastructure, the amount of cloud computing that's required to even properly deploy, let alone run these things, to do something that is vaguely useful or vaguely adversarial is ginormous. And the way you would actually regulate this scenario is of course to talk to a finite number of companies around the world that have compute of this scale to begin with to make sure that doesn't happen on their infrastructure. That has nothing to do with whether the model is going to be released as an open source or as a closed source trajectory. Right? So I think we're kind of conflating model progress with whether open or close matters, when the open side of things actually accrues a lot more benefit and is by definition actually safer and more secure. Which is the kind of thing that I don't think is talked about enough, which is that, you know, none of us want to live by a house next to a park that has no light over it. As soon as there are lights over the dark park, you know the property value goes up. And I think the same you can think about with open models which is that having the weights be actually published and accessible is a much safer way to release much powerful capabilities than the closed ones. At least about as safe as the closed model equivalent over there.

Matt Sheehan

Sort of do open weights matter is it's like a very deep and complicated question. I'm not quite so sort of sanguine that it doesn't matter at all. I think there are a lot of like well resourced actors who could still put these things to use. I think maybe the one of the key questions here is essentially like are we going through a period of a transition period between sort of our status quo to a bit of chaos across cyber globally to a future state that is just more secure systems all around? I am not a cyber person. I have some friends who are, who are very deep in this world and I put this question to them somewhat regularly. And while this is not sort of, this is not like settled a settled matter, I think the consensus is gathering around the idea that yes, it's going to be chaos for a while, but you can effectively eliminate bugs in existing software, you can patch everything that we know, you can create provably secure software. And it's not just that the next level mythos will suddenly discover new vulnerabilities that are in there. And I think it's possible that China sees this transition period, we can get through this transition period with a certain amount of lag time and then we will be able to hold on to all these other benefits of open source that there are. I think you're right to like point to the, I don't know, it is a security. CCB is a security and control first institution. And the idea that they will just be like, yeah, it's cool, it's fine to be sort of hacked by, you know, people in, you know, pick your country. I, they're not going to be sort of sanguine about that, but I, I do think that they might see a path forward in which you can harden the systems within China and then accrue these various benefits of open weight releases.

Jordan Schneider

I just think once there is an example or someone writes a memo in the Chinese state saying that, you know, given X like separatist group from abroad that wants to take down the party, like did this thing and like uncovered, leaked these documents, whatever. Oh, and by the way they weren't part of project Glasswing, they just like had access to some GLM model. Like that is just going to be a really dramatic shock to the system. And to the, to Kevin's point, the question to me on the open versus closed source is like can, if you're just providing API access, like can you actually lobotomize a model such that it just doesn't do cyber things and then you can have all the sort of like productivity uplift benefits but like you just can't hack with it or you just can't make bioweapons or you just can't do scam or you just can't run a scam farm or whatever. You know, pick whatever capability you're really scared of. And if it's an open source open weight model that's like presumably more difficult or like less harder to trace back who does it? I don't know if I can just like buy, I mean we can already smuggle a lot of Nvidia chips, right? Like I could presumably put 20 of them together and, and call some grandmas, right? So I don't know because I just

Kevin Hsu

think it's more than 20.

Jordan Schneider

Okay.

Kevin Hsu

It's a, it's a non trivial amount of hardware. Where does someone really needs to be hell bent on doing this a very powerful non state actor, right, Which I think we might want to talk about at some point, maybe not today but like how does that scenario get regulated? Because I think that is one thing that both China and the US and probably every country in the world actually agrees on is how do we come together to figure out a way to prevent non state actors of malicious intent from one having the resources to even have their own on prem cluster of more than 10,000 Nvidia chips before we also think about how to regulate their access to models whether it's closed or open.

Jordan Schneider

Well, I mean it's funny right, because like I don't think you need to be building your, your server in the jungle.

Kevin Hsu

Like no, you don't even Credit Suisse

Jordan Schneider

like, like Standard Chartered helped Iran, you know, break sanctions for.

Kevin Hsu

And this is with the assumption that commercial APIs run by OpenAI Anthropic is somehow bulletproof, which is not, I don't,

Matt Sheehan

I don't think they're bulletproof but I mean they, they are constantly monitoring what's going on there. Like they're, they're monitoring it for their own goals. Like they, you know recently they had this report of, you know, a handful of people in China have used OpenAI have said a handful of people in China have used their models to try to generate sort of data center backlash material, stuff like that. So I mean that's, you know that's a lot more scrutiny than you can have over open weights once they're released. I really not my area so I'm and I'm not like a hardline anti open source person but I don't just have sort of faith that it's going to work out in that way. I think there are very well resourced non state criminal organizations or North Korea or you know someone like that that will have like the technical capabilities and the money to use open weight models in ways that they think is like beneficial to them. They can make more money than they lose.

Jordan Schneider

I mean we just had Boko Haram like asking chatgpt how to like overrun a military garrison, right? They like you know some, some rebels in the jungle in Nigeria figured out like plenty elder hacks, right? This stuff isn't rocket Science and ChatGPT

Kevin Hsu

did not stop that from being able to help them out, did they?

Jordan Schneider

So maybe they had to go to deep sea. I think they like had to try a few or whatever. Let's not put it all on OpenAI but to Kevin's point Matt, I don't know what can the US and China agree on or what's there even to discuss.

Matt Sheehan

Yeah, this is tough is when Besant sort of made his first announcement which at the time China didn't like co sign or confirm but when Besant first commented on like hey we're talking to China, we're going to see set up a dialogue to talk about and I won't have the exact phrasing right but I think he said something like protocols to make sure these models don't fall into the wrong hands or are not used by non state actors, something like that. So that was like clearly sort of the US's focus from the beginning. And I think when you think about what the US and China can agree on, it should be stuff in this non state actor thing. Like China should operate on the assumption that no matter what the US says or any regulations we have, like the NSA is going to be using the most advanced models it can possibly get its hands on to hack China. We should have the assumption that the MSS is doing the exact same thing in reverse. So like there should be an acceptance between the two sides or an appreciation that they're going to be using these things against each other where it is in both of their benefit. I think would be sort of controlling the non sydatric front. But then you run into this question of like open source and we have different incentives on there. The big US companies don't have a big impetus to play in open source space, the big Chinese companies do. And now that sort of disrupts things. I've heard some people have sort of floated ideas of what you can do around sort of building safeguards either certainly into like the API access, but maybe even certain safeguards and filtering into open weight releases. I don't understand sort of how far that can go, how much that can do for you. But I think we might run into a situation which like in theory they both are faced with the same threat. But the fact that the companies and sort of government narratives, government positioning on this makes it such that they are, they can't get somewhere meaningful when it comes to that. I think the best they're going to do is on sort of domestic testing.

Jordan Schneider

I don't know, I don't see a lot of it, but maybe there's some lessons to be shared. I don't know.

Matt Sheehan

Yeah, I think that there's lots, that there's lots of productive things that could be talked about. Like I, I think it's a sort of a low ceiling, but there's like a lot to do in that space. Like generally speaking, Chinese frontier AI companies do not test for extreme risks to nearly the same level that the US companies do, or they don't, they don't have the same sophistication in the way

Kevin Hsu

they do that test.

Matt Sheehan

It's like it's happening in small places but it's not nearly the same. And there is not a Chinese Casey equivalent, AC Casey equivalent that has like been developing sort of frontier AI risk expertise in that way. I think it would be possible and very good to find a way that the US can safely share information about how do we run these tests? How do we test for, for these capabilities in a way that is not capability enhancing but allows the government or regulators or companies to essentially sniff out dangerous capabilities earlier? I think that's. We're not signing a treaty. We're not both agreeing that because you tested for this, I will test for that or setting the same thresholds, but it's sharing knowledge of things that sort of neither side wants to happen within its own borders. Yeah.

Kevin Hsu

And I think to the extent that we want to bend ourselves backwards to find more overlap in a Venn diagram of like a, you know, bilateral dialogue, there's a lot to talk about when it comes to the actual Societal impact. Right. Like, I think when it comes to just a very simple question of labor displacement, what is the legal justification to say, potentially laying off people in certain companies because of AI productivity? Matt, you've written about this before. I think there's a lot of just like, let's share what is each other's trying to do or not trying to do and see if it could make sense. Because the productivity disruption is coming for companies and outfits in both countries, like, as we speak. Right. And the companies have very similar motivations. And I think that there's a lot that the governments are wrestling with on both sides to figure out what is the right way to transition the everyday people impact. Not just a very serious, scariest scenario kind of cyber security or testing safety impact that, you know, we talk about. And I think there's a lot to talk about there without ever having to agree on anything, which is hard to do, and just be like, what are you doing? What am I doing? Oh, that sounds interesting. We're doing this way. All right. And then we just, like, go off to our own separate spaces to figure it out, you know, for our own people.

Matt Sheehan

I, like, I participated, I participate, I organize dialogues that talk, talk about exactly that stuff. And I think it is, like, really interesting and useful to compare notes on it. I'm a little. I'm a little more skeptical of, like, the value of governments doing it in the sense that, like, I think it's. It's good to have mutual awareness of what the other side is doing. But I think the odds that we are going to really, like, learn from each other's responses to labor displacement, I think is significantly lower. It's just like, the more. More kind of, the more something gets into political culture, social institutions essentially, the less, like, compatible the lessons are. In some ways, even as I'm saying this, I'm like, mentally walking it back because China has, like, picked up on a lot of regulatory mechanisms that we have built, and they've adapted them to their own situation. I think maybe specifically on labor stuff, it's pretty, pretty different attitudes towards labor and sort of legal protections in that way.

Kevin Hsu

But I'm just wondering something that's like, less cultural but maybe more technical. Right. Like robo taxi urban regulation, for example. Right. Like, as far as to the extent that the streets may look different and the cities are laid out different, that is kind of about it, right. As far as, like, how would Wuhan do it differently than Phoenix and vice versa, you know?

Jordan Schneider

Yeah, but I mean, that sort of stuff doesn't Feel like it matters. Like, yeah, maybe it's like get some good meetings, like get, get some miles.

Kevin Hsu

Like it may not matter for podcast, but I think it matters for people on the ground.

Jordan Schneider

Well, no, I mean like I guess from a, from a, like what do

Kevin Hsu

I mean when I say I think it's like better mental energy spent on that than trying to think of separatist groups, you know, triggering the highest wire of the CCP security apparatus personally. But hey,

Matt Sheehan

there is a track record of doing this type of stuff in relation to like environmental stuff.

Jordan Schneider

Like it just, it seems so like 90s, 2000s. I mean it's a nice thought that we could like find some like common ground to collaborate on both staring down our AI driven futures.

Matt Sheehan

Partly depends how AGI pilled you are. And I think one thought I had earlier when you're talking about like, you know, CCP not AGI pilled until they are. I do think that as sort of big and scary as the cyber stuff is, I don't think this leads them necessarily to being AGI pilled in the way of like, you know, fast takeoff rsi. Maybe it's, there's some relationship to rsi. There's a pretty big difference between like sort of losing control over super advanced AI systems and like it's really good at reading software and finding bugs in it. So I'm, I'm going to be curious to see how much we see more like AGI ASI type language because of this sort of new, because of cyber stuff coming to the forefront.

Jordan Schneider

Well, I mean Xi has said he's worried about risks of technological loss of control. So it's, it's bubbling up, it's starting. Even if we're not entirely there. I mean, I guess, sorry, let's qualify my, like it doesn't matter. Like I think it is, I think it is nice and I think it is like net positive for both countries in the world for the US and China to have like a productive discussion about the sort of second order impacts of this technology showing up. Like, I'm not sure it's interesting how like the learning or that we're already seeing regulations like kind of happen in parallel. Like China's had this whole sort of like anthromorphic AI debate about how AI shouldn't pretend it's human or not. AI in the US we're having a big markup on like a children's AI safety bill, which is something that, you know, minor mode is something that's been rolled out in China or talked of over the past Few years, weeks. So like the systems are kind of like moving in parallel. I don't necessarily think that like Senator Hawley was like inspired. Well they kind of, they kind of do actually like when, when you see an article in the US about how like China's banned video games for kids or whatever, like you see there's a

Kevin Hsu

lot of reaction here.

Matt Sheehan

Yeah, I think there's like that's a good example. Like one, you know, pop champagne. I think the regulation went into effect two hours ago. So you know, Companions now regulated in China as of noon our time. But like the China in that case, like the, the Chinese regulation was I think pretty directly inspired by state level bills in California and New York. They use different mechanisms. Like we are largely using the mechanism of private right of action, like essentially enabling lawsuits against companies for harms whereas they are going through this sort of central approval and much more like hands on technical mandates thing. But there are definitely provisions in their bill that have been drawn from our state level bills. I wouldn't be surprised if we start seeing sort of flows in the reverse in terms of ideas and yeah, I do think this stuff matters even beyond just the like, you know, it has a real impact on people's lives. Like the attitude that the publics have towards AI, their receptiveness to it, their opposition to it is going to be rooted a lot in like what did they see happen with Companions and you know, their kids as opposed to like what's their timeline for rsi. So you know, indirect, hard to predict like the sort of the, the, the, the direction that each of these things is going to push in. But there is like mutual learning going on. I think it could continue. But when it comes to like the, the sort of transnational large scale, maybe catastrophic risks I think are in kind of this different bucket that has a different set of mechanisms and like things that we can and cannot talk about.

Kevin Hsu

Yeah, kind of just want to build up on that a little bit too. I think actually taking care of the smaller quote smaller issues of AI's everyday impact will directly impact. What does being AGI pilled even mean as a government to begin with. Like we here can be as AGI peeled as we want, but if we don't actually show some of the benefits or just prevent the harm of AI for everyday people, we're going to get data center moratoriums up the wazoo if we don't already. Doesn't matter how AGL DC is if you know, rural communities won't let you build a data center. So what's the point, right, of being that committed or that pill to the mission or to the goal. Right. And similarly, perhaps in China, taking care of it is the best way to figure out whether the government wants to be AGI pilled or not. And what does that even mean when it comes to a national strategy? So I think those two things are way more linked as far as like cause and effect than we probably give them credit for.

Jordan Schneider

My, like my hypothesis two or three years ago is that the sort of like downsides of AI AI safety at some point would just be framed as like a Western plot in China to get them to slow down relative to the US and that seems like that has definitely not happened at all. Like for their own reasons, like for their, you know, given their own, given their own incentive structures like the Chinese political class, Chinese regulatory structure has internalized themselves that there are downsides that the government needs to play a role in ameliorating as this technology gets rolled out. And it's just in contrast to all of the sort of US China military dialogue or lack thereof over the past 25 years where there's America that is really free freaked out about this thing. Basically, like if a crisis happens and we can't call you guys and say, hey, sorry, this was an accident, that like World War 3 is going to start. That is a thesis that the Chinese government has just not bought into to the frustration and worry of successive American presidents. And the fact that that hasn't happened and does not seem to be end up, we don't seem to be going down that path anytime soon does give me some more like broader optimism that there are things that these two countries can like talk about productively or at least just like work on in parallel productively, which is a nice thing.

Matt Sheehan

I had like the same sort of, if not hypothesis, like worry, you know, two, three, three, four years ago of like AI safety framed as Western plot to hold down China. It would be like a very clear mirror of the attitude they took towards climate for a while. Climate also is instructive. Even though they did maintain. I remember first time I got to China in 2008, like picking up some sort of state media, English language, state media paper and like the top headline was all about like climate change as Western plot to hold China down. That like it had a big impact. It was like one of the first things I read over there. So I was worried about that. But it's also true that like, you know, eventually their own scientists kind of like really came around to the fact that this is very real. It combined with like concerns about air pollution and you know, they, they are still relatively technocratic over there and they chose to like not just, not just start like working on, you know, emissions reduction technology, but actually see this like oh actually this is a huge opportunity and we're actually going to claim this entire green energy space. You know, you could say that AI safety doesn't have, you know, five industries built into it that China can dominate and leverage in the future. But I do think they fundamentally, they still do listen to science over there and the, you know, the, the way that they hear it, the way that they receive it is always going to be colored by, you know, their incentives. Maybe they're concerned, but they have other things that they're more concerned about like competition with the U.S. but it, you know, in many cases thus far, reason has, or reason or science has ultimately guided the direction of things. I mean people could point to Covid and say, you know, that was proved not true and that was a very intense period of a couple of years, but we'll see how it plays out.

Kevin Hsu

In AI I tend to be a bit more worried about how much here in the US we are going down path that look a lot more like what China is doing in ways that doesn't quite represent what we should do. Just, you know, based on our own first principles. Right. Like the whole fable saga, how GPT 5.6 was pre released to a select hand of customers approved by the US government just rings a little too close to home to the tech crackdown that happened in China a few years ago in ways that I spent a lot more time probably worrying about that scenario than anything else as far as like what is our system doing in a way that benefits our own company in the way that makes sense as opposed to being super reactive to a lot of this stuff. So you know, that's not really a happy thought to close on. But I think for me at least, I think when it comes to comparing us and China with what is the other side doing? What are we doing here? Are they going to do this, are they going to do that? There's some usefulness to that, but I think there's enough first principle thinking for us to do the thing in the right way for ourselves that doesn't weirdly resemble what China is doing when we're not learning from what they do well, but we seem to be learning from the stuff that they haven't done really well and copying that really quickly. You know, golden shares and a whole

Jordan Schneider

list of things I Think maybe that's why at the beginning of our conversation Glasswing didn't seem so foreign to Chi to Chinese. Because having this like close hold state directed, like centrally planned thing is having

Kevin Hsu

government being involved in private company matter is a very normal thing. Yeah, right.

Jordan Schneider

And then this sort of like direct CEO negotiation of like who's we're going to do what when. Yeah. I mean what do you. Let's, let's close a little bit on Sam Altman offering up 5% of the company in return for.

Matt Sheehan

I don't know, in return for

Kevin Hsu

protection

Jordan Schneider

I guess. I don't know. I mean Donald, Donald Jr. Clearly missed the boat on that anthropic IPO. Probably regretting that now a little bit. But yeah. Where do we go? Should we tax all these guys? Kevin, what's the public policy answer?

Kevin Hsu

I mean I think first of all backing up here, I don't think the US government getting involved in all matters of private industry is bad. Right. I think if anything we recognize that the government could and should spend time and money identifying areas where they could inject themselves strategically to boost a lot of domestic consumption capacity that's been really, really falling behind. Right. Whether it's real earth mining capacity, whether it's maybe some AI strategic initiatives and all sorts of things. As far as the form should it be taken as equity share owned by the government, I think they're good and bad arguments against whether that's the right way to do it. Right. Should the government be say for example stimulating demand for quantum technology that's been actually bubbling up quite a bit as a customer number one scenario. Right. To boost, to boost the industry. I think that's a very interesting way to look at government involvement in industry that doesn't just feel like almost a carbon copy of what some other system is doing, but to like think about the government's right role. But I do feel like this like OpenAI, but not just OpenAI, like if OpenAI does it all the other Frontier labs will have to offer the government a similar amount of equity. If this is how we go is just going to be this really bizarre new chapter in America so called capitalism. Right. Whereas this like American state capitalism with their own characteristics. I don't even know what this is going to look like but it's something that we should think really, really hard about as far as does this actually enrich the American people? Which I don't actually see how that works unless the shares the government actually owns gets divided by 350 million and we all get like a Deposit in our E Trade account or how does that actually benefit everyday people? Or is it just goes into the treasury but we don't really have any say or knowledge of how that money if appreciated, which it will, it's going to work out for the benefit of the broader public anyways.

Matt Sheehan

I think this is one where like the comparison US China comparison is actually instructive in that like at least China's sort of first draft of how are we going to respond to labor disruptions is to basically say like you company cannot just cannot fire people just because they've been replaced by AI. That's not a valid reason to put an end to a contract under labor law. And it's instructive that here in the US we revert to this kind of economic modeling clean solution in like okay, we're going to tax, take the shares of companies as something like a tax. We're going to tax, we're going to centralize, we're going to redistribute. Nice and efficient. It would be so, so inefficient for us to like get involved in the mechanics of companies. Whereas China is just much more willing to kind of do like the, not the dirty work, but they're willing to like grind through and say no, we're not going to like let these companies lay the people off and then tax them, take the money up and then redistribute it through some system, we're just going to be like look, you got to hold on to those people. You're making more money because you implemented AI, find something for them to do. And it's a much more of a kind of like muddle through approach as opposed to like a you know, UBI system that looks very clean on paper. But you know, whether or not people derive meaning in their lives, whether or not people feel okay about a UBI economy as opposed to a kind of like, what do we call them, make work jobs, you know, essentially being kept in jobs like a, you know, cradle to grave iron rice bowl. Yeah, you know, I'm really not the person to answer that question, but I think really is a manifestation of each sort of political culture in a way that we opt for this type of like clean redistribution and they are at least leaning into this much more like firm driven, state powered retention of people in companies muddle through approach.

Jordan Schneider

I mean I think the 5% is like a purely Trump phenomenon. Is like this is, this is, this is what he likes. He thinks it's cool signing these deals, then maybe he won't do like actual tax policy.

Matt Sheehan

It's also, I mean, it's Trump policy pipeline. You know, Bernie's saying we need 50% Trump. Like, yeah, sounds good.

Jordan Schneider

I don't know, it's. It's also weird that like every economist who's smart, who's thinking about this now works for one of these two companies. It's like, this is like another like, weird public policy thing is just like, I don't know, I've read the anthropic version of this, I've read the OpenAI version of these before proposals, and they are, in their defense, smarter than just like, okay, we're going to take 5% of our company and move on with it. But I don't know, this is a classic econ 101 Tyler Cowan reading merger revolution for 20 years thing is like, you tax what you want to disincentivize, right? So figure out a way to do that where people still have jobs. But you're not, you're not just kind of like walking away from it.

Matt Sheehan

I think the, the policy brain drain though, is just an important thing to like pause on like, you know, five years ago, all the way through today. But especially like five years ago, the big conversation was like, oh, academia, you know, academic AI ML people are being sucked into private companies. Eventually it's going to hollow out academia, it's all going to be private sector and that's gonna have all these bad effects. And like, I think we are seeing the policy version of, of that. Not claiming that, you know, my colleagues or you know, us geniuses at think tanks are like, so priceless that we, you know, we must be protected in some way. But companies are offering, you know, three, four times the salary of working for non governmental organizations or not private companies. And if we don't want to see all of the sort of most sophisticated policy oriented people working for the companies that are building the technology and making money from it, then you need a little bit of work there to keep people sort of in independent organizations.

Jordan Schneider

Yeah, donors got to step up. Come on. AI talent applies to those, applies to those Mass Ave think tanks as well.

Kevin Hsu

Yeah. Don't let Chinatalk get acquired.

Jordan Schneider

Everyone's got a price. All right, Kevin, Sorry, I didn't invest in Nebias. Okay. I regret mine.

Poetic Narrator

I regret.

Kevin Hsu

Microcosm of the larger problem now, aren't we? I mean, it's true, like these companies will end up writing the policies, right? Because those are the he. They. They have on their payroll all the people who've been thinking about the policies for this whole time. Well, you know, I'm just, I'm just riffing.

Jordan Schneider

No, no, no, no. It's, it's interesting because it's like, like

Kevin Hsu

we're getting not have the company's incentives be at least a. One of the primary top three concerns that you say when they think about the policies first they have to be maybe not the first one, but certainly one of the top three.

Jordan Schneider

Sure stole this from some Ezra Klein articles. Like, is like there's, there's like some, it's like a reverse horseshoe on where companies can really have impact. Where there's like the tiny problems that no one cares about. Where yes like you as a, as a citizen can like get your streetlight lamp fixed or something. And then there's this like messy middle that voters don't really care about that you know, there's some like public. There's like a, I don't know where an engaged company can like change some banking law that no one is actually going to stare about and then that gives you lot of power. But once we're on the other side of the horseshoe where all of a sudden it is big national stuff that is touching everyone's lives that voters are voting on. I think we end up in a world where the corporations actually have less to say and you know, like you can debate that and like how that played out in the, in the financial crisis. Right. But. And I think now AI isn't quite big enough where I think we're still in the middle where the, where the companies can really drive the discourse. But you know, fast forward 1, 2, 3 years, we may end up in a different place where you have more politicians who are just running on, you know, screw these guys. Here's what I'm going to do. I don't care what you think. And that is like something that legitimately wins votes such that maybe, maybe today is like peak impact of labs on the policy discourse around these topics.

Kevin Hsu

And I think the labs timelines, their policy papers also align with that projection which is they want to get all these things that they want to done done before 2028. Because I think the next presidential AI will be at the other end of that curve if not going to, we're going to have a preview of this, this midterm already. But it's a midterm at the end of the day. But it's going to be that it's issue anyway. Cool.

Jordan Schneider

Cool. Wait,

Poetic Narrator

You say you're a frontier model? Yeah, everyone's a frontier model tonight, sweetheart. There's A model in the building that can hack the world. So the hottest club in Beijing just got exclusive. Girl party owns the door, the floor, the lights, key code, the clipboard. And it's not polite. No lab picks, the guest list, no CEO plus one. The rope went up at midnight. Better get a permit, Hun. State grid. Come on in. Sinopec. Looking gorgeous. Ministry of Water Resources. Love the jacket. Everyone else, sidewalk. You're not on the list. You're not on the list. This is glass wing with Chinese characteristics, kid. You're not on the list. You're not on the list. Dress code is critical. Infrastructure, that's it. Ministries, dancing while the startups freeze from each's GPUs to each's API keys. Here comes Jack Ma crying at the door. Your whole government cloud is running on my cores. Bouncer taps the earpiece. Long pause. Fine, but you're a working coat check buddy. Get in line. Provincial gas utility just waltzed on through running Windows XP and a fax machine too. Cause the worse your legacy systems, honey, the better your seat. The most hackable girls get in the VIP suite. Provincial governments come back Friday. Local SOEs, also Friday. Hugging face. You got a calendar? You're not on the list. You're not on the list. It's commie glasswing, baby. And the party rules with an iron fist. You're not on the list. You're not on the list. Standing committee sent the invites, kid. Banks come first, while the founders just sit. Hey, chin up, kid. Everybody gets in eventually. That's the beauty of the place. Wide release, open to all three months, maybe four. Call it six. Six. Six, six. Six. Sidewalk. Sidewalk.