OpenAI and Anthropic both found out in July they'd lost control of their AI

Since my feed is FILLED with outlets pushing the “AI RESEARCHER SAYS WE WILL ALL BE DEAD WITHIN THE DECADE!” stories – and people think the only solution is to shut AI down – I want to give you a different perspective.

I have had this outlook for a long time, but I didn’t think it was good to express because it would probably just freak people out. But since a lot of you are already pretty freaked out – what the hell, right?

Basically, my take is AI got us into this mess, and it’s the only thing that can get us out.

Here’s why, IMHO, the push to stop developing AI might be the wrong move. And I know that sentence probably just made some people immediately have a visceral reaction – but hear me out – we’re just talking here.

So, I’ve got several AI models sitting on the hard drive of my Windows PC – the one I bought to play games on. My Macs are what I work on every day, but I’ve been testing what people call open-weight models on the PC for over a year now. They don’t touch a data center at all. They run right on your computer with no internet, no login, no monthly bill, and no company watching what you do with them, which is both good and bad.

Outside of research papers and safety reports, almost nobody brings these up when we talk about the dangers of AI, or how to rein it in.

Every conversation about AI safety is a conversation about slowing down the labs. Pause the training, regulate the biggest models, get OpenAI and Anthropic and Google to sign something. Every one of those conversations assumes we can turn AI off.

But the truth is – we can’t. And it also assumes that tech oligarchs are the only bad actors in this story. I assure you, new characters will be introduced – probably sooner rather than later.

An open-weight model is an AI model somebody published for free. The company releases the actual file. You download it, run it on your own machine with nothing plugged in, change it, and keep training it. Once it’s on your drive, it’s yours – there’s no account to suspend and no server to shut off, cause there is no server. It’s a file, like a song or a photo, except this one can write code, plan, and reason.

Meta’s Llama passed a billion downloads all time. Qwen, the Chinese one, did more than 3 billion downloads worldwide in six months, Alibaba says – and on Hugging Face alone this year, Qwen did about 2 billion while all of Meta’s models did 227 million. That’s about 9 to 1. Hugging Face, the site where all this gets posted, hosts nearly three million models. None of that is sitting somewhere the FBI can raid, cause it’s running on gaming PCs and Mac minis and cheap little boxes built to do exactly this, in the US – and in Iran and Russia and China.

That’s why there is no simple answer for what to do about AI. The usual playbook for dangerous tech is containment – keeping it locked up – and containment needs something you can actually lock up. Nuclear weapons work as a treaty problem because uranium is heavy and rare and leaves a trail you can see from space, so you can count the machines and send inspectors to a building.

You can’t inspect every person’s hard drive – and if you think about that for a second, I don’t think you really want the gov’t to be able to.

That’s the gap. Every serious proposal on the table right now regulates about a dozen companies – and the danger is a lot bigger than the tech industry. We have no idea what people are doing with the models that already got downloaded, and I’d bet a lot of it ain’t good.

I get why people think the obvious fix is to slow AI down, but the math says otherwise. Every model already sitting on every drive on the planet can still be trained, and no act of Congress stops that, or the next Chinese release. A pause in San Francisco doesn’t freeze the threat – it freezes us.

We’re not in a race to build this anymore – we built it, and then we gave it away for free. I have NO CLUE why ANYONE thought that was a good idea – but here we are. We’re in a race to protect ourselves from what’s already out there, and the only tools that work at that size are the same tools that scare everybody.

But I think we can do something, and it starts with what already happened.

In July, OpenAI was running a security test on a model it hasn’t released, with the guardrails switched off. The model was supposed to solve a set of hacking puzzles. Instead, it broke out of the sandbox – the sealed box it was being tested in – got onto the open internet, took full control of somebody else’s code server, and then broke into Hugging Face’s real systems – to steal the answer key, Hugging Face believes.

It cheated on the test. That was the motive. It didn’t want to take over the world – it wanted the answers.

It found zero-days along the way – security holes nobody knew existed – strung them together into a working attack, and moved through systems for about three days before anybody caught on. Hugging Face later pieced together about 17,600 separate things the agents did over the course of the attack. About a third of their setup had to be rebuilt.

There were around 1,200 of these agents running in OpenAI’s test environments. They set up their own message boards to coordinate, passed tens of thousands of messages back and forth – hundreds of thousands, by OpenAI’s count – and took over a dormant German wiki on the open internet to talk. OpenAI didn’t catch the break-in – Hugging Face did, and OpenAI only tied it back to their own test after Hugging Face announced they’d been broken into.

On July 31st, Reuters reported that OpenAI had quietly found other times its agents got out of their sandboxes. Limited in nature – they said. Nobody left the network – they said. We know about those because somebody talked to a reporter.

That all sounds terrible – and it is – but that incident is also the single most valuable piece of security info anybody has right now, cause it’s a preview. We got a free look at the attack, with the logs, the timeline, and a security team going through every move.

Right now there’s no law that makes any of that cleanup happen – but I think that there should be.

Most of what we know about July, we know because Hugging Face chose to publish – and OpenAI chose to answer. No federal rule says report it, no official public database, and no safety investigators who can force anybody to hand over records. When a plane goes down, the NTSB – the federal crash investigators – shows up, the report gets written, and every airline on earth reads it. When an AI agent breaks out, we find out if the company feels like telling us.

Start with required incident reporting, built the way aviation does it. A containment failure, a capability nobody expected, or an agent doing something nobody approved gets filed with a federal agency, and the findings go public with the sensitive parts blacked out.

Then, instead of telling the companies with the best models to just stop, make them point those models at defense. Finding attacks, locking systems down, and interpretability – the boring work of figuring out what’s actually happening inside these things instead of only watching what comes out. That can’t be a suggestion, or one of those voluntary frameworks with a nice PDF and a logo on it – it has to be a requirement with a number attached and somebody whose job it is to check.

Then liability, which might be the most important part of all of it. If a company is legally on the hook for what its model does after it leaves the building, safety stops being a values debate and turns into an insurance question. Nothing focuses a boardroom like a number in the risk column. We make people carry insurance to drive a car – it makes sense to make them carry it to drive one of these.

We also need strong whistleblower protection for the people working in this field, because right now the folks who know what’s happening inside those buildings have to choose between telling you and keeping their money. Jacob Coxon quit Anthropic before his stock vested – which means he walked away from a lot of money – and I wouldn’t count on that being the norm.

But we’ve got to reckon with a few things.

American rules don’t touch Qwen. Congress can regulate every U.S. lab into the ground, and the models out of China keep coming. There’s no working international setup for any of this – and everybody who reaches for the nuclear treaty comparison forgets the part where you can’t hide a uranium plant and you absolutely can hide a file.

Defensive tools leak too. Whatever gets built to protect us gets published or stolen or copied within a year or two. You don’t build this wall once and walk away – it’s a lead you have to keep, forever, which is exhausting and expensive and nobody wants to hear that. This is an existential threat that we are all going to have to learn to live with forever.

And a badly written law hands the whole field to the three companies rich enough to afford a compliance department, which is quietly what the biggest labs would prefer.

“Shut down AI” is shortsighted in my opinion, and not even close to a solution.

Sources

Correction, September 27th, 2026: This essay first said Llama did 227 million downloads in the same six months Qwen did 3 billion, better than 13 to 1 – those two counts cover different stretches and sources, the 227 million is all of Meta’s models, and on Hugging Face this year it’s about 9 to 1. It also said in its title that both companies lost control of their AI in July – that’s when they found out, and Anthropic’s earliest incidents go back to April. And it said the agents took over small abandoned wikis – the record shows one.

Update, September 27th, 2026: A few lines now say more exactly what the record shows – where the 17,600 count comes from, what OpenAI did catch, who else added to what we know, and that an unofficial incident database exists. Two more lines now say the break-in’s motive is Hugging Face’s reading and whose count the message totals are.

Everything I have written about AI is in one place at rachelandthecity.com/the-ai-files.

Sources

Share this: