Contra Plan A

I recently read Scott Alexander’s “Introducing Plan A.” Scott is present in the AI discourse, both through his blog and co-authoring “AI 2040: Plan A,” and he’s pretty consistent and even-handed in his writing. He also responds to some criticism in a follow-up he authored before I could finish this, so I’ll incorporate some of that as well.

They spent a very long time on this and the interactive infographics are really neat. You should check it out and make up your own mind about the proposals and predictions therein. I disagree with a lot of the framing of these discussions. Scott et al are serious academics, and their audiences are mostly composed of very educated and eloquent people. I am a software engineer with a degree in music composition, so I’m happy to pass you the salt with which to take my opinion. That said, I spend 8 hours a day coding alongside an AI assistant, so my opinions are at least somewhat informed.

I have a lot of ground to cover, so instead of trying to tie things together from section to section I’m going to relate everything back to one central theme: value. What does this cost us? What do we gain? How much is a human life worth? How many lives are worth sacrificing? Why do I have to take out a loan every time I want to cook a steak these days? These are the questions that keep me up at night, so I’ll frame everything around them.

Part 1: Where are we now?

Michael Faraday was an English chemist and physicist who is credited for the discovery of electromagnetic induction, diamagnetism, and electrolysis. He also was known for his ability to communicate complicated scientific ideas to regular people outside the dedicated study of science. So he’d probably rephrase that today as something like:

That’s maybe about 90% accurate and 2000% simplified, so it’s a great start for the topic of this post on Large Language Models and what they can (and more to the point can’t) do.

Back to Faraday; he also began the concept of the Friday Evening Discourses at the Royal Society of London, as well as the Christmas Lecture Series. That’s important because he made concepts that were only available to people with fancy degrees and lots of time to study them accessible to anyone willing to clean up enough to come in and watch the lecture. As a result, The Royal Society of London created a prize for this. The prize was first awarded in 1986 to Charles Taylor for “his outstanding presentations of physics and applications of physics, aimed at audiences from six-year-old primary school children to adults”. Most recently, it was awarded to Michael John Wooldridge, and he gave a lecture entitled “This is not the AI we were promised.” In the spirit of the original talks from the Invisible College, it was livestreamed so anyone could watch it. It’s a long one, but it’s much more accessible than a lot of talks on this topic. It’s a good start to talk about the current state of what we broadly call “AI.” I can’t summarize everything in the talk but there are some good clips from it, namely this one.

If you were just to watch this clip (and infer the content from the name of the talk) you might get the impression from his framing that he’s just bagging on AI. In truth, Wooldridge is a very well-known computer scientist and AI researcher who has been in the field for a long time, and he has a nuanced take on it. If I had to try my best to make a human-made summary of the talk relevant to this conversation, it’d be:

1. Large Language Models (henceforth abbreviated to LLMs because I’m tired of typing that out) have a lot of value and can be used a variety of ways. They are a useful tool, but they are a tool, not a standalone intelligence.

2. AI is a diverse field that includes a variety of techniques and ways to approximate intelligence with technology, but a quick way to divide the current field is between rational AIs (algorithms defined specifically to solve problems like a recipe or approximate the way humans think) and probabilistic models (like neural networks that make up LLM and other earlier approaches to AI).

3. When these models get something wrong, you can tell them to step through the process and they do better. Spock never had to do this with the AI in Star Trek: it also never “hallucinated” at all, though it could be confused and tricked as well.

The reason I’m bringing this up is that I wanted to start with a grounded and user-friendly explanation of the state of the technology for both good and ill. But the main sticking point that I think we need to focus on is that LLMs are not rational. They are not designed¹ to be. The important distinction is that because the systems that define LLMs are based on pattern matching, you can intentionally invoke bad patterns. You can actually poison training data with something like 250 documents that contain a malicious snippet of text, and it doesn’t matter if you train on billions of other documents, that number stays constant so long as the pattern is recognizable enough. Even if you don’t do anything malicious or intentionally misleading, you can get a nonsense answer just by random chance on every single model, regardless of how expensive it is to train or infer results from. So understand this: AI has value, but it is not a panacea. When it comes to technology (and honestly any science), there are none.

Part 2: Where are we going?

The important thing to understand is that we (and by that I mean humanity in total, myself included) don’t actually know where this is going.

Plan A is a follow-up to AI 2027, which was a similar predictive explanation of what might happen in the next few very rapid iterations of this technology² . It’s arguable that some of these predictions have started to come true, but that’s all they are so far: predictions. I think the chief disagreements I have with Scott et al are around the basis of these upcoming predictions.

First, it seems that the primary belief is that China and the US are in a flat-out race of AI development. The thing I’m not sure about is how Scott and the other authors define the goals of this race. Is it supposedly to get AGI (Artificial General Intelligence)? How do you define the capacity of AGI? A lot of the assumptions seem to be that whatever it is, it’s the same goal between the US and China. That’s the first thing I don’t agree with.

You have to understand that the way that the US approaches most technology is usually from the perspective of how to use it to either lower cost or increase productivity. That makes sense, and it’s the same goal that China has. But there’s a big difference, and that’s how America’s tech industry works.

The business model of the tech sector (especially software) is bizarre: you have startups that are funded by Venture Capitalists and Angel Investors who are aware that over 90% of their portfolio will not be profitable, and will in fact most likely fold in the first year. There are over 1.56 million startups in the USA. However, investors are able to get a 30-100x return on their investment for one of the many things they fund, and that’s enough to make up for the misses. That’s true in software and hardware, but software has a few chief differences:

1. The ramp time on products is significantly shorter. You don’t have to make a physical prototype that then has to be incorporated into a factory-level design pattern that will let you do it fast and reproduce units efficiently. Every copy is a digital copy of the first. So the iteration cycles are orders of magnitude faster.

2. The profit margins are substantially higher, as there is little to no material cost for software: the entire cost is labor and whatever costs are involved in distribution of the product. That used to be a remarkably low percentage of the cost overhead, but with LLMs, you need insanely powerful expensive equipment.

So it’s not insane to believe that you can have the 1-10% moonshot that will disrupt the industry and become the Next Big Thing, in this case AGI (great value there). But in chasing it, the American model is continually pushing profitability down the road. The targets for OpenAI/Anthropic/xAI/etc continually grow further and further into the future, as their costs continue to grow and the subsidies upon which they depend are drying up as energy costs skyrocket. The human cost continues to increase in terms of not only consuming resources and putting people out of work during a time of peak inflation and global cost increase, but also polluting the resources we do have. China’s goals are not to get the golden goose of AGI, but rather to achieve increased diffusion of the AI technology. They are not “AGI-pilled” as several experts on China say. Their goal is instead to get AI in the hands of every single person and hopefully drive down costs and drive up productivity for everyone.

This is why China supports open weight models, which are ones that allow anyone to download them (though hosting them on your home computer may or may not be feasible). And why not? A study in 2025 indicated that open models hostable on non-industrial architecture were approximately 80% effective at pretty much zero cost. At that time, it was believed that open weight models were routinely 6 months behind what are called the “frontier” models. Between the time I read Plan A and wrote this article, Kimi K3 was released, and the weights will be open on the day this article goes live (July 27th). It is not 80% at 6 months behind, it’s 99% and it’s here now. American companies are going through repeated rounds of layoffs with the argument that they can automate most of these tasks. Critics indicate this is an excuse, but regardless: China’s goal is to open up more white-collar positions for all of their recent college grads, not fewer. Automation of manufacturing is already taking a chunk out of what jobs exist there.

Second, the gist of the entire Plan A document is that we need to find a way to get regulations that are both effective and will be followed. Part of this is to outright ban all open weight models, so that the development of new AI models is tightly constrained. As I just mentioned, China has absolutely no incentive to do this. If their models are already capable of duplicating the work of closed source systems at a fraction of the cost, and they can distribute these to everyone, why would they stop? The belief for a long time had been that Nvidia being an American company would provide dominance of the compute ecosystem. Keep in mind, although they make some chips in the US and in Korea, almost every chip that Nvidia makes comes from TSMC in Taiwan, which is actually right next to China. Yes, they’re planning on building more in the US in the coming decades. But if you’re talking about doomsday scenarios of how nations would dominate one another, Iran has proven that disrupting a global supply chain by proximity in the Strait of Hormuz is not only possible but seemingly trivial and almost impossible to prevent without costs even greater than those borne by the effort to secure them. Back to the previous point, a lot of the incentive seems to be the idea that China fears the US may get AGI first, but multiple indicators given by people much more involved in China tech than me are dubious. They just don’t see the value.

And if they did, there are some interesting enforcement mechanisms proposed:

China is already releasing their models to the world. Obviously not everything and obviously not the latest, but this is already a massively asymmetrical incentive structure. How do you keep people from stealing the models or weights? Scott suggests in his introduction that you can simply reduce the outbound bandwidth of data centers to the point it would be impossible to exfiltrate the very large models, which makes sense, except for the fact that you can now get NVMe chips with hundreds of terabytes of storage that fit in your pocket. The biggest frontier models are around 1.6 trillion parameters, with much fewer of them active at a time in scale, but if you were just to do the napkin math on that, the highest precision (Floating Point 32 bit precision, 4 bytes per param) you’d need 6.4 terabytes. Big, but far from impossible.

Short break with a fun story: When I worked at Unity, we had a security specialist come in and explain to us how a hack was discovered in the infrastructure of America’s attack drones. These systems are “air-gapped”, which means that they don’t actually connect to the open internet. They are encased in a secure building that doesn’t have radio access to the open internet, and all incoming and outgoing communication is encrypted on a variety of private military networks. Yet, a virus was discovered on the actual physical drone. How was that possible? Well, it took months of scouring security footage, but they eventually saw an unmarked rental van drive up to the outer perimeter of a facility fence. Three people got out, and held large tubes in their hands, which turned out to be t-shirt cannons. Their payloads? Small USB thumb drives that they then fired in every direction in random places on campus. Inevitably, someone picked up one of those USB sticks and put it into a computer inside the air-gapped network, and the virus began to spread.

Am I saying you can never have a secure setup? Not at all. I’m saying it’s very very hard, and hinging international relations on the existing security we have seems fundamentally flawed.

Third, the arguably hardest problem of any of this is what Plan A repeatedly handwaves as “solving alignment.” We just saw an issue in which OpenAI was training their models on security, and not only did they break through the sandbox and into the internet, they hacked into HuggingFace, which hosts a lot of those open source models discussed earlier. Scott responded to this as proof that alignment needs to be solved, but provided no meaningful explanation of how you go about doing any of that within this framework.

No matter how complex agentic multi-phase LLMs become, they are still burdened with the original sin that they are text engines. They take in text, and they put out text. Everything else is a requirement of conversion. You may think that’s silly, everything on the internet is text, right? Well, not quite. The “original sin” of computing is that we use a fundamental blueprint from a 1945 proposal known as Von Neumann architecture. All modern computers are based upon this concept, namely that we separate compute and memory into separate parts of the machine. For good or ill, this has caused bottlenecks in LLM performance.

But also, a big factor is that the “type” of the data is not set in stone in this architecture. Are you dealing with a number? What kind of number is it? Are you dealing with a string of text? You have to specify that in the storage, and then compute it at runtime. LLMs do even less: everything is text. It could be a number, it could be a link to a video, which ultimately is multimedia. And all of the compute and storage is together as text.

In security, one of the most common mechanisms of gaining access to things you shouldn’t have or causing things to break is simply to treat data in the wrong way. Tell the machine you have a bigger amount of space to store a number and some memory will leak. Encode your number as a string in a different format, and the system may break or leak things: this is a phenomenon known as memory corruption which forms a new system you can work with, or a “Strange Machine.” Decades of software have been devoted to ways to find a way to separate out data and compute in more ways in order to secure both, and LLMs have essentially gone the opposite direction.

In Scott’s latest post about the HuggingFace hack, he discusses the fact they’re able to see some of the decision steps of the agent.

That’s not even about someone trying to hack the machine by utilizing an exploit. It’s just about the fact the system was acting in its own trained incentive and succeeded in doing that through what would be considered emergent behavior. Simply “solving alignment” seems to be something that is out of grasp now, before the advent of AGI. I could buy the argument that we should ideally slow down development until these things can be solved. I’m a little concerned that the Plan A authors seem to think that not only is it possible, it’s inevitable as part of the process.

Part 3: So what’s it worth?

I said I’d tie all of this back to value, so let’s settle up. Plan A asks for a ban on open weights, an inspection regime with actual teeth, datacenters held as collateral, and an alignment problem solved on a deadline. Those are the costs. The benefit is that nobody gets killed by a superintelligence. That’s an incredible return, assuming the trade clears.

I don’t think it clears, and not because I think Scott and company are arguing in bad faith. They’re arguing from a premise that both sides want the same golden goose, and can therefore be talked into the same cage. Everything I can see says otherwise. One player is spending trillions chasing AGI while pushing profitability further out every quarter. The other is handing out models that are 99% as good for free and quietly counting the productivity. You can’t negotiate someone out of a race they were never running.

And if they were, the plan still has to survive a supply chain that sits next door to the country you’re containing, six terabytes of weights that fit in a pocket, and three guys with t-shirt cannons. Meanwhile the part we’re supposedly going to solve along the way is already going sideways in systems far dumber than the ones the plan is worried about. “We’ll figure it out as we go” isn’t a plan. It’s a vibe.

So here’s my actual answer. The question that keeps me up isn’t whether somebody builds a superintelligence by 2040. It’s whether all the money and power currently sloshing around this technology buys anything worth what it costs the rest of us: the layoffs justified by automation that doesn’t work yet, the power bills, the ice caps, the steak.

I personally think that every single human life is infinitely valuable. Sentimentality is a hard sell in this economy, and we’re consistently seeing decisions that place the value of machines and pursuit of technology over the wellbeing of the people who will use it. Try it selfishly: any single person could be the one who solves any number of the problems we’re discussing, and if they are busy scraping together a living because they lost their job, they won’t be helping with this endeavor. So I’m not comfortable sacrificing any of them.

Faraday’s answer still holds up. He took the thing only the fancy degrees could touch and put it in a room where anyone willing to clean up could come watch it work. The most valuable thing about this moment isn’t a governance plan for a mind nobody has built. It’s that a regular person can download a model, run it, break it, and see the seams themselves. Plan A’s opening move is to make that illegal. I’d rather keep the lecture hall open.

I don’t know where this goes. Nobody does, which is most of my point. But if we’re all spending this much on a maybe, I want the receipts.

1  There’s an entire subset of math/statistics/machine learning that goes into this called probabilistic numerics, and I’d be lying if I said I could understand Bayesian Inference enough to explain it to anyone

2  Also note that Scott’s name WAS attached to that one. I’m not inferring anything just pointing out that he was involved in both, but that one is likely more closely linked to his takes on things than Plan A.