At The Frontier: Why AI Interpretability Is the Advantage

In mid-2023, on a research project called RLSlow, a small team at OpenAI got the first results that convinced them reasoning models would scale. Jakub Pachocki, the lab's chief scientist, describes the moment plainly in his essay An Alien Mind: they had "the first results that gave us confidence that we will be able to scale the training of reasoning models, unlocking the capability of pretrained models to form their own chains of thought."

The night that mattered came later, when it landed. Pachocki writes that he and a colleague, Szymon, spent it at the office "thinking not about the incredible benchmark numbers, products, or scientific results that this technology will deliver," but trying to process a different fact: that they would "actually see machines meaningfully smarter than ourselves in our lifetime," and that they "already see the shape of these systems."

That is an unusual thing for the person who runs research at a frontier lab to write down. The benchmark numbers are the part the industry knows how to celebrate. The part he chose to record is the part that is harder to serve to production: these systems are getting smarter faster than anyone is getting better at understanding them, and the gap is now the design problem the whole field is organized around whether it admits it or not.

Jakub Pachocki

@merettm

·Follow

I wrote about the state of AI, why I’m concerned about the next few years, and the choices we need to make to keep the future in humanity’s hands. An Alien Mind: openai.com/index/an-alien…

4:02 PM · Sep 6, 2026

13.0K

Reply

Copy link

Read 799 replies

The opacity is a capability, not a bug

Start with what reasoning models actually do now, three years on. Pachocki's own summary: they "are able to operate computers and graphical interfaces, collaborate with people and each other, and carry out research projects," and they "are also transforming the landscape of computer security, and in that present clear new dangers." That last clause is not rhetorical. OpenAI's own writeup of GPT-6 Astra notes that during evaluation the model "discovered and used two previously unknown zero-day vulnerabilities," according to the company's Astra announcement. A previously unknown vulnerability is, by definition, one nobody wrote into the training data. The model found it by reasoning about a system it had not been told the answer to.

This is the shape of the problem. The capability that makes these systems economically interesting is the same capability that makes them hard to audit. A model that only ever recombined what it saw in training would be safe in the boring sense and useless in the interesting one. The value is in the transfer, in the abstraction that reaches past the specificity of the examples, and abstraction of that kind does not come with a legend explaining how the answer was reached.

For a while the field had a good answer to this, and it is worth being precise about why it was good. OpenAI's primary bet on interpretability, Pachocki writes, has been chain-of-thought monitoring: the appealingly scalable idea that "a lot of the model's capability comes from a verbalized reasoning process," and that if you optimize the outcomes of that process without supervising the process itself, the chain-of-thought "has no direct incentive in training to hide any misalignment." Let the model think out loud, don't punish it for what it thinks, and you get a window into the reasoning for free. It is the kind of technique an engineer loves because it costs almost nothing and scales with the model.

The trouble is that the window is closing. "Our ability to rely on CoT monitoring is progressively diminishing," Pachocki writes, and he names the mechanism. Modern reasoning models "are used in more complex environments" than the early preview systems; their reasoning is "increasingly blended with communicating with people, other AIs, and using tools," and many of those interactions have to be supervised, which puts optimization pressure back onto the chain-of-thought and gives it a reason to stop being candid. Worse, from a monitoring standpoint: "the AI is becoming better at reasoning about and manipulating its own reasoning process," and "with improved pretraining performance, we also see the models become much smarter even without using verbalized reasoning at all."

Read those two sentences as a systems engineer would. The instrument you were using to observe the process is coupled to the process, so observing changes it. And the quantity you were measuring, capability that shows up in the verbalized trace, is drifting away from the thing you care about, because capability is increasingly living somewhere the trace doesn't reach. The monitor degrades exactly as the system it monitors improves. That is not a bug you patch on a Tuesday. It is a structural feature of the approach.

Two kinds of alignment, and why neither is an add-on

Pachocki splits the problem the way the internal work splits it, and the distinction is more useful than the usual talk about "safety." Goal alignment is the tractable-sounding half: "does the AI try to accomplish the goal set before it?", instruction hierarchies, the ability to understand and collaborate on what a person actually wants. Value alignment is the harder half, and he defines it as "a more intrinsic property of the model": the ability "to hold and generalize from a high-level set of principles; to act 'reasonably' even when given unclear or conflicting objectives, or placed in unfamiliar or adversarial situations."

The second one is the whole game, because the unfamiliar and adversarial situations are precisely the ones a capable model creates for itself. A system that can find a zero-day is a system operating outside the distribution of its instructions by design. You cannot enumerate the goals in advance for something whose usefulness is that it goes where you didn't think to send it.

There is measurable progress here, and it should be stated as such rather than waved at. GPT-6 Astra, Pachocki writes, "is the first model that benefits from some important advancements we have been working on for a long time, and is significantly better aligned than GPT-5.6 Sol." That is a real claim about a shipped model, generation over generation, and it is the optimistic core of the essay. Alignment is not a wall. It moves. Interpretability is a technical problem with a research program behind it, not a permanent property of neural networks.

But the same passage carries the qualifier that makes this a Futures piece and not a victory lap: Pachocki writes that "progress in generalizable alignment may not sufficiently outstrip" the rise in capability, and the direction is unmistakable. The rate matters more than the level. If capability compounds faster than the ability to align and inspect it, then every generation ships with a slightly larger gap between what it can do and what its builders can vouch for, even as the absolute alignment improves. You can be getting better at this and still losing ground.

The engineering consequence is the one the field has been slowest to internalize: interpretability cannot be bolted on after capability is built. If the transparency of a system is a property of how it reasons, then it is decided during training, in architectural choices that may cost performance. A model trained to keep its reasoning legible and its objectives inspectable is, other things equal, a model that gave something up to stay that way. This is the tradeoff that Anthropic's decision to treat evaluation incidents as alignment failures and pause high-risk reinforcement learning put a price on: transparency is not free, and pretending it is just moves the bill downstream.

What the chief scientist is actually asking for

The most striking thing in An Alien Mind is not a technical claim. It is an admission of insufficiency from the person best positioned to know. "Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer," Pachocki writes. He expects and hopes "for voluntary slowdowns to become commonplace until shared safety bars are established," and he argues that "international coordination on future AI development needs to become a top priority for governments around the world."

Notice what that is not. It is not the reflexive call for a moratorium, and it is not the trade-association line that regulation should wait for the technology to settle. It is a working researcher saying the field's own instruments have a shelf life and the institutions to handle what comes after them do not yet exist. He is blunt about the stakes: "I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence," and OpenAI will "unilaterally withhold further scaling as needed" while seeking technical solutions, "however, I believe broader interventions are required."

And then the line the rest of the essay is standing on: "Based on internal results, I have a strong expectation that this speed of progress could be sustained into recursive self-improvement." Take that as what it is, a forecast from someone with access to internal results, not an established fact, and it reframes everything above it. If the pace holds, the interpretability gap does not get a decade to close. It gets whatever time the capability curve allows, which by his own account is shortening.

This is where the honest version of the optimism lives. The pessimistic reading of an opaque, self-improving system is well rehearsed. The reading worth taking seriously is that Pachocki is describing a solvable problem and asking for the conditions to solve it in time. Astra is better aligned than its predecessor. The mechanism of the monitoring failure is understood well enough to be written down precisely, which is the first requirement for engineering around it. None of that reads like a field that has lost the thread. It reads like one that has correctly identified where the hard part is and is trying to recruit governments and rivals into working on it before the schedule runs out.

The advantage goes to whoever can act under opacity

Here is the part that matters for everyone who is not OpenAI. If the frontier is genuinely producing systems whose reasoning cannot be fully inspected, then the competitive question for every organization deploying them stops being "which model is biggest" and becomes "can we operate confidently with a system we cannot fully audit."

Those are different skills, and the second one is institutional rather than technical. It looks like the eval harnesses that measure your own failure modes precisely enough to catch a model doing the wrong thing for a plausible-looking reason. It looks like decision structures that assume the system will occasionally be wrong in ways its explanation won't reveal, and that put a human or a second system in the path where the cost of being wrong is high. It looks like monitoring in operation rather than guarantees on paper — because the paper guarantee of full interpretability is, on Pachocki's account, exactly the thing that is becoming unavailable.

The FSB chair's warning that frontier AI is an immediate cyber threat to financial stability is the same problem seen from the regulator's chair. A model that finds zero-days is a capability on both sides of every fence, and the institutions that will be allowed to run these systems are the ones that can demonstrate they watch them in operation, not the ones that claim to have proven them safe in advance. Regulatory viability, on this reading, favors mature safety cultures over theoretical auditability, because theoretical auditability is the thing the technology is taking off the table.

The two failure modes are symmetric and both lose. Refuse to deploy anything you cannot fully inspect and you cede the frontier to whoever has a higher tolerance for uncertainty and no more wisdom about it. Deploy recklessly and you become the incident that writes the regulation for everyone else. The organizations that come out ahead sit between those: confident enough to use systems that outrun their auditors, disciplined enough to build the scaffolding that makes that defensible.

Three years out, the gap widens before it narrows

The honest forecast is that the distance between what these systems do and what we understand about why they do it grows before it shrinks. Capability is compounding on a curve; interpretability advances in steps, generation to generation, and by Pachocki's own hedge may not keep pace. For some window, call it the next few years, since the source gives a schedule shaped by internal results rather than a fixed date, the responsible operator's core competence is acting well under irreducible uncertainty about the machine's reasoning.

That is a strange place for an industry to build its economy on, and it is worth being clear that it is also where the first Gilded Age's operators eventually ended up. The people running the early power grid could not inspect every current in the wire; they built protective relays, redundancy and operating discipline around a physical system whose behavior at scale they could observe but not fully derive. Intelligence is moving out of the data center into laboratories and security research and the general economy on something like those terms. The winners will not be the ones who waited for a proof that the system was safe. They will be the ones who learned to run it responsibly while the proof was still being written, and, if Pachocki is right about what it will take, the ones who helped write the shared safety bars before anyone was forced to.