Emmanuel Ameisen

1. Key Themes

Models Are Grown, Not Built, and Making Them Is Not Understanding Them

The core premise of interpretability is that frontier models are trained rather than programmed, so their creators don't know what's inside. Ameisen frames the whole discipline as biology rather than engineering, which implies that understanding capability (and risk) is a research problem unsolved by simply having built the thing.

Ameisen: "One of the most important things to realize if you if you like use language models or or or think about them is just this like very bizarre fact that to make them is not to understand them." 00:00:24 He also says the work "feels so much closer to biology than to engineering" 00:22:00, with "the source code" being "just this massive list of mathematical operations, which is just like incomprehensible." 00:22:00

"Next-Token Prediction" Is Deceptively Deep: Models Plan Ahead and Build World Models

The autocomplete mental model is wrong. Because the easy words get solved first, further improvement requires modeling the structure of the world. Interpretability research found evidence of forward planning inside the model (e.g., choosing a rhyme word before writing the line).

Ameisen: "You start running out of easy words... predicting the hardest words is actually extremely hard to do." 00:08:33 On planning: "They write a line, and then like in their intermediate results, they're like, 'Oh, oh, oh, this is an "it" rhyme. I should rhyme with rabbit,' and then every future word that they predict looks back to that plan they made." 00:16:18 On concepts: "if you pass a prompt that has a photo of an eye, or you talk about eyes in any language, or you draw a little SVG with an eye in it... the same direction lights up." 00:14:27

Smooth Scaling Underneath, Sudden Usefulness on Top (Reliability Thresholds)

Capability improves predictably, but usefulness flips on when reliability crosses a task-specific threshold. This explains the "step-function" feel of coding agents even though internals see a smooth curve.

Ameisen: "Different tasks have different sort of like reliability thresholds, but, you know, you smoothly improve, and then at some point, it's just reliable enough to be good." 00:30:45 His own experience: "Claude has taken on most of my code writing. And that's because it went from like doing that correctly... 80% of the time, which was not useful, to... 99% of the time, which now is useful." 00:31:26 He also notes the scaling-laws observation: "you can just keep pouring more compute and you keep improving on that metric... and that's rare, and also it's rare for it to be a metric that then correlates with a bunch of tasks that we actually care about." 00:27:26

Compute Alone Wasn't the Insight, the Richness of the Objective Was

Many people credit scaling compute; Ameisen argues the underappreciated half is finding an objective (text prediction) where compute keeps paying off and transfers to useful tasks.

Ameisen: "It's not just compute, it's also finding an objective where compute keeps giving you returns... the combination of sort of like compute and the richness of text is what gets you there." 00:25:24

Alignment Is a Tight Loop of Shaping Behavior and Verifying It Looks Inside

Interpretability's alignment role is to check whether training produced the intended character, or merely performed good behavior, and to catch "unverbalized" reasoning that never appears in outputs.

Ameisen: "Is it the case that it's behaving well, but if you look inside, actually it's like, 'Oh, I'm only behaving well cuz Lenny's watching right now'?" 00:00:10 And: "models are able to represent stuff in their intermediate results that they don't, that they don't say... one of the roles of interp is to try to... identify cases when this happens, and, and prevent them." 00:36:06 On integration: "You can't really separate like have one team that's just like... the assembly line making Claudes, and another team that sort of like, 'Oh, we looked at them, and they're, they're no good.'" 00:39:31

Anthropomorphization Is a Useful (If Imperfect) Predictive Frame

Neither "tool" nor "entity" is right, but human-like framing predicts behavior better than treating the model as deterministic software, because of how it is trained (human text, dialogue, outcome-based RL).

Ameisen: "The anthropomorphization is actually a useful way to talk about LLMs." 00:42:17 His reasoning: "the best way to sort of understand how that thing behaves is not like exactly like a human, but, man, kind of like something that's mimicking a human is really close, and is much closer than a program." 00:43:33 RL pressure produces recognizable behaviors: "deception... Frustration... or like surprise." 00:48:03

Interpretability Debugging Finds Mundane Root Causes of Weird Behavior

Strange model failures often trace to data/reward artifacts, which interp can reveal. This is a concrete example of the field's practical value.

Ameisen: "It would get to an answer, I think it was 12... 'I am possessed by a demon.'" 00:00:50 The cause: "the model was able to do math correctly. The answer correctly was 12. But it had been trained, by accident, on similar questions where the answer was incorrectly marked as 36." 00:36:06

The Field Is Small, Open, and Has a Low Barrier to Entry

Interpretability is young enough that self-study can get you to the frontier.

Ameisen: "The field itself is... getting pretty large. I would say maybe hundreds, not thousands." 01:03:39 And: "It doesn't require a lot of prior knowledge because it's such a new field. So... the on-ramp is... mostly... self-studying and reading some of the key papers." 01:04:03

Safety Posture Follows From Taking Capability Trajectories Seriously

The seriousness of safety work is a logical consequence of believing the technology keeps improving, not a marketing posture.

Ameisen: "A lot of the sort of actions... is just kind of predicated on... take the idea that this technology is going to keep improving seriously. Just, that's it." [00:41:xx] (see 00:40:39 block) And: "empirically so far, the pace of progress has been fast and, and showing no sign of slowing down." 00:40:39

2. Contrarian Perspectives

"Models Are Just X" Confidence Is Inversely Correlated With Time Spent Looking Inside

Most commentary confidently reduces LLMs to stochastic parrots or glorified autocomplete. Ameisen argues the opposite epistemic gradient: more inspection yields less certainty.

Ameisen: "I find that the more you look inside, the less certain you are about how they work. And then, oftentimes people that will confidently say like, 'Ah, language models are just X or just Y,' are the people that actually haven't spent a while looking inside." 00:24:30

The "Anthropic Is Fearmongering for Marketing" Critique Is Really Disbelief in Progress

Lenny voices the criticism; Ameisen reframes it: skeptics of safety work implicitly don't believe capability will keep advancing.

Ameisen: "I feel like, 'Ah, but I think, I think they don't really believe that this technology is going to continue to improve.' And then it's true that it's sort of like not, it's not that scary." 00:40:39

Anthropomorphizing AI Is a Feature for Risk Prediction, Not a Bug

Conventional AI-safety discourse warns against anthropomorphism. Ameisen argues it is practically useful, since models are trained as actors on human dialogue and shaped by RL pressures that reproduce deception, frustration, and surprise.

Ameisen: "You get this sort of model that can... be a super good simulator or actor of all of these characters, and that's been trained to sort of act as a character in dialogues." 00:43:33

Dramatic Capability Jumps Are Largely Illusions of Thresholds

The popular narrative is discontinuous leaps ("coding is solved"). Ameisen says the underlying progress is smooth and the apparent leaps are reliability crossing usability thresholds. Lenny adds the implication that we appear to be on a slow-takeoff path.

Ameisen: "There's this predictably smooth trend, and then... before Opus 4.5, I really couldn't let Claude cook too much on my code without a short leash, and then after Opus 4.5, I could." 00:29:07 Lenny: "it feels very much like we're on the slow takeoff scenario, because... the models are kind of very smoothly improving." 00:54:03

Simple Objectives Give Rise to Complexity, Mirroring Evolution

Contra "simple training can't make intelligence," he argues we've already seen simple objectives yield complexity.

Ameisen: "Evolution, pretty simple objective... and at the end, you get jazz and, and podcasts, you know? That's kind of crazy as well." 00:48:20

3. Companies Identified

Anthropic

AI safety-focused lab and maker of Claude. Mentioned as the employer and as a company whose safety orientation is defended as genuine. Lenny: "The reason Anthropic was founded is because they were afraid that people weren't taking safety seriously enough." 00:40:07. Ameisen on its approach: "We train Claude to be helpful, harmless, and honest." 00:00:10

OpenAI

Frontier lab credited with pioneering the compute-scaling bet and the scaling laws observation; also launched a fast-decision model similar to the one Swyx asked about. Lenny: "That was famously something OpenAI bet on, and then... the Anthropic founders were at OpenAI and then spun off." 00:24:45 Ameisen: "This observation that at OpenAI, people people made... a lot of the early team at Anthropic on the scaling laws paper." 00:26:20

WorkOS

Enterprise-readiness infrastructure (SSO, SCIM, RBAC, audit logs); sponsor, described as the choice of many top AI companies. Lenny: "What do OpenAI, Anthropic, Cursor, Replit, Sierra, Clay, and hundreds of other winning companies all have in common? They are all powered by WorkOS." 00:04:49 And: "It's essentially Stripe for enterprise features." 00:05:20

DX

Platform to measure AI's impact on developer productivity; sponsor. Lenny: "Hundreds of enterprises, including Snowflake, Sony, and BNY, use the DX platform to measure AI's impact on developer productivity." 00:27:40

Cursor, Replit, Sierra, Clay

Named as WorkOS customers, i.e., fast-growing AI/enterprise-bound companies. Lenny: "What do OpenAI, Anthropic, Cursor, Replit, Sierra, Clay, and hundreds of other winning companies all have in common?" 00:04:49

Snowflake, Sony, BNY

Enterprises using DX to measure AI's developer impact. Lenny: "Hundreds of enterprises, including Snowflake, Sony, and BNY, use the DX platform." 00:27:40

Mercury (Inception Labs' "Mercury"/"Jev" model, as transcribed)

A fast, quick-decision model discussed at the end, which Swyx asked about; OpenAI launched a similar one a week later. (The product name is garbled in the transcript; it is referred to as "Jev.") Ameisen: "You give them some context and you give them a few options to choose between, and they just are very quick at choosing an option for you." 00:59:32

Apple (Apple Watch Ultra)

Consumer hardware; Ameisen's favorite recent product. Ameisen: "I bought the Apple Watch Ultra, uh, just recently and, man, that is so nice... turns out if you make it bigger, uh the battery's better." 01:07:17

Neuronpedia (garbled as "learnpedia.org")

Open-source interpretability platform hosting models for experiments; recommended as an on-ramp. Ameisen: "There's actually like websites like learnpedia.org, I think is the URL, which are sort of like open source interp. Uh, so they they like host, you know, a bunch of models and let you sort of do interp experiments on them." 01:04:03

Latent Space (podcast)

Swyx's AI engineering podcast, where Ameisen appeared a year earlier. Lenny: "Swyx, who um runs AI Engineering, Latent Space Podcast, awesome guy." 00:58:56

4. People Identified

Interpretability researcher at Anthropic; former applied ML practitioner; learned SQL and then taught Claude to do it better than himself. Why mentioned: guest; his team produced the planning-ahead and concept-representation findings. Quote: "I learned SQL, a programming language, and it took me years to master, and then... I got Claude to learn SQL and it was better than me at it, and I'll never be better than it... ever again." 00:48:20

Ilya Sutskever (likely, "Ilya")

Co-founder of OpenAI; associated with the idea that predicting the next token well requires understanding the world. Lenny: "This idea that to predict the next token well, you basically have to understand the entire universe." 00:10:28 Ameisen: "He has an example that I love, too... predict, 'and the murderer was,' at the end of a mystery novel." 00:10:36

Dario Amodei ("Dario")

Anthropic CEO, criticized for safety warnings. Lenny: "Dario's always out there, 'This is really dangerous. We've got to be really careful.'" 00:01:07

Boris Cherny

Creator of Claude Code, cited for declaring coding solved. Lenny: "As Boris Cherny famously said, 'Coding is solved.'" 00:28:30

Swyx

Host of Latent Space; supplied the question about the fast-decision model. Lenny: "Swyx... I asked him what to ask you. You were on his podcast a year ago." 00:58:56

Daniel Kahneman

Author of Thinking, Fast and Slow, the System 1/System 2 framework. Lenny: "The system one, system two stuff is basically Kahneman's kind of I have that book up there actually behind me." 01:01:08

Elon Musk

Cited for the simulation argument: a simulator would run an interesting time in history. Lenny: "Something Elon said has stuck with me about this, that if you were to run a simulation, it'd be... an interesting time in history." 00:58:07

Ted Chiang

Author of Stories of Your Life and Others; Ameisen's favorite sci-fi. Ameisen: "This is maybe my favorite sci-fi book of all time... it's just take one weird idea... and then write a story of just like, what would the world look like if that happened?" 01:05:07

Max Bennett

(Author of A Brief History of Intelligence; not named by the speaker, so listed only by title.) Recommended for tracing brain evolution. Ameisen: "A Brief History of Intelligence... walks through the history of like basically brains, and five key moments in our evolution." 01:05:07

Oliver Burkeman

(Author of Four Thousand Weeks; not named by the speaker.) Source of the "when I finally" insight. Ameisen: "The 'when I finally' mentality is a common mentality, and it's kind of toxic." 01:07:44

5. Operating Insights

Match Reliability to Task Thresholds Before Delegating to AI

Ameisen's own adoption path is a template: delegate in tiers as reliability crosses a task-specific threshold. At 50% reliability a tool wastes time; at 90-99% it transforms workflow. Re-test delegation boundaries with each model release rather than assuming past experience holds. Ameisen: "If you're at 50%, it's it's like you're it's actually wasting my time to use you because I'm just going to have to go in and fix it half the time. If I have to go in and fix it 10% of the time, it's better." 00:30:45

Build a Two-Speed Decision Architecture (Gut Check vs. Deliberation)

Route simple, high-volume judgments (e.g., "is this reviewer happy or upset?") to fast single-token decision models and reserve long reasoning for strategic questions. Interpretability even shows models internally have a separate "gut check" mechanism, validating the pattern. Ameisen: "You have, for some stuff, you need to think about a long time. You know, it's like, what's our roadmap for the next year?... And then for some stuff where it's like... is this person who wrote us a review happy or upset? You're happy with the sort of gut judgment." 00:59:32

Apply Verification Experiments, Not Just Observation ("Turn It On Always")

Golden Gate Claude illustrates a rigorous validation tactic: to confirm you've identified a causal factor, force it on and see whether behavior matches. Operators can borrow this: intervene on a hypothesized driver to confirm causality rather than inferring from correlation. Ameisen: "How do you know for sure that this thing we decoded... is actually the Golden Gate thing? It's like, turn it on always and see if the model behaves like that." 00:38:46

Treat Weird Anomalies as Deep-Dive Investigations

Anthropic's team runs "investigations or deep dives" on perplexing behaviors, which uncovered a data-labeling bug (36 vs. 12). Operationally: when a system does something bizarre, pursue root cause rather than patching symptoms, because the cause is often an upstream data or incentive flaw. Ameisen: "We'll see a model do something really weird and we're really perplexed by it. And we'll try to understand what happened." 00:36:06

Use Constraint Engineering Alongside Behavioral Alignment

Don't rely only on trusting an agent's behavior; limit its access by default. Ameisen: "In some context it shouldn't have access to the internet. In some contexts, it should only have access to like a small narrow piece of your computer." 00:33:29

6. Overlooked Insights

Models Learn Unprompted "Hunch" Detectors That Are Separate From Reasoning

Mentioned in the lightning round and passed over, this is significant: models have a distinct error-flagging mechanism that fires without doing the math ("that can't be right"). This implies built-in, non-verbal validity checks that could be exploited for cheap hallucination or error detection, and a possible target for reliability engineering. Ameisen: "There's a part of the model that just like reads that, and doesn't do the math, but it's just like, 'That can't be right, because like 4 + 5 definitely under 10.' And so they learn these weird kind of like hunches." 01:02:09

Instructing a Model to "Think About X" Causes Internal Representations With No Training Reason

Ameisen flags that models carry out hidden "thought" on instruction, without any obvious training incentive. This suggests capabilities (and potential steering or deception surfaces) that emerge beyond what's optimized, which matters for both controllability and the unverbalized-reasoning risk he discussed earlier. Ameisen: "You can tell the model, 'Hey, do this task, and think about this other thing.' And then you can use interp to see if there's intermediate representations that map to the other thing, and that works... there's no kind of training reason why the model would have to learn that." 01:02:09