AI researchers debate how close we are to recursive self-improvement

1. Key Themes

The Moravec's Paradox of AI Progress

A recurring frame across the episode is that AI keeps clearing benchmarks that were expected to signal transformative impact, yet the real-world economic effect lags. John Schulman set up the entire episode with this framing: "There's been a classic thing, almost like Marvec's Paradox, where we think of the AI, if it can do this, it's going to be amazing... And then it solves these things, and then it's not that impactful." 00:00:55 Charlie O'Neill added that this creates a repeating hype cycle: "a new model comes out and people are blown away and they're like, this is it, this is AGI. But then they use it a bit and then it starts to feel dumb after a month or so." 00:02:11

Distillation Is the Great Equalizer Against Centralization

Despite huge scaling advantages held by frontier labs, the panel argued that distillation prevents winner-take-all dynamics. Charlie O'Neill: "distillation is the main thing that fights against the centralizing force. Because basically anything that can be learned through RL can be distilled very easily." 00:18:53 The speaker referred to as Unknown/Speaker_04 (likely Baron Milich or Charlie, given context about lab internals) made a striking claim that Chinese labs (GLM, Kimi) may already surpass some frontier US models partly because of access to router/proxy service data capturing real coding-prompt distributions used in China: "isn't it weird how Sonnet 5 and Opus 5 are like almost objectively worse models than like GLM 5.3, Kimi K3, even though they've had access to like not only distillation but logic distillation" 00:23:15

RSI Is Cumulative, But Real-World Deployment Is Non-Stationary

A key distinction drawn is between tasks that "lock in" progress permanently (like AI R&D discoveries) versus tasks requiring constant relearning (like being a law firm associate). "I think the real world and the reason like people are thinking so much about like continuing learning is it's not really a cumulative task... That's not as clean of an example of a cumulative task like RSI is." 00:48:05 This matters because it suggests RSI (research automation) may progress faster than general-purpose deployment automation.

The Sample Efficiency Gap Might Be the Real Bottleneck

Baron Milich raised that models may be a "million fold" less sample-efficient than humans at learning from real-world experience: "plausibly million fold behind humans in terms of how much data a human sees from birth to adulthood versus how much a model sees." 00:46:39 This is proposed as one of the few genuine technical reasons recursive self-improvement might not happen on schedule.

Data Explains Far More of Pre-Training Progress Than Architecture

Baron Milich shared original research: "the data seems to explain like 9x of a compute efficiency gain. But the architecture improvements explains like a 3x compute efficiency gain at a very small scale." 01:06:42 This reframes the popular narrative that algorithmic breakthroughs drive most of the gains — data curation may matter more.

RL's Success Comes from High Signal-to-Noise Ratio, Not More Bits

John Schulman explained why RL scaling worked better than skeptics (including his own past writing) expected: mid-training does ~80% of the work, and RL's single bit of reward signal is uncontaminated by the noise of exact-token matching in SFT. "For RL, you only get the one bit. And that means that that signal is not drowned out in the noise of all the other bits the model has." 01:19:43

Objective-Setting and Alignment Remain the Last Human Job

Multiple speakers converged on the idea that even full automation of AI R&D leaves a durable human role: deciding what "good" means. Charlie O'Neill: "the last job for humans... will last the longest is like defining the objective and like deciding what we actually want." [01:15:46 - actually 00:15:46] Baron Milich summarized: "Alignment is the final job." 00:16:31

Continual Learning / "Hive Mind" Learning Is Beginning, But Bottlenecked by Incentives and Catastrophic Forgetting

The panel described early forms of live learning from deployment (e.g., Cursor's Composer model), but flagged deep technical barriers: "when you zoom in close enough at like a micro level... all the methods kind of break down a bit... you see both catastrophic forgetting... and degradation of general... capabilities." 00:54:58 Beyond technical issues, Charlie O'Neill noted a purely economic constraint: "companies aren't going to want to have the model provider learn from all of their deployment because that might just reduce the advantage of their business." 00:52:51

2. Contrarian Perspectives

Frontier Labs May Have No Real Advantage in RL Environments Anymore

Contrary to the assumption that frontier labs' proprietary, hard-to-replicate RL environments are their moat, one speaker argued real-world deployment data (prompt distribution) may matter more than environment sophistication: "the frontier labs don't necessarily have much of an advantage, if at all, in aural environments now... the best measure of a capability is the very, very hard aural environments you've made at the frontier. And so if you have access to those aural environments as anthropic and you have access to logic distillation and you've still made a worse model, then maybe..." 00:23:42 This directly challenges the "compute moat" thesis common in AI investing circles.

Domain-Specific Training (e.g., baking in finance/PowerPoint skills) May Be a Sign Labs Don't Actually Believe Their Own AGI Timelines

Baron Milich pointed out a logical tension: if labs truly expect near-human general learning ability, why train so hard on narrow domain skills? "why would you try to bake in all these skills of like working with PowerPoint or something? Wouldn't you just expect the model to be able to pick that up on while it's deployed?" 00:36:06 The alternative explanation offered is cynical: labs are optimizing for revenue to fund the "real" RSI project happening in parallel — a contrarian read on the strategy of major labs like Anthropic.

RL Doesn't Actually Deliver More Information Than SFT — It Delivers Cleaner Information

Against the intuitive view that RL succeeds because it generates more valuable data, John Schulman argues the opposite — RL works because it deliberately ignores most of the available bits, keeping only the answer-correctness signal, avoiding the noise of imitating another model's reasoning tokens exactly. This reframes RL's power as arising from restriction, not abundance: "you only get the one bit... that signal is not drowned out in the noise of all the other bits the model has." 01:19:43

Post-Training Quality May Be Regressing Due to Over-Optimization, Not Data Scarcity

Contrary to assuming newer, larger models are strictly better, the panel discussed evidence that Opus 5 may be worse in "feel" than Opus 4.6 due to excessive AI-judge self-checking baked in via distillation: "Opus 5 really feels like it's got this like AI as a judge checking every possible thing it's done... it doesn't necessarily have the big model smell... to know when to like stop." 00:24:30 This is a rare public acknowledgment that frontier model quality can regress version-over-version.

Sim-to-Real Transfer May Never Fully Work for the Highest-Value Tasks

Rather than assuming simulated RL environments will eventually generalize to running businesses, winning court cases, or day trading, Baron Milich raises the possibility that "you do need weight updates from these kinds of interactions in order to get better at them," and that current sample inefficiency in weight updates "comes to bite us" precisely in these highest-economic-value domains. 00:47:07

3. Companies Identified

Zyphra — Open-source AI model developer; Baron Milich is CTO. Mentioned as the affiliation grounding his credibility as an open-model researcher. 00:00:00

Thinking Machines — AI lab where John Schulman is chief scientist; he previously co-founded OpenAI and led the RLHF work behind ChatGPT. Mentioned as a leading open-ish lab whose chief scientist can speak candidly. 00:00:00

BaseTent — Company where Charlie O'Neill is head of model training. Cited as one of the "open-ish" labs represented in the discussion. 00:00:00

OpenAI — Referenced repeatedly as the birthplace of RLHF/ChatGPT and Radford's original GPT next-token-prediction insight. Also referenced re: an early giant sparse (Switch Transformer-style) 1-trillion-parameter model experiment. "That was what they did before OpenAI. That was like Google the switch transformer." 01:15:00

Anthropic — Cited repeatedly as the pacesetter lab whose environment lineage (coding → finance → the "long tail of the working economy") is seen as the model other labs are following: "Anthropic's lineage of environments is like a very clear example of this." 00:35:07 Also cited for Opus 5's regression issue and its ability to find bugs "through tens of thousands of human and LLM" hours that could be compressed into a single environment. 01:02:04

Antithesis — Automated software testing company. Jane Street began using it in early 2025 and was impressed enough to invest. Ron Minsky (Jane Street) praised it directly: "it was able to shake out bugs that were otherwise going to be really hard to find... it helps make those systems more reliable, but also because it helps the teams that build it to just move faster." 00:17:38 He frames it as solving the "verification bottleneck" created by AI-generated code. 00:18:02

Jane Street — Quant trading firm; described using Antithesis in production and launching an ASIC design competition (protocol emulator chip) with cash prizes for chip tapeouts. 00:17:24, 00:40:01

Cursor / Composer — Cited as a leading real-world example of near-live continual learning: Cursor's Composer model reportedly did online reinforcement learning on actual generative output (not just the tab-completion model), redeploying every five hours based on CursorBench performance. "if it improved on CursorBench, they would deploy the new model like every five hours. And if it didn't, they would like throw that version out." 00:45:00

Harvey — Legal AI company mentioned as doing similar rapid deployment-data-driven fine-tuning as Composer, specific to legal agents. 00:42:55

GLM (Zhipu) / Kimi (Moonshot) — Chinese open models (GLM 5.3, Kimi K3) singled out as possibly outperforming Anthropic's Sonnet 5/Opus 5 despite the West's compute and environment advantages — used as evidence for the "prompt distribution matters more than environment sophistication" thesis. 00:23:15

DeepSeek — Cited for architectural choices (e.g., compressed/sparse attention) explicitly optimized around real-world inference efficiency rather than pure pre-training loss minimization. 01:09:36

Periodic Labs (via Liam) — Referenced for historical color on an early OpenAI-era 1-trillion-parameter sparse model experiment. 01:14:38

Falcon — Open-source 1-trillion-parameter model cited as evidence that huge parameter counts have existed for years without linear year-over-year doubling. 01:14:38

x.ai / Grok (Grokbot) — Referenced by Baron Milich as a production tool he personally uses for podcast editing workflows, demonstrating practical agentic tooling. 00:59:32

4. People Identified

John Schulman — Chief scientist at Thinking Machines, co-founder of OpenAI, led RLHF work behind ChatGPT. Repeatedly the voice of technical grounding on scaling laws, RL signal theory, and historical context ("wisdom of the ages") on pre-2012 deep learning. Praised implicitly through his detailed, well-substantiated explanations of why RL works: "the objective ignores all the other bits... that's what really, it's really a super dramatic increase into the signal to noise ratio." 01:19:43

Charlie O'Neill — Head of model training at BaseTent. Provided detailed operational insight into post-training team structure, distillation mechanics, and the difficulty/realism axes of RL environments. Noted for nuanced framing of why big models generalize better than distilled smaller ones.

Ron Minsky — Co-leads Jane Street's tech group. Cited for direct praise of Antithesis and articulating the "verification bottleneck" problem created by agentic code generation as a top operational concern for engineering orgs today.

Ryan Greenblatt — Referenced (not present) as a prior podcast guest whose framing — that AIs could get better at general science, not just AI R&D, via simulation-incentivized progress — was cited approvingly multiple times. "Ryan Greenblatt was on the podcast recently. And it made this point that you could imagine as the AIs get more and more capable..." 00:05:29

Alec Radford — Referenced as the originator of the "next token prediction" GPT insight, credited as a singular researcher whose approach beat DeepMind's "solve intelligence via superhuman game-playing" strategy: "one random researcher like Radford being like, I'm going to try and just predict the next token off a very wide swath of data." 00:15:22

Dario Amodei — Referenced (not present) via Baron Milich recounting a prior podcast conversation where he pressed Dario on why Anthropic bakes domain-specific skills (PowerPoint, finance) into models if true continual learning generalization is expected soon. 00:36:29

Jerry Han — Student at Princeton who co-authored Baron Milich's research comparing pre-training recipes (2019–now) against datasets (2019–now) in a full grid, finding data explains ~9x compute efficiency gains vs. 3x from architecture. 01:06:42

Liam (Periodic Labs) — Cited for a Twitter post detailing an early OpenAI-era experiment training a very sparse 1-trillion-parameter model, "very, very good at like knowledge, but terrible at reasoning because it was so sparse." [00:15:00 / 01:15:00]

5. Operating Insights

Prompt Distribution Realism Beats Benchmark Difficulty for Producing Usable Models

Charlie O'Neill laid out a two-axis framework for RL environment design — difficulty vs. realism — and argued that naive distillation only captures benchmark-style difficulty, missing the "realistic coding agent setting" nuance: "if you try to do distillation naively, you end up just sort of matching the teacher on the benchmarking distribution... if you don't have enough of the environments that really exercise the capabilities in these like trickier realistic settings, then you're not going to get those into your student model." 00:26:15 For operators building AI products, this argues for prioritizing real usage-pattern data collection over synthetic benchmark-style RL tasks.

The "Diff the Bugs" Method for Continuous Model Improvement

One speaker described the actual practical workflow at frontier labs for updating models: rather than retraining from a blank slate, teams take the delta of bugs/improvements found since the last version and turn them directly into new training environments: "we go up to the very like edge of the lineage and say, okay, like here are the bugs like... Anthropic's found in their training stack in the last few months. We'll turn those into environments." 00:29:29 This is a concrete, replicable pattern for any team doing iterative model or product improvement — treat "diffs since last release" as your training curriculum.

Router/Proxy Traffic Data Is an Underrated Distillation Asset

Charlie O'Neill flagged that Chinese router/proxy services designed to access blocked US frontier models are simultaneously harvesting the ideal training signal for distillation: "these router services are collecting and selling some of the data. So I think this is like a very useful data set for distillation because it gives you the perfect prompt distribution." 00:21:11 For operators, this suggests that any wrapper/proxy layer sitting between users and a frontier model accrues significant proprietary strategic value beyond its nominal function.

Use RL Sparingly for Knowledge, Not Just Capability

A specific technical/operating distinction: RL is good at capability-shaping but bad at injecting explicit factual/contextual knowledge, which requires heavier compute investment in bespoke environments: "RL is not very good at like, it is good at like getting capabilities in, but it's not as good as getting like knowledge in... you have to put in a lot of compute to create the right environments to get the knowledge in." 00:55:57 Operators building domain-specific fine-tunes should default to SFT/mid-training for knowledge injection and RL for behavior/judgment shaping — not the reverse.

6. Overlooked Insights

The Chinese Model Performance Anomaly Signals a Coming Commoditization of Frontier Labs' "Secret Sauce"

Buried in the discussion is a genuinely under-examined empirical anomaly: despite Anthropic having superior proprietary RL environments and even the ability to distill its own logic traces, Chinese open models (GLM 5.3, Kimi K3) reportedly outperform Sonnet 5/Opus 5 in practice. The tossed-off explanation — that these Chinese labs simply have better real-world prompt distribution data via router/proxy services scraping US frontier model usage — is a much bigger claim than it sounds: it implies the entire "compute + proprietary RL environment" moat thesis that underpins most frontier lab valuations may be secondary to something almost anyone could acquire (deployment-traffic access). If true, this is a serious threat to the durability of frontier lab competitive advantage that neither speaker fully interrogated before moving on.

The Economic Incentive Against Continual Learning Is a Bigger Bottleneck Than the Technical One

While the conversation spent enormous energy on the technical hurdles to continual learning (catastrophic forgetting, plasticity, sample inefficiency), Charlie O'Neill's single-sentence aside was arguably the most important point in the whole episode and was never followed up on: "a big part of that is actually about incentives rather than being a technical question. So like companies aren't going to want to have the model provider learn from all of their deployment because that might just reduce the advantage of their business." 00:52:51 This reframes the entire "hive mind" / recursive self-improvement race: even if the technical continual-learning problem gets solved, enterprise customers have a structural reason to withhold their deployment data from being absorbed into shared foundation models — meaning the eventual solution (per Speaker_04) is likely modular, customer-owned adapters ("cartridges," compressed KV caches) rather than one continuously-improving shared super-model. This has direct implications for how value accrues in the AI stack — favoring infrastructure that lets enterprises own their "learned" module rather than labs that want to absorb everyone's data centrally.