World Models, Explained

1. Key Themes

Intelligence Per Sample Is the Central Unsolved Problem in AI

Francois frames the two great unsolved problems in AI as "intelligence per watt" and "intelligence per sample." The latter — how much smarter a model gets from each additional data point — is the crux of why current systems fail at novel tasks despite having ingested the entire internet.

"The two major problems that we have left to solve is intelligence per watt and intelligence per sample. Intelligence per watt is like how many perplexity points we get per watt of spend. And then intelligence per sample is basically if I have one additional sample in my data set, how much more intelligent am I getting?" — Francois Chaubard 00:00:49

A Perfect World Model Enables Zero-Sample Learning

The theoretical ceiling of sample efficiency is zero samples — and we already have a proof of concept: Newtonian physics. NASA can intercept an asteroid years in advance without collecting a single additional training example because it has a perfect, differentiable world model. This reframes AGI as a world modeling problem.

"The perfect sample efficiency would be zero samples... imagine I had a perfect world model. Then I should never go to the environment to go and collect samples to train on... It's called Newton's second law of motion." — Francois Chaubard 00:02:37

Non-Differentiability Is the Core Structural Barrier to Scaling RL

The moment you introduce another agent (an adversary, another driver, another person) into an environment, the transition function becomes non-differentiable — you can't backprop through another agent's brain. This forces the "awful" area of reinforcement learning and explains why chess-style optimal control does not generalize.

"It was all differentiable until this new variable. And I can't backprop through your brain... It's completely non-differentiable now. And I am resorting and I have to resort to this awful area called reinforcement learning, which is just super brutal." — Francois Chaubard 00:12:32

The Action Space Explosion Is What Actually Breaks AlphaGo-Style Approaches

People celebrate AlphaGo, but its MCTS approach fundamentally cannot scale. Even the jump from Go (361 actions) to a simplified self-driving car (36,000 actions) — let alone a six-axis robot arm (10^16 actions) — makes the test-time planning cost completely intractable.

"For a similar depth, you need like 2 million of them. So for a similar depth, and then that's still to do a depth of 30, I would still have to do this times 30. This would be 60 million invocations of the model. So that better be a small model... And that's just to do one action." — Francois Chaubard 00:34:54

The Dreamer Paper Series Is the Intellectual Foundation for Modern Robotics World Models

Danijar Hafner's Dreamer series (V1 through V4) demonstrated that a policy trained entirely on synthetically imagined rollouts — generated by a world model — can outperform policies trained on real data. The capstone: Dreamer V4 was the first system to mine diamonds in Minecraft, using almost entirely synthetic data.

"He trains a world model on this type of data and then injects action conditioning on a very small amount of data to get to this world model that can sample a lot from it and then train a policy on those synthetic imaginated rollouts. And it's the first paper to mine diamonds in Minecraft... And it did it all on synthetic data, which is kind of crazy." — Francois Chaubard 00:51:21

Pretrained Video Diffusion + Small Action-Conditioned Fine-Tuning Is the Current State-of-the-Art Path

The practical recipe emerging now: take a pretrained video diffusion model (Sora, CogVideoX, Wan, etc.), fine-tune it with a small amount of action-conditioned teleop data, use that as a neural simulator, and train your robot policy inside the simulation. A recent paper (Dream Zero) accomplished this with only 500 hours of teleop data.

"If you have your CogVideoX or your Sora or Wan, exactly all those models — basically the idea is now we have them and they're already trained and they're great. Let's do a small amount of action conditioning on them to get to this world model. And then we can sample from it a bunch and then train." — Francois Chaubard 00:53:26

"They have this joint model of state transitions and actions. They train it by first instantiating it with the open source Wan video diffusion model. And then it only takes them about 500 hours of teleop data, which is basically exactly this to get it to be pretty good." — Ankit Gupta 00:54:05

Cross-Embodiment Gap Is a Deeply Underappreciated Problem in Robotics

Even within Tesla's fleet — where the same software runs on essentially similar cars — policies trained on one vehicle model fail on another. For humanoid robots with radically different morphologies, this problem is dramatically worse, making generalization across robot bodies one of the hardest unsolved issues.

"If I were to train this policy on Tesla Model X and I were to put it on a Tesla Model Three, it wouldn't work. Like it totally wouldn't work... Lane McIntosh, who now runs Tesla FSD — I would bet money that they shard the data per model per car type." — Francois Chaubard 00:47:58

The Human Brain Is a World Model — Supported by Both Neuroscience and Cognitive Science

Multiple lines of evidence are cited: the 1967 cognitive science study showing mental practice of basketball layups produces 23% improvement (almost identical to physical practice at 24%), the great cortical expansion 10 million years ago interpreted as world modeling hardware growth, and a recent University of Washington paper explicitly stating the cortex predicts consequences of actions.

"Each cortical area estimates both latent sensory states and actions and the cortex as a whole predicts the consequences of those actions. That sounds like a world model to me." — Francois Chaubard 00:56:19

Sleep Is the Missing Architectural Element in Current AI Systems

Francois makes a striking evolutionary argument: every intelligent species sleeps, and sleep involves hippocampal sharp-wave ripples that replay experiences in reverse seven times — what looks exactly like offline policy training. No current AI architecture has an equivalent awake/sleep cycle for consolidating experience.

"There's this thing called shortwave ripple where the hippocampus, when you're sleeping, emits these spike trains that are actually reversed from when they actually happened... for like seven times. And then it stops. So there's something happening there that's very... training something. And if you don't sleep, you don't have long-term memory." — Francois Chaubard 00:11:04

---

2. Contrarian Perspectives

We Will Not Have Rosie the Robot by 2026 — But the Path Is Now Clear

Against the mainstream hype cycle predicting near-term household robots, Francois dismisses 2026 confidently while simultaneously arguing the technical path is now legible — it just requires more data and compute than people acknowledge.

"This is 2026 will be the year of the robot. We're going to have Rosie the robot in your house. You know? Um, yeah, no, I don't think so." — Francois Chaubard 00:51:51

Taste and Entrepreneurial Intuition Are Forms of World Modeling — Not Mystical Talent

Most people treat "taste" and business judgment as ineffable. Francois argues they are literally world models built over years of calibrating predictions about what customers will want — the same mathematical object used in robotics.

"What is taste? It's like predicting that other people are going to like this thing. And so we built this world model over years of entrepreneurship, 10 years of getting it wrong. Maybe Bill Gates, Steve Jobs, and Jensen have 50 years of world modeling experience to know what people want." — Francois Chaubard 00:04:05

The Transformer Architecture Will Be Superseded — Sam Altman Agrees

The transformer does no compression in the time domain. It retains all context without any consolidation mechanism, which may be the wrong architecture for world modeling. Francois notes Sam Altman recently expressed agreement that a better architecture exists.

"Sam Altman just kind of came and said that he thinks that there's definitely an architecture that's going to be more performant than the transformer. I think he's right. The transformer doesn't do compression in the time domain at all. It just keeps around everything." — Francois Chaubard 00:06:57

The Brain Is the Optimizer, Not Just the Model — and This Is Why Current AI Doesn't Pass the Squint Test

Most AI researchers are trying to build better models. Francois contends the brain's profound insight is that it is simultaneously the optimizer and the model executor — and that distinction is what current architectures miss entirely.

"I'm getting to the conclusion that I think that the brain is the optimizer, not the model. And that the brain emits, has models that it invokes, but the brain is somehow also the optimizer itself. And so in that way, it doesn't pass the squint test." — Francois Chaubard 00:10:09

Physics-Informed Neural Networks (PINNs) Don't Actually Work — a Quiet Failure Most Ignore

Despite years of hype around PINNs as a way to inject physical laws into neural networks, Francois flatly states they don't work in practice, attributing the failure to SGD's inability to reach the numerical precision required for physical simulation and likely an architectural mismatch.

"Pins doesn't really work. What is pins? Physics informed neural networks... I think it's an SGD issue. I think it's probably an architecture issue." — Francois Chaubard 00:54:28

---

3. Companies Identified

Tesla / Tesla FSD

Autonomous vehicle division of Tesla. Mentioned as the only company with a true flywheel of state-action paired driving data from its entire deployed fleet — a structural competitive moat that makes its dataset fundamentally different from any competitor using dashcam footage alone.

"Unless you're Tesla. Tesla has this. So this is a huge competitive moat — like what do people do in that state? And then so you can behavior clone to go from here to here." — Francois Chaubard 00:42:07

"Lane McIntosh, who now runs Tesla FSD — I would bet money that they shard the data per model per car type." — Francois Chaubard 00:48:23

Waymo

Autonomous vehicle company. Mentioned as one of the few examples of self-driving technology that actually works at a level consumers can experience, used to ground the self-driving car analysis in real products.

"Many people have started to experience for the first time because we have some self-driving cars that actually work. You have Waymo and Tesla FSD and whatnot, that seem like they kind of work." — Ankit Gupta 00:36:46

Wayve (and Gaia)

UK-based autonomous vehicle company building world models for self-driving. Cited as the primary real-world implementation of the video-diffusion-to-world-model pipeline for self-driving, having raised $1.5 billion to pursue this approach.

"This is exactly what Wayve did with Gaia. And Gaia — I think they've raised $1.5 billion to basically run with this idea for self-driving car." — Francois Chaubard 00:53:55

NVIDIA

Semiconductor and AI infrastructure company. Mentioned alongside Wayve and the Dream Zero paper as implementing the video diffusion → action conditioning → world model pipeline for robotics.

"I think a bunch of companies — NVIDIA — this paper here is basically talking about doing exactly the same, this Dream Zero for robotics." — Francois Chaubard 00:53:55

Figure / Pi (Physical Intelligence)

Humanoid robotics companies. Cited as the exemplars of the hardest version of the action space problem, where the robot action space reaches approximately 10^16 combinations for a six-axis arm alone.

"If you're like a humanoid company like Figure or Pi or whatever, again, same star framing — a is now even bigger." — Ankit Gupta 00:45:58

SpaceX

Aerospace company. Used as a real-world example of model predictive control (MPC) in action — landing rockets on ocean platforms by leveraging a perfect Newtonian world model, zero training data needed.

"This is like the way that SpaceX lands the rocket on some platform in the ocean... you minimize your loss function." — Francois Chaubard 00:09:15

DeepMind (AlphaGo / AlphaZero)

AI research lab. The AlphaGo system is analyzed in depth as the canonical example of MCTS-based test-time planning — praised for its achievement but dissected to show why it fundamentally cannot scale beyond small, discrete action spaces.

"We did 800 MCTS simulations to cover 361 possible actions on average. So that gives us about two samples roughly on an expectation for every single action. So here you need like 2 million of them for a similar depth." — Francois Chaubard 00:34:54

OpenAI (Gym / ChatGPT context)

AI company. OpenAI Gym is cited as the benchmark environment used in Schmidhuber's original world models paper. Sam Altman is also quoted as agreeing a better architecture than transformers likely exists.

"Sam Altman just kind of came and said that he thinks that there's definitely an architecture that's going to be more performant than the transformer." — Francois Chaubard 00:06:57

---

4. People Identified

Danijar Hafner

AI researcher, author of the Dreamer paper series (V1–V4). Identified as the single most important practitioner pushing world models for robotics forward over the last seven years, with the Minecraft diamond-mining result as the capstone achievement.

"Danijar Hafner publishes Dreamer One, I think in November of 2018. And then now he's been on this rampage for the last seven years publishing these papers and Dreamer Four, I think, is the capstone of it." — Francois Chaubard 00:51:21

Jürgen Schmidhuber

AI researcher, named inventor of the original "World Models" paper. Identified as the intellectual originator of training a policy entirely on imagined rollouts from a world model — the first time this was demonstrated to work.

"The first real person that went after this was Jürgen Schmidhuber. He has this really cool paper called 'World Models,' very aptly named. And it's basically... the first time in my understanding that that actually happened and it actually works really well." — Francois Chaubard 00:49:29

François Chollet

AI researcher at Google, creator of the ARC-AGI benchmark. Cited as being "on the forefront" of redefining intelligence as the rate of skill acquisition rather than the accumulation of skills — a framing that motivates the entire episode.

"François Chollet has been on the forefront of this thinking and talking about intelligence as a rate of skill acquisition versus skill acquisition... how fast do we get smarter with more and more samples?" — Francois Chaubard 00:01:15

Yann LeCun

Chief AI Scientist at Meta. Cited for the JEPA (Joint Embedding Predictive Architecture) concept and the memorable "squint test" framing that we didn't need flapping wings to achieve flight, used to evaluate whether world models biologically resemble the human brain.

"The squint test for me comes from Yann LeCun. We didn't need flapping wings to achieve flight. And to that I say, well, we did need two wings." — Francois Chaubard 00:09:24

Shawn Druckmann (Shaw Druckmann)

Neuroscientist at Stanford. Cited for the view that the entire purpose of the neocortex's expansion during the great cortical expansion 10 million years ago was to develop better and better world modeling.

"There's this neuroscientist at Stanford named Shawn Druckmann, who basically is of the view that the entire point of the growing neocortex during the great cortical expansion 10 million years ago was to get better and better world modeling." — Francois Chaubard 00:05:13

Lane McIntosh

Head of Tesla FSD. Mentioned as a personal acquaintance of Francois (played hockey together at Stanford), cited as a knowledgeable source on how Tesla likely handles cross-embodiment data sharding internally.

"Lane McIntosh, who now runs Tesla FSD — I played hockey with at Stanford — I would bet money that they shard the data per model per car type." — Francois Chaubard 00:48:23

Richardson (1967)

Cognitive science researcher. Author of a 1967 study demonstrating that mental practice of basketball layups (imagined, no physical action) produces 23% improvement vs. 24% for physical practice — empirical evidence that humans run high-fidelity internal simulations.

"A study by Richardson that basically showed that if you take a cohort... who just blindfold them and they imagine laying up basketball, they improve 23 percent. Against the control. I mean, that's insane. It means that we have this crazy good world model." — Francois Chaubard 00:04:27

Steph Curry

NBA player. Cited as an example of elite human world modeling — feeling a dead spot in a basketball court mid-dribble because his internal physics model was precise enough to detect the anomaly in real time.

"I saw this one video of Steph Curry dribbling a basketball on a court and he just felt that there was a dead spot in the court... he knew it wasn't him. It was the court and he found a dead spot in the court. Like that's how good the human brain is at world modeling." — Francois Chaubard 00:06:28

---

5. Operating Insights

Data Mixing Composition Is a Critical, Often-Neglected Training Detail

When training neural networks on imbalanced real-world data (e.g., 99% normal driving, 1% edge cases), the model will hallucinate normal conditions even when placed in an edge case — because SGD follows the gradient of the dominant class. Carefully controlling mini-batch class composition is essential and underappreciated by practitioners.

"You'd have to train on — you have to be very careful about your data mixing to make sure you get this right to solve this problem that no one really has... when you're training a neural network, it has a tendency to collapse if you don't keep the mini-batch composition very even over the class space." — Francois Chaubard 00:05:02

The Competitive Moat in Robotics/Autonomy Is Paired State-Action Data, Not Model Architecture

The structural insight that Tesla has and everyone else lacks is not a better model — it is paired (state, action) data from a deployed fleet where users don't know they're training the system. Any robotics or autonomy company that can build a similar flywheel at scale will have a durable moat that model improvements alone cannot close.

"Unless you're Tesla, we have a bunch of video of people driving cars... what we don't have is the actions they took. Tesla has this. So this is a huge competitive moat." — Francois Chaubard 00:41:34

Sharding Training Data by Embodiment Is Probably Necessary — Even for Superficially Similar Hardware

The cross-embodiment gap suggests that mixing data across different robot or vehicle types without explicit sharding will degrade policy performance. The practical operating implication: when deploying across multiple hardware variants, maintain separate data pipelines per embodiment type rather than assuming a single model generalizes.

"I would bet money that they shard the data per model per car type. I just — because that's what I would do. There's no way that I would trust data that was collected on a Model X on a Model Three." — Francois Chaubard 00:48:23

---

6. Overlooked Insights

The World Model for Literal Weather Prediction (GenCast) Is an Under-Discussed Breakthrough

Mentioned only in passing and then immediately pivoted away from, Google DeepMind's GenCast paper applies the exact same video-diffusion world modeling architecture — the same mathematical structure discussed for robotics — to global weather prediction. This is significant because it implies the world modeling paradigm is domain-agnostic and may be one of the most powerful approaches in computational science broadly, not just robotics. Weather forecasting has enormous economic value (agriculture, insurance, energy, logistics) and if the same architecture that mines diamonds in Minecraft also outperforms numerical weather prediction, it suggests the GenCast approach could quietly become one of the most commercially important AI applications of the decade.

"We have to talk about the world model for the world... basically they do this exact same thing where the key unlocks for this whole thing was getting diffusion to work in very high dimensional state spaces... and they did this for the entire world with this exact same diffusion steps... predict the next state of the world based on those things with these Langevin diffusion rollouts." — Francois Chaubard 00:55:27

Eliminating the Cross-Entropy Head via JEPA-Style Embedding Loss Could Fundamentally Change LLM Training Economics

Buried in the JEPA discussion is an offhand observation that researchers are experimenting with replacing the expensive cross-entropy softmax head in LLMs with a simple embedding distance loss between predicted and actual next-token representations. If this works — and Francois notes people are actively exploring it — it would dramatically reduce training compute costs for every LLM trained, because the cross-entropy head over a vocabulary of hundreds of thousands of tokens is one of the most expensive components of transformer training. This was mentioned in one breath and never returned to, but the implication is significant for anyone involved in LLM infrastructure, training efficiency, or foundation model economics.

"What you can actually do is have the LLM output the next token... and actually have this be close to E T plus one. And a lot of people are playing with this idea and getting rid of the cross-entropy loss entirely... the cross-entropy head is actually very expensive. So this is very cheap... people are playing around with this idea as a cheaper proxy for the cross-entropy loss." — Francois Chaubard 00:01:40