Robot-Use Agents: Why General-Purpose Models May Win in Robotics

1. Key Themes

The Bitter Lesson Applied to Robotics

The central thesis is that general-purpose LLMs trained on massive, diverse data (code, web text, computer use) may outperform specialized robotics foundation models trained narrowly on robot data. This is a direct application of Sutton's "bitter lesson" to physical AI.

"I feel like at the end of the day, it's a lot about the better lesson, right? If you give the agents or you give the AI model more autonomy and you, if you unshackle it a bit more and give it more resources, it can actually do a lot of the things that we fine tune it to do." - Jay 00:04:07 "The key difference between like these VLAs and we're like GPT-6 or LLM isn't necessarily the architecture, but kind of the approach you take towards training... why not just build a really good foundational LLM and then use that to control robots rather than training like a model that's specifically for robotics" - Han Mei 00:06:28

Computer-Use Data as an Unexpected Unlock for Spatial Intelligence

A non-obvious finding: training on computer-use data (dragging cursors, orbiting CAD objects in Blender) appears to transfer surprisingly well to physical robot control, because GUIs were originally designed to mimic physical-world manipulation.

"If you feed a computer use data into a big model like Astra, and by computer use data I mean like you drag a cursor around on a screen to orbit some CAD object in order to design in blender. This tells you like how to reason about spaces... all of these concepts that you need to control a robot." - Han Mei 00:24:12 "We literally made the graphical user interface to be more like the physical world so that we could interface with it... we couldn't get robots to work in the physical world. So we just trained on that. And then now it works." - Francois 00:25:11

VLA Architecture Isn't the Bottleneck — Data Modality Is

Both RT2-style VLAs and pure LLM-code approaches use the same underlying language model architecture. The difference in performance comes from what data unlocks the model's latent capabilities, not architectural innovation.

"VLA's by design architecturally, they're built on top of language models as well... perhaps the better lesson here isn't necessarily what architecture you build necessarily, but it's what data is most useful" - Han Mei (Waddle Labs) 00:04:53

Code-as-Policies as a Historical Bridge to Today's Moment

Early 2022-era "code as policies" research from Google DeepMind demonstrated LLMs could one-shot write robot control code using libraries of primitive functions, well before coding agents were mainstream — foreshadowing today's breakthrough.

"Coding agents can do this very one shot. They did not need additional robot data in order to work with this code because they're already trained on so much coding data... this one shot ability, this in context exploration ability that really motivated a lot of later work" - Han Mei 00:10:11

The Platonic Representation Hypothesis Applied to Robotics

Referencing MIT professor Philip Isola's paper, the guests argue that sufficiently strong models trained on enough diverse data converge toward similar internal representations of reality — implying a strong language model and a strong robotics model should converge in capability.

"The very, very strong language models will have very similar representations of the world with very strong robotics models. And if that's the case, then if you have a really strong language model, you also have a really strong robotics model." - Jay 00:22:13

In-Context Learning Has a Hard Ceiling, Then Needs Distillation Into Weights or Skills

There's a hierarchy of learning efficiency — in-context learning is cheap and fast but caps out based on context window and post-training multi-turn ability; beyond that, learnings must be compressed into tools, skills, or eventually weights (via SFT/RL/LoRA).

"After it hits the context window, context length of the model that it was trained on... you don't improve anymore. You actually just get worse because the model can't attend over everything." - Francois 00:12:45 "If you're Tesla and you have all the data, infinite data, and you're learning, doing self-driving car by ICL, what are you doing? Like, are you kidding? It's like, clearly we wouldn't do that." - Francois 00:13:45

Latency Is the Practical Bottleneck to Real-Time Robot Control, But Improving Fast

Current systems bottlenecked by the reasoning/inference latency of frontier models like Astra, but the trend line suggests this resolves within the year.

"For Fable class LLMs, their latency is improving by around 2X per month, which is very, very fast... If the trends continue, we could get real time control by end of the year." - Jay 00:17:22

The "Sleep" Metaphor: Consolidating Experience Into Faster, Cheaper Policies

Drawing an analogy to biological sleep and memory consolidation (hippocampal replay), the guests describe a future architecture where in-context experience gets periodically distilled into weights or reusable skills/code for speed and cost efficiency.

"Almost everything that is intelligent sleeps, like tell me an intelligent system that doesn't sleep... during sleep compression happens... maybe what the right thing to do is similar to dagger... you're collecting a bunch of data and then you're somewhat reflecting on it. And then you're using it to update your weight file." - Francois 00:27:39

2. Contrarian Perspectives

General-purpose LLMs will beat specialized robotics models, not augment them

Most of the robotics field has spent years building VLAs and robotics-specific foundation models trained on curated robot demonstration data. The guests argue this may be the wrong bet entirely — a sufficiently strong general model (like Astra) may simply subsume specialized robotics models.

"Why not just build a really good foundational LLM and then use that to control robots rather than training like a model that's specifically for robotics and that's more dependent on just robot data." - Han Mei 00:06:52

Frontier lab researchers didn't know they were solving robotics

The massive investment in coding-agent capability (from OpenAI, Anthropic, etc.) was not aimed at robotics at all — it was an accidental byproduct that is now displacing purpose-built robotics research.

"I don't think they had the insight. I don't think they were so insightful to know that, oh, if I did this, then we will have LMs that we good enough to write policies for robotics. And I can displace all of robotics with actual coding agents." - Francois 00:08:52

Society is unprepared for general-purpose robots arriving within ~2 years

This is a much shorter and more confident timeline than most public discourse assumes, and it's framed as near-consensus among frontier labs and robotics foundation model companies, not speculation.

"There's some consensus within the frontier labs and also in the robotics foundation... models companies that we will have general purpose robots within the next two years. Or even earlier. And this is something that society is probably unaware of or even unprepared for." - Jay 00:26:31

Architecture debates in robotics (VLA vs. LLM-as-controller) are a red herring

The field has spent significant energy debating model architectures (VLA vs Vision-Language-Action vs World-Action Models), but the guests suggest this entire debate may be secondary to data strategy.

"Perhaps the better lesson here isn't necessarily what architecture you build necessarily, but it's what data is most useful, right? Like people have been sort of nagging at this VLA data bottleneck for years now, and we're seeing very, very slow progress." - Han Mei 00:04:53

3. Companies Identified

Waddle Labs — Startup building LLMs (harnesses) that control robots by combining agent harnesses with data collection to train better models for robot control. Founded/represented by Han Mei and Vincent. Mentioned as one of the two frontier startups whose viral demo videos (unscrewing caps, multi-robot communication) were cited in Philip Isola's robot-use-agent essay.

"We work on building LLMs that control robots and we do this by doing two things. Building a harness that allows the LMs to do this very effectively, collecting data and using that data to train better LLMs." - Han Mei 00:00:40

RoboCurve — Evals company for physical AI, benchmarking any robot or model (LLMs, VLAs, World-Action Models) across embodiments (hands, grippers, arms, humanoids, quadrupeds). Co-founded by Jay. Also cited in Isola's essay for viral demonstration videos.

"We are an evals company for physical AI. We measure everything, any robot, any model, including LLMs and also classical approaches such as VLA's... and quadrupeds, all kinds of stuff." - Jay 00:00:54

OpenAI (Astra model) — Referred to throughout as "Astra," described as a frontier model whose vision/reasoning capabilities have made a step-function leap in spatial intelligence and robot control performance, likely due to massive pre-training on computer-use and CAD data.

"What Astra does incredibly well is it's like vision capabilities. It was probably pre-trained on way more computer use data than ever before." - Han Mei 00:23:36

Google DeepMind — Creators of the RT2 paper and early "code as policies" research, foundational to the entire lineage of work discussed.

"One of the earliest successful approaches of using AI on robots is the RT2 paper where they use a pre-trained language model on web, text and images and use that to control robots." - Jay 00:02:11

Tesla — Referenced as a company with the scale of proprietary data (self-driving) that would make in-context learning approaches nonsensical compared to full weight training.

"If you're Tesla and you have all the data, infinite data, and you're learning, doing self-driving car by ICL, what are you doing? Like, are you kidding?" - Francois 00:13:45

Figure — Mentioned briefly alongside Tesla as a company operating at a data/deployment scale where full training (not in-context learning) is the appropriate paradigm.

"And if your figure or something like that, but, um, it's amazing how good in the, in low data regimes, uh, you can do with ICL." - Francois 00:14:11

4. People Identified

Philip Isola — MIT professor whose viral essay on "robot-use agents" and earlier paper on the "Platonic Representation Hypothesis" frame the entire episode's thesis: that models trained on enough diverse data converge to shared representations of reality, implying strong LLMs can become strong robot controllers.

"This led MIT professor Philip Isola to suggest in a recent viral essay that we may be entering the era of robot use agents, where general-purpose models could make different robots more capable." - Host 00:00:00

Francois Chollet — Cited for his "measure of intelligence" work distinguishing transduction (learning direct X→Y mappings) from program induction (learning a generator function/code that produces the mapping), a framework used to explain why code-based robot policies are more data-efficient than direct action-output models.

"A lot of conversations I've had with Francis Chalet. And in 2020, maybe even 2018, when he did, uh, on the measure of intelligence, he talked a lot about transduction versus... program induction." - Francois (host) 00:19:13

Han Mei — Co-founder of Waddle Labs; articulated the thesis that feeding LLMs coding, computer-use, and egocentric data (rather than only robot data) produces the most capable robot-use agents, and demonstrated live robot-control examples on the podcast.

"If we take this to the extreme, it's like, why not feed every kind of data, uh, coding computer use egocentric into the same model. I think that's how we get to the most capable like robot use agent." - Han Mei 00:24:59

Jay — Co-founder of RoboCurve; provided the benchmarking perspective and specific data point on the 2x/month latency improvement rate of frontier models, plus the two-year general-purpose robot timeline.

"One thing that we saw is that for Fable class LLMs, their latency is improving by around 2X per month, which is very, very fast." - Jay 00:17:22

Vincent — Co-founder of Waddle Labs alongside Han Mei (mentioned but not directly quoted at length in the transcript).

"I'm Han Mei, I'm from Waddle Labs together with Vincent." - Han Mei 00:00:40

5. Operating Insights

Build the harness before betting on the base model getting smarter

Waddle Labs' architecture explicitly separates fast, repeatable, code-compiled "skills" from slow, expensive live-Astra-in-the-loop reasoning — a pattern applicable to any agentic system where cost/speed constraints matter.

"If it's a repetitive task, you don't want Astra in the loop. Maybe you want to write code. It can run repeatedly very, very fast. Or if it's a task you've done a similar task before, you should be able to call a skill that has compiled and use that skill to do the task faster and handle edge cases better." - Han Mei 00:16:56

Insert a VLM checkpoint at variation points, keep the rest deterministic as code

A concrete operating pattern for building reliable agentic systems in unpredictable physical/real-world environments: make the skeleton of a task deterministic code, but inject model-based judgment only at the specific points where failure detection or object variation requires flexibility.

"While the coding graph is sort of deterministic, there's points of variation that allow it to generalize... when you detect an object, maybe you use a VLM as your tool call. Or when something fails, for example, how do you check that it's failed?" - Francois 00:18:52

Measure per-sample in-context learning improvement to find your model's practical ceiling

A specific, reusable diagnostic: hold out a task, feed in-context examples one at a time, and plot validation performance — this reveals non-monotonic improvement and a hard saturation point tied to the model's post-training context length, which tells you exactly when to switch from ICL to weight updates.

"It caps out very quickly. And so after like 20, 30, maybe 40 examples, it is basically saturated and more examples back into the context... you're constrained by the model's ability to intelligently use all of its context." - Francois 00:12:17

6. Overlooked Insights

GUIs were accidentally training data for robotics all along

This point was made almost as a joke ("it's so funny") but is a genuinely significant historical insight: the 1980s Xerox PARC decision to design graphical interfaces (file systems, folders, CAD rotation tools) to mimic physical-world intuition means decades of human computer-use data inadvertently created a bridge dataset for teaching AI models spatial/physical reasoning — a data source no one deliberately built for robotics but that may prove more valuable than actual robot teleoperation data. This reframes an enormous existing internet-scale dataset (screen recordings, CAD interactions, cursor movements) as an unmined robotics goldmine that any lab with computer-use data already possesses.

"You go to 1980s Xerox park or whatever, like we literally made the graphical user interface to be more like the physical world so that we could interface with it. And like, what we ended up doing is building an environment that was actually helpful for robotics to learn how to use the physical world." - Francois 00:25:11

The "sleep" architecture implies a completely new infrastructure layer nobody is building yet

Buried in the closing discussion is an offhand comparison to DreamCoder's library-refactoring "sleep phase" — implying that the next major infrastructure bottleneck in robot-use agents won't be the base model or even the harness, but a missing "consolidation layer" that periodically compresses growing libraries of in-context skills/traces back into either updated weights or reorganized skill libraries. Neither guest claims to have solved this, suggesting it's a wide-open startup opportunity for whoever builds the equivalent of "memory consolidation infrastructure" for agentic and robotic systems.

"It sounds very dream coder-esque... it had this library of skills and then during the sleep phase, it basically refactored everything into more compact representation... how you manage our growing context of skills, um, of deployment data, all of that is going to be quite interesting." - Han Mei (Waddle Labs) 00:28:36