Why Diffusion Will Win AI Inference with Inception Co-Founder and CEO Stefano Ermon

1. Key Themes

The Inference Bottleneck Is the Real Battleground, Not Training

Ermon frames the entire architectural bet around inference economics rather than training efficiency, arguing autoregressive models have a structural flaw at inference time that diffusion fixes. "If you think about inference, now not training, inference generation, autoregressive models are still sequential. The computation is one left to right, one token at a time... That kind of workload does not map well to GPUs. That kind of workload is extremely memory bound." 00:08:50 He draws a direct historical parallel: "There was an inflection point in 2017 when people switched from RNNs to transformers... [transformers] scaled better for training... But if you think about inference... the equivalent at inference time is a diffusion based model." 00:08:02

Diffusion Models Are Structurally Parallel, Which Compounds Across the Stack

The parallelism advantage isn't just about chat latency — it cascades into RL post-training economics. "A lot of the bottleneck is generating rollouts... inference is again a key bottleneck for RL post-training. And so if you have a model that scales better at inference time, then automatically you're going to get better scaling during RL training." 00:10:33

Software-Level Speed Gains Are Multiplicative With Hardware Gains

Rather than positioning diffusion as a hardware play, Ermon frames it as complementary to custom silicon, letting commodity GPUs match specialized chips. "If you have a diffusion-based LLM that it's built to be parallel, it's accelerating at the software level, then you can get the same speed on NVIDIA GPUs, which means much more availability... and lower costs, higher quality." 00:18:29 Sarah Guo adds: "The gains that you get from the software, they are multiplicative with the gains you get from the hardware." 00:19:17

Latency-Sensitive Workloads Are a Real, Sizable, Addressable Market Today

Ermon isn't claiming diffusion wins everywhere — he sizes the addressable wedge concretely. "I was doing some estimates and I think that is like between 20 and 30% where latency is really, really important... Addressable by models that are within a given latency budget [that] will give you the highest possible quality." 00:29:45

Generative Modeling Is Fundamentally a Compression Problem, Which Explains Why Diffusion Generalizes to Discrete Data

Despite skepticism that diffusion (built for continuous data) should work on discrete text/code, Ermon grounds the empirical result in information theory. "Whenever you train these models, you're effectively trying to identify common structure by trying to find an efficient way of compressing the data... we were actually able to identify at the GPT-2 scale the same amount of structure as an autoregressive model." 00:22:33

Diffusion's Course-to-Fine Generation Enables Fundamentally Different Controllability

This is a differentiator beyond speed that could matter more long-term. "We know that diffusion models are typically easier to control compared to autoregressive models... [with autoregressive] you kind of have to wait until you've generated the full object to know whether or not it satisfies... a constraint... But a diffusion model, it's more course-to-find generation... you can steer the generation in the direction provided by an external reward function." 00:26:01

Data Efficiency May Be Diffusion's Next Frontier Advantage

Beyond speed, Ermon flags an emergent property that could matter as data becomes scarce. "There is a decent amount of evidence in the academic literature... that diffusion-based models are more data efficient compared to autoregressive models... it's effectively doing data augmentation in the sense that the same image is augmented by many noisy views." 00:28:04

Building the Full Stack (Not Just the Model) Is the Actual Moat

Because no ecosystem exists for diffusion LLMs, Inception had to build serving infrastructure from scratch — and that infrastructure itself became defensible IP. "You cannot run this diffusion-based LLMs on VLLM or SG-Lang. We had to build our own serving engine... even if you train the diffusion based LLM, if you don't have the serving engine... you're still stuck and you still cannot use it." 00:12:53/00:21:20

Academia Remains a Source of Contrarian, High-Leverage Bets Despite Compute Disadvantages

Ermon pushes back on the narrative that academic AI research is dead, citing his own track record. "One of the nice things about academia is that it allows you to take these contrarian bets... People are not scared about taking bets. And that's why academia has been so impactful." 00:37:12

2. Contrarian Perspectives

Betting Against the Entire Industry's Architectural Consensus

Every frontier lab has converged on autoregressive transformers for text/code; Ermon argues this is not settled science. "There is discrete modalities, text and code, where primarily all the big labs are kind of like betting on the same architecture, autoregressive models... Nobody knows. I think that the jury is still out there." 00:07:33

Speed-Optimized Models From Big Labs Are Already Beatable by a 50-Person Startup

Rather than conceding ground to frontier labs on efficiency, Ermon claims parity or superiority against their fastest models today. "Our Mercury models are on par with the Haiku models, Flash models, Mini Nano models from OpenAI, if you look at benchmarks, while being significantly faster." 00:12:23

Custom AI Silicon May Be a Dead End Compared to Software-Level Parallelism

This challenges the widespread VC and infra thesis that specialized inference chips (Groq, Cerebras, etc.) are the path to speed. The OpenCall case study is the proof point: they abandoned Cerebras custom chips for Inception's software approach on commodity GPUs and got comparable speed with better economics. "They switched over to our diffusion-based LLMs because they can essentially get the same speed as what you would get if you were to run an autoregressive model on custom hardware." 00:17:59

Recursive Self-Improvement Hype Is Overstated — Human Ingenuity Is Still the Bottleneck

Despite working at the frontier of AI research, Ermon is skeptical that AI-accelerated research has yet replaced human judgment, a position that cuts against the "AI is about to automate AI research" narrative common in the Valley. "Right now, the human ingenuity is still like super important. And the ability to come up with the right ideas and kind of like prune the space... has been really important to us." 00:33:42 He adds the bottleneck today is compute, not people: "Often I feel like the bottleneck is more compute than people." 00:34:22

Open Sourcing Everything Is Not Obviously Correct

Against the default expectation (especially from academic-rooted founders) that new architecture research should be open, Ermon deliberately withholds. "We have to build our own stack for doing SFT, for doing RLHF, doing RL. I mean, that becomes IP to some extent. So it's one of the reasons we decided not to open source everything." 00:31:14

3. Companies Identified

Inception — Company building commercial-scale diffusion-based large language models (branded "Mercury"), founded by Stefano Ermon. Mentioned as the subject of the entire episode; roughly two years old, ~50 people, serving production customers today with proprietary serving infrastructure built from scratch since no open-source tooling exists for diffusion LLMs. "We've been able to train diffusion-based LLMs that are comparable in quality with the speed-optimized models for Frontier Labs... while being significantly faster." 00:12:23

OpenCall — Voice AI agent company, named as a public customer of Inception. Notable because they switched off custom AI silicon (Cerebras) onto Inception's diffusion models running on standard NVIDIA GPUs, getting equivalent speed at lower cost with better availability. "They were previously using, serving their LLMs on Cerebras... And then they switched over to our diffusion-based LLMs because they can essentially get the same speed as what you would get if you were to run an autoregressive model on custom hardware." 00:17:59

Cerebras — Custom AI chip maker, mentioned as the prior (now displaced) infrastructure choice for OpenCall, illustrating that software-level parallelism can substitute for specialized hardware in latency-sensitive use cases.

Open Router — Mentioned as a useful analytical resource for understanding AI workload taxonomy across use cases (research, conversational, coding, log processing), which Ermon used to estimate the addressable market for latency-sensitive applications. "Open Router has this very nice way of kind of like looking at all the different use cases... they have like a nice hard taxonomy basically of tasks." 00:29:19

Stability AI / Midjourney (referenced generically as "stable diffusion mid-journey") — Cited as downstream commercial successes that grew directly out of Ermon's academic diffusion research. "All of that started from ideas that were developed in academia in my lab. But that's not the only one." 00:36:15

4. People Identified

Stefano Ermon — Co-founder/CEO of Inception, former Stanford professor, one of the original researchers behind score-based/diffusion generative models. Identified by the host as "one of the fathers of Diffusion." His research lineage includes originating diffusion models, contributing to Flash Attention, and DPO (Direct Preference Optimization) — all with outsized industry impact originating from academic work. "I worked on flash attention, for example. That's another thing that came out from academia that then eventually had a huge impact in industry... [DPO] is an algorithm that is used to align... LLMs and diffusion models and everywhere." 00:36:15/00:36:42

Yang Song — Ermon's PhD student, co-creator of score-based generative models, the technical precursor to modern diffusion models. "We kind of like came up with this idea of let's train a neural network to denoise images... And that basically became the kind of like underlying technology of modern diffusion models... with my PhD student, Yang Song." 00:03:40/00:03:40

5. Operating Insights

Force Yourself Into Production Early to Generate Proprietary Infrastructure IP

Rather than staying in research mode, Inception deliberately built and shipped a commercial product early specifically because the act of serving real customers forces creation of durable, hard-to-replicate infrastructure (serving engines, eval sets, data pipelines) that pure research would never produce. "By forcing ourselves from the very beginning to go out and deploy something end to end, we're learning a lot about how to serve these models and how to build software that it's kind of like needed to run these models. And that again becomes IP." 00:20:52

Use Customer Feedback Loops as a Second, Compounding Moat Layer Beyond the Core Model

Beyond infra, Inception treats real-world deployment feedback itself as defensible: "We are working with real customers and we're getting feedback on the models. We figure out what works, what doesn't. We collect data sometimes from them. We create evals based on what they're seeing. And so that again becomes part of the technical moat." 00:21:20

Preserve Backward Compatibility to Eliminate Adoption Friction for a Fundamentally New Architecture

Despite building an entirely different model architecture, Inception kept the exact same API surface as incumbent providers, which let customers switch with zero integration cost — a tactic any infra startup competing against entrenched standards should note. "We've built everything to be backwards compatible. So it's still like the API is the same. It's still OpenAI compatible text in text out." 00:25:12

Resource-Constrained Teams Should Borrow Aggressively From the Existing Stack and Concentrate Differentiation on One Axis

Inception explicitly chose not to reinvent architecture components that were already solved (attention, public datasets, benchmarks) and put all their scarce research effort into the one dimension where they could win. "We're a startup. We try to be scrappy. We try to use existing things as much as possible and kind of like focus on the aspects where we can have the highest impact and then where we can be the most differentiated. And right now it's speed." 00:14:56

6. Overlooked Insights

The RL Post-Training Bottleneck Argument Is a Much Bigger Deal Than It Sounds

Buried in a broader architecture discussion, Ermon makes a claim that quietly reframes the economics of the entire current RL/reasoning-model paradigm: since RL post-training requires generating and scoring many rollouts, and rollout generation is bottlenecked by inference speed, any model that generates faster at inference time compounds into meaningfully cheaper/faster RL training loops — meaning diffusion's speed advantage isn't just a user-facing latency perk, it could become a training-side cost advantage that compounds against autoregressive competitors over time as reasoning/RL scaling becomes more central to frontier lab strategy. "Even if you think about RL post-training, a lot of the bottleneck is generating rollouts... inference is again a key bottleneck for RL post-training. And so if you have a model that scales better at inference time, then automatically you're going to get better scaling during RL training." 00:10:33 Neither speaker returns to unpack the magnitude of this, but it implies diffusion architectures could eventually out-compete on the training side too, not just inference — undermining the common assumption that architecture choice only affects the serving/deployment layer.

Diffusion's Steerability Could Enable Entirely New Product Interfaces That Don't Exist Yet

Mentioned almost as an aside in response to a question about voice alignment, Ermon reveals that diffusion models allow real-time steering toward a reward function or constraint during generation (not just after), which is categorically impossible with autoregressive models. This is understated in the conversation but implies a whole category of products — real-time constrained generation, live-steerable outputs, mid-generation correction — that literally cannot be built on top of GPT-style models today. "There is a lot of evidence in the academic literature, at least, that diffusion models are easier to control and there are different ways of steering them that are just not possible with autoregressive models... What's the right product experience that we can build around the new capabilities that are just not provided by autoregressive models?" 00:26:42/00:27:12 This is a potential new product category, not just a performance optimization, and it passed by with almost no follow-up in the conversation.