148. 对游凯超3小时访谈：开源Infra、和模型Co-design 、“如果vLLM失败，我们会后悔一辈子”

1. Key Themes

The Hardware Lottery and Why Algorithms Must Chase Hardware

You Kaizhao argues that the Moore's Law era — where software performance improved automatically with hardware improvements — is over. We are now in a specialized computing era where algorithms that don't exploit specific hardware primitives simply get left behind. He calls this the "hardware lottery."

"If your algorithm doesn't utilize these specialized computing resources, you haven't won the hardware lottery — you can't enjoy the dividends of this era. For example, Hinton strongly promoted Capsule Networks, which conceptually made great sense, but that thing was GPU-unfriendly, so it never got widespread adoption." 01:55:49

Model-Infra Co-Design Is Now Mandatory, Not Optional

You describes a world where model teams and infrastructure teams that operate independently have "no future." The analogy he uses is hydroelectric power: a generator must be designed around local water conditions to be efficient.

"The analogy translated to large model design is: infra and algorithms must be tightly integrated, they must be co-designed. If the model team and infra team each do their own thing, I think that team has absolutely no future." 02:10:41

The Open-Source Project Lifecycle Requires a Company Behind It

Drawing on Berkeley's history — Spark→Databricks, Kubernetes→Google, PyTorch→Meta — You argues that without a dedicated company, even critical open-source infrastructure dies. He cites OpenSSH nearly collapsing before a foundation was formed.

"Ion Stoica said, if vLLM doesn't have a company to support it, it will definitely die. He said it's like Linux — if Linux didn't have Red Hat's support, Linux wouldn't have developed to where it is today." 01:07:55

DeepSeek's Infra Team Is Globally Elite, and Their Strength Comes From Quantization Origins

You attributes DeepSeek's exceptional infra capability to their history building quantization systems from scratch, which forced full-stack hardware control. He views their self-built datacenter model as both a source of strength (control) and limitation (hard to scale rapidly).

"DeepSeek's deep infra foundation probably comes from their previous work on quantization — the extreme squeezing of performance at that time. Because they did quantization and needed to build their own data centers, they have deep control over every detail of the machine, so they can do full-stack optimization." 01:59:58

Tokens Are Not Fungible Like Electricity — They Carry a Model's Imprint

You introduces a critical distinction from the popular "tokens = electricity" analogy. Unlike electricity, tokens cannot be transformed between models — they carry the identity of the model that generated them, making standardization fundamentally impossible.

"You have no way to convert a token from a DeepSeek model into a token from a Kimi model. This token carries the model's imprint." 02:02:06

The Open-Source Community Is Being Disrupted by AI Coding Agents Creating "AI Slop"

vLLM is now the most active project on all of GitHub, but this creates a new governance problem: training institutions are submitting meaningless AI-generated pull requests to embellish student resumes, shattering the assumption that contributors are acting in good faith.

"We discovered that some training institutions were submitting junk code to package students' resumes. They submit a useless PR hoping to get it merged into vLLM, so they can put on their resume 'I am a vLLM contributor.'" 01:25:39

Open-Source Models Will Eventually Win Because Model Capability Cannot Be Hidden Long-Term

Unlike nuclear technology (which only needs to demonstrate capability once and can then be kept secret), AI models must be deployed at mass scale to generate value — and mass deployment creates enough data for others to replicate the capability.

"The model is not something that can be hidden long-term. Once you've given it to people to use for a long time, users' own data can already train a model. So the model is not something that can be kept hidden from users indefinitely." 02:42:09

First Principles Thinking Is the Scarce Resource, Not Coding Skill

You argues that in a fast-moving field flooded with practitioners, the ability to reason from first principles — rather than trial-and-error pattern matching — is the primary differentiator. He illustrates this by how he found the "CUDA CoreDump" debugging technique that the entire AI infra community had overlooked.

"I thought: this programming model should have such a tool. The GPU driver should be able to support this kind of task. So I kept asking NVIDIA's team, 'You should have such a solution, find it for me.' And they did find it — the solution had already existed, it was just unknown." 02:35:23

The Founding Team's Permanence Is the Critical Variable in Open-Source Success

You states plainly that when core founding teams leave open-source projects, those projects rarely recover. This is the lens through which he views OpenCSS (now largely AI-governed) skeptically.

"I think after a leadership change, it's basically very hard for a project to develop well." 01:29:48

The Coming Co-Design Stack: Harness → Infra → Model → Hardware Must All Be Designed Together

You extends co-design beyond model+infra to include the agent/harness layer. He gives concrete examples of poor harness design (embedding live timestamps in system prompts, scheduling all cron jobs on the hour) that create massive, avoidable inference load spikes.

"A group of crayfish all launch simultaneously on the hour to attack the moon. These things are actually very simply optimizable — if your hourly task doesn't require the exact top of the hour, you can randomly pick a time within the hour." 02:27:53

---

2. Contrarian Perspectives

The Entire Academic ML Paper Ecosystem Collapsed Into a Lottery System

Most people treat top ML conference publications as signal of quality. You argues publication became essentially random as submission volume grew geometrically while reviewer quality collapsed — making paper acceptance a poor proxy for work quality.

"At this point, the quality of reviews dropped sharply. Your submission was like drawing a lottery — whether you got in or not may no longer have much to do with the quality of the work itself. I witnessed this process firsthand." 00:09:23

Context Windows Beyond 1 Million Tokens Are Largely Unnecessary for Human-Facing Models

While the industry races toward longer contexts, You bets that 1M tokens is sufficient for models interacting with humans, and that long-horizon needs can be addressed through external memory modules, skill modules, or subagents — not ever-longer native context.

"My hot take is that models' context needs will basically plateau at the million-token level... Models handling long tasks or lifelong learning needs can all be achieved through external tools — memory modules, skill modules, subagent modules." 02:36:21

Elon Musk Doesn't Actually Understand AI Infra — He Understands Infrastructure-of-Infrastructure

When asked about Musk being known for infra, You draws a sharp distinction: Musk builds clusters (power, land, physical infrastructure), which is a capital allocation problem, not an AI infra engineering problem.

"That we might call it 'infra's infra' — how to build clusters, how to light up the machines, how to hand them over to the AI infra team. I don't think there's much technical depth there; a lot of it is: how much money do you have, how much land can you find, how much electricity can you find." 02:07:03

Individual Code Contributors Are Becoming Obsolete in Open-Source — Future Communities Will Split Into Maintainers and Users Only

Counter to the prevailing view that open-source communities get stronger as contribution tools improve, You argues that AI coding agents have made code so cheap that maintainers' time is better spent writing code themselves than reviewing others' PRs.

"With coding agents, it's not worth spending time reviewing others' PRs — I might as well rewrite it myself. I have more context about how I want the coding agent to write it." 01:26:36

The Right Model for Open-Source Governance Is a Startup, Not a Foundation or Big Tech

Both foundations (insufficient technical authority) and large companies (inherent bias, competitive dynamics) fail as long-term stewards of critical open-source projects. Only a startup founded by the original creators has both the technical credibility and the neutrality to coordinate competing stakeholders.

"If it's a large company, it's very easy to bring their original biases or positions, making it hard to integrate the community. Other large companies won't follow them... A startup that is the project's founding team, having gone through wind and rain together with the community — their convening power, their neutrality and fairness, everyone recognizes." 01:33:27

---

3. Companies Identified

DeepSeek Description: Chinese AI lab known for open-source frontier models including DeepSeek V3 and R1. Why mentioned: Cited as having the world's top-tier AI infra team; pioneer of FP8 training, MoE at scale, speculative decoding (DSpRek), and native sparse attention. You spent extended discussion analyzing the sources of their infra excellence.

"DeepSeek's infra team — I'd say it's globally top-tier. Whether they're number one I can't say, but absolutely top-tier." 01:59:34

Databricks Description: Data and AI company founded by Ion Stoica, the commercial entity behind Apache Spark. Why mentioned: Used as the canonical example of how an open-source project (Spark) requires a dedicated startup (Databricks) to reach its potential. Ion Stoica's experience founding Databricks directly informed his push to have the vLLM team start a company.

"Spark needed Databricks' support. The vLLM project has already reached the scale of the previous projects; its historical significance is already very clear." 01:08:24

Red Hat Description: Enterprise Linux company, subsidiary of IBM. Why mentioned: Active co-maintainer of vLLM within the PyTorch Foundation governance structure; seen as a direct parallel to how Red Hat supported Linux. Red Hat has already bundled vLLM into enterprise Linux distributions.

"Red Hat is investing in vLLM. Why? Because it thinks vLLM is the next Linux. It has already packaged vLLM into many of its enterprise Linux distributions." 02:45:57

Thinking Machine Labs Description: AI startup with rapidly rising valuation. Why mentioned: Wu Suk (Page Attention co-creator, vLLM co-founder) was recruited to lead its infrastructure — illustrating the magnitude of opportunities he passed up to join Infract.

"When Thinking Machine Labs was founded, the founders invited him to lead infrastructure. Later, Thinking Machine Labs' valuation kept soaring." 01:15:07

Character AI Description: Consumer AI chatbot company. Why mentioned: Simon (CEO of Infract) worked there prior to returning to his PhD and joining vLLM, bringing startup experience and open-source community expertise to the founding team.

"Simon had also spent some time at Character AI, so he tried through Character AI to help support vLLM's development." 01:13:53

Qwen (Alibaba/通义千问) Description: Alibaba's open-source large model series. Why mentioned: Cited as an example of a Chinese model that vLLM supports; Qwen 2.5 Coder specifically praised for excelling in coding capability, likely due to superior data flywheel.

"Qwen is doing very well — their Qwen 2.5 Coder's coding capability is very good." 02:42:49

MiniMax Description: Chinese AI company. Why mentioned: Named as one of the significant recent model releases (MiniMax M3) that vLLM has been actively supporting in 2025.

"This year's DeepSeek V4, MiniMax M3 are both relatively large releases." 01:00:32

PyTorch Foundation (Linux Foundation) Description: Foundation that stewards the PyTorch ecosystem. Why mentioned: vLLM was donated to the foundation as a legal guarantee of perpetual open-source status; the foundation holds the vLLM trademark. You explains carefully what the foundation does and does not do.

"vLLM's trademark is now owned by the PyTorch Foundation — you can't close-source vLLM and then claim to support the vLLM project." 01:05:03

NVIDIA Description: Dominant GPU and AI accelerator company. Why mentioned: Co-published a technical blog with vLLM on Draft-Flash speculative decoding support; active co-maintainer of vLLM within the governance structure; cited extensively for hardware architecture decisions.

"vLLM and NVIDIA recently just published a technical blog introducing vLLM's support for Draft-Flash, which allows certain models to reach speeds of over a thousand tokens per second." 02:04:12

AMD Description: GPU and chip manufacturer. Why mentioned: Active contributor to vLLM governance alongside NVIDIA and cloud providers; example of how large chip companies are co-maintainers but cannot make strategic decisions for the project.

Tencent (Hunyuan team) Description: Chinese tech conglomerate, AI research team. Why mentioned: Their Hunyuan team published DCut, a speculative decoding work solving the same acceptance-rate problem as DSpRek, predating DeepSeek's announcement.

"Before DSpRek was proposed, Tencent's Hunyuan team had already published a DCut work, and researchers from Shanghai Jiao Tong also proposed something called Domino — all solving this problem through similar ideas." 02:05:09

OpenCSS / OpenClaude (likely OpenClaude → likely refers to a different open-source project; transcript appears to reference "OpenClaude" which in context means an open-source project that has transitioned to fully AI-governed contribution) Description: Open-source project referenced as an example. Why mentioned: Described as having transitioned to a fully AI-governed community where AI submits, reviews, and merges PRs — used as a cautionary example of losing human judgment in open-source governance.

"It seems to have already become purely AI-governed — AI submits PRs, AI reviews, AI merges. It's already become a self-governing AI community." 01:28:10

LM Arena (formerly LMSYS Chatbot Arena) Description: Leaderboard and evaluation platform for LLMs. Why mentioned: Ion Stoica is involved as a founder; cited as an example of his continued engagement in the AI ecosystem.

---

4. People Identified

Ion Stoica (Yang Stoica) Description: UC Berkeley professor, co-founder of Databricks, founder of the RISELab/SkyLab lineage, vLLM advisor and co-founder of Infract. Why mentioned: The intellectual and strategic godfather of the entire vLLM venture. His insistence that vLLM needed a company, his encouragement of You as an undergrad, and his lab's tradition of spinning out foundational open-source infrastructure (Spark, Ray, vLLM) are all central to the story.

"Ion Stoica told me: based on his experience as someone who's been through it all, in the long run you should definitely continue supporting vLLM." 01:12:06 "As long as you build good software, someone will always use it." 01:21:52

Li Zhuohan (李卓翰) Description: Co-author of the PagedAttention paper, one of vLLM's original creators; now at OpenAI then Meta TBD. Why mentioned: The originator of vLLM's core algorithm; his early departure to industry (OpenAI, then Meta TBD) after graduating in 2024 before the company was formed is a key plot point.

"Zhuohan graduated early — in 2024. At that time if he started a company he'd be alone, because the other students were students, very inexpensive — just needed food, housing, a computer and internet to work." 01:18:14

Wu Suk (乌苏克, likely Woosuk Kwon) Description: PagedAttention paper co-author, vLLM co-founder, CTO of Infract. Why mentioned: The hardest person to recruit to the founding team — had offers from XAI, Thinking Machine Labs, and Google DeepMind. His eventual commitment was the pivotal moment for the company formation.

"We asked him: if you don't co-found with us, you'll earn a lot of money, but ten years later if the vLLM project fails, will you be happy or not?" 01:15:37

Simon (CEO of Infract) Description: PhD student at UC Berkeley, serial startup founder, CEO of Infract. Why mentioned: Described as the natural CEO given his prior startup experience (including at Character AI) and demonstrated community management skills. Was the third core full-time contributor after Zhuohan and Wu Suk.

"He had already participated in some startups. He had experience from Character AI. He also agreed with Ion Stoica's view that vLLM must have a company to support it for long-term development." 01:13:23

David Patterson Description: Turing Award-winning computer architect at UC Berkeley, co-creator of RISC, RAID; wrote "How to Build a Bad Research Center." Why mentioned: His thesis that the post-Moore's Law era requires domain-specific hardware, and that matrix multiplication is the foundational primitive worth specializing for, is cited as the intellectual foundation for why systems matter more than algorithms today.

"David Patterson wrote a paper called 'How to Build a Bad Research Center' saying each lab should run for only five years — exactly one PhD cycle." 01:39:44

He Kaiming (何恺明 / Kaiming He) Description: Researcher at Meta AI, creator of ResNet and influential computer vision algorithms. Why mentioned: Used as the archetype of a "one algorithm = permanent legend" researcher, and as evidence that algorithmic research at scale requires institutional resources that most researchers don't have.

"We'd read Kaiming's paper and think the idea was so natural — almost as if it should have always been designed this way. Yet we had similar ideas months earlier in lab meetings." 01:10:50

Michael Jordan Description: UC Berkeley professor, renowned for machine learning theory and statistics; You's host during his 2019 visiting research. Why mentioned: One of two mentor figures who shaped You's direction; advised him (while completing his exchange) to continue supporting vLLM rather than pursuing lucrative industry offers. Distinguished from Ion Stoica as more theory-oriented vs. systems/impact-oriented.

"From his perspective, based on his experience as someone who has been through it all, long-term you should definitely continue supporting vLLM." 01:12:06

Wen Jiayi (翁嘉义 / Jiayi Weng) Description: You's Tsinghua undergraduate classmate; now an OpenAI trainer (credited on ChatGPT). Why mentioned: Co-built the Tianshu reinforcement learning open-source project with You in 2020 during COVID lockdown; held up as a parallel example of someone who also chose systems/infra over algorithms. You notes he now trains ChatGPT.

"Wen Jiayi is now a ChatGPT trainer — his name is on every OpenAI blog post. He's already been inscribed in history." 01:23:21

Edward Yang Description: PyTorch core maintainer. Why mentioned: Wrote a blog post reaching the same conclusion as You about AI-generated PR pollution: it's now more efficient for maintainers to rewrite submitted code themselves than to review it.

"Edward Yang also wrote a similar blog saying: we no longer have the energy to review everyone's PRs, so rather than spending time reviewing others' PRs, I might as well rewrite it myself." 01:27:59

Song Lin (松林) Description: Researcher, likely referring to the Mamba/linear attention lineage researcher. Why mentioned: Cited for a blog post on designing inference-friendly harness frameworks, specifically around prefix caching; also credited for the "chunk parallel" algorithm enabling efficient linear attention training.

"Last year, Mandos' Jichao — that's Peak — also wrote a blog about how to design an infra-friendly harness framework, with the most important part being fully utilizing prefix caching." 02:26:25

Roger Description: Long-time vLLM community contributor, founding engineer at Infract. Why mentioned: One of the founding engineers who worked on the vLLM project for years before the company formed, embodying the commitment-over-compensation culture.

"Roger has also been working with us on the vLLM project for many years." 01:16:03

---

5. Operating Insights

The "Are You Still Happy?" Framework for Recruiting Mission-Aligned Founders

When logic and financial incentives fail to recruit critical co-founders, You's team used a values-clarification question rather than making a competing offer. The question reframes the decision from present compensation to future regret.

"We asked him: if you don't co-found with us, you earn a lot of money, but ten years later if the vLLM project fails, are you happy or not? We asked this question. He thought about it for a long time. In the end he found his answer was the same as ours." 01:15:37

Paper-First vs. System-First: Publish After Adoption, Not Before

You describes inverting the traditional academic sequence: build a system people actually use, let adoption accumulate, and only then write the paper. Reviewers are often users of the system and won't reject work with proven real-world impact.

"When we write it into a paper — list the valuable things inside — the reviewers at that point generally won't say anything. They're actually users of our system themselves. So the submission process goes quite smoothly." 00:35:46

Prioritize Deleting Over Adding — Reduce Complexity Before It Becomes Unmanageable

With 200+ model architectures supported and 2,000+ contributors, You's most important managerial act is aggressive removal: deprecated features, inactive models, low-signal PRs. He treats deletion as a first-class strategic tool.

"We also need to continuously remove unnecessary complexity. So as I mentioned, we delete some models that are no longer used, delete some unnecessary features. In this process we also increasingly focus on what are the major models people are currently using." 02:20:13

Design Agent Harnesses Around Prefix Stability — Never Put Time-Varying Data at the Start of a Request

A concrete, immediately actionable optimization: placing dynamic data (timestamps, session IDs) at the start of system prompts destroys prefix cache hit rates. This single mistake can cause catastrophic inference costs in production agentic systems.

"Some harness frameworks embed the current timestamp into requests, precise to the second, and their cron jobs are always set to run on the hour — causing the inference system to receive a massive wave of traffic right on the hour, creating enormous pressure." 02:27:24

Build for "Supply-Side" Impact: Measure by Tokens Produced and Cost Saved, Not Engineer Hours Sold

You explicitly rejects the "technology outsourcing" model of charging for engineer time. The correct metric for an infrastructure business is value delivered through throughput and cost reduction — a per-unit economics model that scales independently of headcount.

"We are not technology outsourcing, we are not selling our engineers' time. The value we create depends on how many more tokens we produce for you, how much cost we save for your machines. This is our overall logic." 01:50:26

---

6. Overlooked Insights

VCs Were Funding the Open-Source Project Directly for Two Years Before the Company Existed — As an Options Strategy

This detail was mentioned almost in passing, but it is a significant structural insight about how top-tier VCs now operate. Andreessen Horowitz (a16z), Sequoia, and Zhenfund were donating capital to vLLM as an unincorporated open-source project in 2023-2024 — with no equity, no board seats, and no legal entity on the other side. This was not philanthropy; it was a pre-commitment strategy to earn the right to lead the round when the founding team eventually decided to form a company.

"Like a16z — they provided a grant to the vLLM project back in 2023. In 2024, Sequoia and Zhenfund also donated money to us in the form of an open-source community. When we finally told them we had decided to start a company, they were quite straightforwardly ready — 'My money is already prepared. You've finally decided.'" 01:39:24

The implication for investors and founders: for genuinely critical open-source infrastructure, the right move is to fund the community before incorporation as a way to establish trust, information advantage, and allocation priority — not wait for a formal fundraise. This pre-seed-before-seed structure is likely to become more common as the next layer of AI infrastructure emerges from academic labs.

The "Top Executive Called to Offer $20M Per Founder Annually" Story Validates That Incumbents View vLLM as Existential

You mentions almost as an aside that a "number-one executive at a top-tier major tech company" called the four co-founders just before incorporation and offered each of them $20 million USD annually in salary to abandon the startup and join the company instead. This detail is significant not for what it says about the founders' character (though it does), but for what it reveals about how large technology companies valued having vLLM's team inside rather than independent. A $20M/year offer per person implies an implicit valuation of the team's ability to influence the trajectory of inference infrastructure that dwarfs the $150M seed round they ultimately raised.

"A number-one executive at a top-tier major tech company called us directly and said, 'Can you not start a company? Come to our company — we can give each of your four founders an annual salary of $20 million USD. Can you come work on our infrastructure and inference?'" 01:16:59