孙宇涛 | 张小珺Jùn｜商业访谈录 Summary

1. Key Themes

The Architecture of Kimi K3 Is a Synthesis, Not a Single Breakthrough

Sun Yutao frames K3 not as one revolutionary idea but as a disciplined integration of the best available techniques from across the research community. The model combines Kimi's own prior work (Kimi Linear Attention / KDA), community innovations like GatedDeltaNet, MLA from DeepSeek, and architectural ideas from DenseNet and HyperConnection.

"做一个公司比较科学的方式，就是如果你最后想把这个模型做好，你肯定是集百家之长。" ("The right way for a company to build a good model is definitely to draw on the strengths of hundreds of others.") 00:18:11

Scale Remains the Most Fundamental Driver of Model Intelligence

Sun repeatedly anchors the conversation to a first-principles claim: model parameter size is still the primary determinant of intelligence ceiling. K3's jump from ~1T (K2) to 2.8T total parameters with ~100B activated is framed as a non-technical but strategically decisive bet.

"更大的模型才有更大的智能上限...模型的参数大小还是解决模型智能最本质的参数量。" ("Larger models have larger intelligence ceilings... model parameter size is still the most fundamental parameter for solving model intelligence.") 00:20:31

"因为有size它才有智能，所以说K3你把激活多了三倍，把模型也快大三倍，它就是下一个level。" ("Because size gives you intelligence, K3 tripled its activations, almost tripled the model — it's just the next level.") 01:59:43

Hybrid Attention Is Not a Trade-Off — It Is a Free Lunch

Sun argues that the community has empirically validated that hybrid attention (linear + full attention interleaved at ~3:1 ratio) does not sacrifice model capability. This is presented as an underappreciated non-obvious finding.

"混合注意力，它其实虽然从架构上来讲是一种trade off，但是从最后模型本身的表现来说，并不是一个trade off。包括大家其实发现在保持一定的全助力力比例的基础上，整个模型其实是可以获得无损甚至更好的一个常长上下文表现的。" ("Hybrid attention, while architecturally a trade-off, is in terms of model performance not a trade-off at all. With a sufficient ratio of full attention, the model can achieve lossless or even better long-context performance.") 00:07:16

The KDA (Kimi Delta Attention) Lineage: Channel-Wise Decay as Key Innovation

The evolution from RetNet → Mamba → GatedDeltaNet → Kimi Delta Attention is traced in detail. The key innovation in KDA is upgrading the scalar decay coefficient to a channel-wise decay, giving the model strictly greater expressive power — at the cost of harder kernel engineering.

"KimiDelta Attention它其实就是把这个Gatey的DeltaNet这个衰减项，然后从一个标量，然后变成了一个更fine-grained control这样的一个量，然后这样的话它就可以带来就是每个channel可以的衰减系数是不一样的，这样的话我们这样可以带来一个严格更好的一个能力的上限。" ("Kimi Delta Attention changes the decay term of Gated DeltaNet from a scalar into a more fine-grained controlled quantity, so each channel can have a different decay coefficient — this gives a strictly better capability ceiling.") 00:28:48

Latent MoE Is the Key Infra Innovation Enabling Communication-Overlap Strategy

Sun explains that Latent MoE (compressing the dispatched hidden state before Expert routing) is not just an algorithmic improvement — it structurally changes what infra optimizations are feasible. Reduced communication volume means simpler, more reliable overlap strategies.

"因为latent moe它本身是极大的降低了通信开销的，所以说它在整个的计算开销里边，它其实是减小了特别多的...所以其实如果想把它overlap掉，就会变得容易很多。" ("Latent MoE itself dramatically reduces communication overhead... so overlapping it becomes much easier.") 01:44:19

"我用不一样的moe的架构，然后带来了不一样的overlap的一个策略，然后这个策略它带来一个好处就是推理的critical path的latency它是免费的。" ("By using a different MoE architecture, I bring a different overlap strategy, and the benefit is that the inference critical path latency is free.") 01:44:48

NoPE (No Positional Encoding) on Full Attention Layers Is an Elegant Long-Context Solution

In hybrid attention models, positional information is already injected via the linear attention layers (with decay terms). This makes RoPE on full attention layers redundant or even harmful for long-context. Removing it (NoPE) simplifies long-context scaling dramatically.

"如果在这个Hybrid Attention的架构里，如果我们可以把这个Full Attention从Rope改成Nope，也就是说在全注意力的情况下把位置编码去掉，它其实是可以带来一个更好的...它在这个长上下文是更容易扩展的，也就是说Nope它是在模型扩长的情况下，不需要调整任何模型架构的参数，就可以达到一个长上下文的这样的一个效果。" ("In hybrid attention, replacing RoPE with NoPE on full attention layers can bring better performance... it's easier to extend to long context — NoPE requires no architectural parameter tuning to achieve long-context capability.") 01:20:13

Quantile Balancing as a More Principled Replacement for Loss-Free MoE Routing

K3 introduces Quantile Balancing as a mathematically principled alternative to the heuristic bias-update in Loss-Free routing (DeepSeek V3). The key insight is computing a per-step quantile of expert activations to derive a bias, using histogram bucketing for scalability.

"Quantile Balancing，其实是主要是尝试用一个更principle的一个思想，然后来去解决模型推理的一个模型Moe附载稳定性这样的一个问题。" ("Quantile Balancing tries to use a more principled approach to solve MoE load balancing stability.") 00:57:55

MTP (Multi-Token Prediction) and Draft Models Must Be Designed Into Pretraining, Not Bolted On

Sun's conversation with the author of DFlash (Chen Jian) surfaces a non-obvious point: the effectiveness of speculative decoding / draft models is heavily gated by how well the pretraining stage provides an MTP interface.

"对于Draft Model来说，我们它是需要利用大模型的一定的推理能力的基础上，然后再去使用Draft Model进一步的去扩大...对于MTP来说，MTP是需要给在Pre-Train阶段需要给MTP提供一个接口，然后这个接口会对下游的Draft Model的提升会特别王主。" ("The draft model needs the large model's inference capability as a foundation... MTP requires an interface built into pretraining; that interface is critically important to downstream draft model improvement.") 01:32:33

Cosine Decay over WSD: K3's Contrarian Learning Rate Choice

Most frontier labs adopted WSD (Warmup-Stable-Decay) schedulers after MiniCPM. K3 reverts to cosine decay, arguing it is easier to tune (fewer hyperparameters), and that WSD's claimed advantage of decoupling training token count from learning rate is less compelling than advertised.

"Cosine Decay它的方案就是它其实更好调，就它只有两个变量，一个就是你要跑多少Token，然后你要一个就是你的那个Maximal Learning Rate具体设多少。" ("Cosine Decay is easier to tune — it only has two variables: how many tokens to run, and what the maximum learning rate is.") 01:16:47

2. Contrarian Perspectives

Sparse Attention Is Not Worth It on Modern Hardware (Blackwell)

Sun argues that Sparse Attention, widely considered a natural next step in efficiency, actually provides negligible decode speedup on Blackwell because the index-gathering step is too expensive. This directly contradicts the apparent momentum behind sparse attention adoption.

"每层都用Sparcer Attention这样一个比较简易的方案，它其实在Blackwell上，它的收益是特别特别小的...甚至其实在Decode当中，这种Sparcer Attention的一个方案，它其实甚至跟这个Full Attention并不会带来一个显著的一个加速。" ("Using sparse attention naively on every layer has very little benefit on Blackwell... in decode, sparse attention may not even provide significant speedup over full attention.") 01:04:44

RoPE Hurts Long-Context Performance — It Was Never Designed for It

The conventional wisdom is that RoPE enables positional awareness and thus helps long-context. Sun argues the opposite: RoPE only helps short-context via recency bias and is actually harmful for long-context. Removing it from full attention layers in hybrid architectures is strictly better.

"Rope它并不带来任何的长文能力，它甚至是损害长文能力。" ("RoPE does not bring any long-context capability — it actually damages long-context capability.") 01:21:41

MLA Is Not a Fundamental Innovation — It's a Better-Tuned MQA

Sun deflates the significance of MLA (Multi-head Latent Attention), which was credited as a major DeepSeek V2/V3 architectural contribution. He argues MLA is essentially MQA with better parameterization, and that its compute-memory tradeoffs are increasingly unfavorable in agentic settings.

"MLA它本身就是MQA的一个增加的一个形式，所以说MLA其实本质上并没有带来新的东西，它只是一个更好的水道。" ("MLA is essentially an augmented form of MQA. It doesn't fundamentally bring anything new — it's just a better channel.") 00:36:27

There Will Be No More Fundamental LLM Architecture Innovations

Sun makes a bold claim that the era of architectural breakthroughs in large language models is essentially over. Future progress will be incremental refinement, not paradigm shifts.

"大模型可能没有太本质的创新，后面都是一些改样性的进行进步。" ("Large language models probably have no more fundamental innovations ahead — what follows are all incremental improvements.") 02:01:29

Training Stability Matters More Than Peak Capability in Architectural Choices

Sun argues that for any Pretrain run, training stability should be weighted higher than raw model performance when choosing architectural components. An unstable run that collapses wastes everything.

"从模型能力上如果不是最优，但也没有关系对吧，因为模型的能力其实本质上最主要的因素是模型的参数大小。所以说好一点，差一点，它都不如稳定性要重要，因为如果稳定性不好的话，它会带来很多麻烦的问题，甚至比较大的问题，就是你整个Run都迅崩了，然后你前面整个的Triple Point都废掉了。" ("If model capability is not optimal, it doesn't matter — because the main factor is parameter size. Better or worse by a little is less important than stability. If stability is bad, the entire run could collapse and all prior checkpoints are wasted.") 00:34:41

3. Companies Identified

Moonshot AI (Kimi)

China-based AI lab, developer of the Kimi model family. Mentioned for the K3 model: 2.8T total parameters, ~100B activated, native 1M context, full open-source release, and a technically rigorous approach to model internals (model autopsy culture).

"Kimi K3它其实在2.8T这个Scaling做了一个有效的Scaling，然后这个模型参数大小，包括模型的一个激活，它来到了100B左右这样的一个量级。" ("Kimi K3 did effective scaling at 2.8T, with activations reaching around 100B.") 00:19:32

DeepSeek

China-based AI research lab known for DeepSeek V2/V3/V4 model series. Mentioned extensively for architectural contributions including MLA (V2), Loss-Free routing (V3), DualPipe pipeline parallelism (V3), Sparse attention (V4), and MoE design.

"DeepSeek V4，然后包括Safone，那些他们应该都是用了MioA，然后在MioA的基础上，因为他们又都是用的GQA，没有用MioA，所以直接用QKNORM，它其实是本质上是一个更简单的办法。" ("DeepSeek V4 and others use Mamba-style optimizers; since they use GQA not MLA, they can use QKNorm directly, which is fundamentally simpler.") 00:55:58

MiniCPM (Tsinghua / ModelBest)

Small model research group, credited with originating the WSD (Warmup-Stable-Decay) learning rate schedule and the insight to pack higher-quality data into the cooldown phase.

"MiniCPM当时那个Paper提出的WSD...WSD考虑的一个问题就是如果我们如何能把这个Data Schedule和Learning Schedule结合起来。" ("MiniCPM's paper proposed WSD... WSD considers how to combine data scheduling with learning rate scheduling.") 01:12:50

Qwen (Alibaba)

Mentioned for adopting Gated Attention for training stability (Qwen 2.5, 3, 3.5) and for having the largest open-source model prior to K3 (Qwen 72B), which set the previous size benchmark.

"千万的Gatey的Attention，这个千万三应该是没有用，后来的这个千万next包括千万3.5 3.6应该都采用了这样的一个架构，虽然它会带来一定的额外的计算开要，但是它会对模型的稳定性会带来特别大的一个好处。" ("Qwen 3, Qwen Next, 3.5, 3.6 all adopted Gated Attention; it brings extra computation but greatly improves stability.") 00:35:10

Thinking Machine Lab

Credited with the concept of "on-policy distillation" (reverse KL distillation), which has become a standard technique in post-training pipelines at scale.

"Thinking Machine Lab，另一个名字就叫on policy distillation，大家会意识到这个是一个比较有效的从蒸馏的角度是一个比较有效的方式。" ("Thinking Machine Lab — another name is on-policy distillation — people realized this is a very effective form of distillation.") 01:26:07

GLM / Zhipu AI

Mentioned for adopting DeepSeek-style Sparse Attention and for developing Index Cache to solve the expensive index-gathering overhead in sparse attention during decode.

"一个是这个3.2用的，一个是那个GLM5也用了Dipsic Sparse Attention叫这样一个架构...GLM团队做了一个工作叫Index Cache，它当时的做法就是把这个球解Index就是求解哪一部分哪些Token需要算Attention，这样的一个竞争操作，然后把不同层之间共享了起来。" ("GLM5 used sparse attention... the GLM team developed Index Cache, sharing the index-gathering operation across layers to reduce overhead.") 01:04:44

Xiaomi (AI division)

Mentioned for achieving 1000 tokens-per-second inference by combining TiledRT (operator fusion) with DFlash speculative decoding — demonstrating the frontier of inference efficiency for small batch sizes.

"之前小米他们推出了一个能达到一个1000TPS的一个方案，那个主要是用了两个技术，一个就是太阳亮团队，然后他们做的一个可以高度融合不同阶段的一个算子...然后小米是基于太阳亮团队的那个一个结果，然后再加上那个DFlash这个投机推理带来的额外的加速比，最后能带来一个比原模型强很多的这样一个推理速度。" ("Xiaomi achieved 1000 TPS using two technologies: the TiledRT team's highly fused operator, plus DFlash speculative decoding speedup.") 01:29:36

4. People Identified

Su Jianlin (苏建林)

Independent AI researcher, known for his technical blog. Credited with the mathematical proof enabling efficient GPU computation of DeltaNet (chunk-recurrent form), and for the Quantile Balancing blog post. Sun views him as a rigorous first-principles thinker.

"宋林是在DeltaNet的基础上，然后引入了就是开发了一个拖乱东特瑞的，其实就一句话，就是如何能让这个DeltaNet转化为一个GPU可计算的一个Chunk Recurrent这样的一个计算模式。" ("Su Jianlin, building on DeltaNet, developed — in one sentence — how to convert DeltaNet into a GPU-computable Chunk Recurrent computation pattern.") 00:24:53

Luo Fuli (罗弗利)

Mentioned as a practitioner who raised concerns about MLA's compatibility with MTP (Multi-Token Prediction) in agentic settings, and the compute waste MLA introduces in inference.

"罗弗利当时是在VR Flash里面采取的是MTP，他觉得MLA对Chat来说是一个优秀的模型结构，但是他觉得不那么适合Asian的范式...虽然利用打破仿存瓶颈是在这个架构下设计的，不发现计算剩余的实在太多了，他觉得MTP能够更有效的能够利用起来。" ("Luo Fuli uses MTP; he thinks MLA is a good structure for chat but not well-suited for agentic paradigms... there's too much compute left on the table with MLA; MTP can utilize it more effectively.") 00:37:25

Dong Laoshi / Prof. Dong (董老师)

Referenced as the advisor at Zhiyuan Institute (上海创制学院?) under whose supervision the on-policy distillation (OPD) work was done. Also connected to B.ROM, another student who won Tsinghua's top award.

"OPD刚开始也是董老师团队这边比较早去讨论的一个方式...B.ROM，这个是预显做的，预显去年也是清华特奖，这个是他在董老师在艳演员院这边去做的。" ("OPD was first discussed by Prof. Dong's team... B.ROM was done by a student who won Tsinghua's top prize last year, under Prof. Dong.") 01:24:38

Yang Zhilin (杨植麟)

Founder and CEO of Moonshot AI (Kimi). Mentioned for creating an environment where no one is afraid to challenge him, and for the culture of reason-over-hierarchy.

"我觉得Kimi有一个特点是所有人好像都不怕杨志林吧。" / Sun responds: "对啊，为什么要怕呢，有道理你就，谁有道理谁说了算吧。" ("One characteristic of Kimi is that nobody seems to be afraid of Yang Zhilin." / "Right — why would you be? Whoever makes the better argument wins.") 02:01:02

Huang Gao (黄高)

Creator of DenseNet, cited as the precursor work to Attention Residual and HyperConnection. Sun regards DenseNet as among the most intellectually honest and far-sighted architecture papers.

"DeathNet这个是一个更早的一个工作了，就是它是黄高老师在那个Resnet之后，它其实是一个连接方式，DeathNet其实和Attention Residual它其实在思想上一个是一个相当密切的一个方式。" ("DenseNet is an earlier work by Prof. Huang Gao, post-ResNet — its connectivity idea is closely related to Attention Residual.") 00:43:38

He Kaiming (贺凯明, referenced as 凯文)

Credited for the pre-LayerNorm vs post-LayerNorm analysis already present in the original ResNet work (CVPR and ICCV versions), which laid the foundation for modern training stability choices.

"比如在Resnet那个时代，凯文就已经讨论过就是pre-learnum和post-learnum和训练稳定性的这样的一个关系了。" ("Even in the ResNet era, He Kaiming already discussed the relationship between pre-LayerNorm, post-LayerNorm, and training stability.") 00:40:16

Sun Yutao (孙宇涛) — Guest

PhD candidate at Tsinghua Computer Science, Purui Scholar at Shanghai Zhiyuan Institute. Pioneer of chunk-recurrent linear attention, the Yoco (You Only Cache Once) KV cache architecture, and loop language model research.

"然后Yoko当时做了，是一个就是所有层共用一份KVCache这样的一个架构的方式。" ("Yoco implemented an architecture where all layers share a single KV cache.") 00:09:15

Chen Jian (陈建) — DFlash Author

Researcher who created DFlash, a speculative decoding method combining diffusion language model properties with autoregressive models. Sun credits him with the insight that pretraining MTP interfaces are essential for downstream draft model quality.

"我跟陈建，就是DFlash那个同学也交流过，他倒是提了一个点，是对于整个模型是很重要的，即使我们用DFlash这种更先进的Draft Model的策略，它跟预训练的关系仍然是很大的。" ("I discussed with Chen Jian, the DFlash author; he raised a point important to the whole model — even with DFlash's advanced draft model strategy, its relationship to pretraining is still very significant.") 01:32:02

5. Operating Insights

Don't Move Working Infrastructure — Architectural Changes Have a Right Timing

Sun explicitly frames K2→K3 evolution as a lesson in organizational sequencing: K2 prioritized getting the base model pipeline stable; K3 introduced architectural innovations only after that foundation was proven.

"能跑的东西就不要动，如果MLA它不是一个特别大的问题，可能就暂时不会动。" ("If something is running, don't touch it. If MLA isn't a big problem, it probably won't be changed for now.") 00:38:40

"先把Base Model整体的一个Pipeline做稳，在这个稳的基础上的话，然后后来K3再去追求一些更精益求精的一个优化。" ("First stabilize the overall base model pipeline; on that stable foundation, K3 then pursued more refined optimizations.") 01:11:38

Use On-Policy Distillation to Simplify Post-Training Team Management

K3's post-training uses on-policy distillation (student generates, teacher corrects) as an organizational tool: different reward signals or RL tasks can be expressed as different teacher models, avoiding the complexity of combining heterogeneous RL rewards in a single training loop.

"OPD的话它就会变得很简单，我们可以把不同的reward model，或者说不同的RL范式，然后简化成不同的模型...因为对于RL策略或者说对于Data，它是一个高度移构的状态，但是对于模型来说，它是一个高度同构的状态。" ("OPD becomes simple — different reward models or RL paradigms can be simplified into different models... RL policies and data are highly heterogeneous, but models are highly homogeneous.") 01:28:36

Add QAT (Quantization-Aware Training) at the SFT Stage, Not Pretraining

K3's approach to low-precision training: use full precision for pretraining (safer, avoids uncontrolled variables at scale), then introduce FP8/QAT at the SFT stage. Empirically, the timing of QAT introduction doesn't affect final capability if introduced after a sufficient amount of training.

"对于低精度推力而言，本身就是从from scratch刚开始就引入，并不会带来任何模型能力的一个提升...所以在SFT阶段引入，是一个更保险，然后在技术上也不会带来额外损失的一个方式。" ("Introducing low-precision training from scratch doesn't improve model capability... introducing it at the SFT stage is safer and technically incurs no additional loss.") 01:23:39

Two Consecutive Linear Projections Always Need a Normalization Layer Between Them

A general engineering rule stated with emphasis: whenever architecture forces two matrix multiplications in sequence (as in MLA's low-rank projection, or Latent MoE's compression), insert an RMSNorm between them. This is essential for training stability.

"如果你不得不把两个矩阵连程拆开来写，而不适合背到一起的话，中间一定要加一个类似Normalization的一个手段。" ("If you must write two consecutive matrix multiplications separately rather than fusing them, you must add a Normalization-like operation between them.") 00:50:49

6. Overlooked Insights

The Cohere Team's NoPE Paper Deserves Far More Credit Than It Receives

Sun mentions that the core insight behind K3's elegant long-context solution — removing positional encoding from full attention layers in hybrid models — originated from a Cohere team paper. He personally gave it a Strong Accept in peer review (one of very few he's ever given), yet it "didn't go viral." The insight is architecturally profound: the linear attention layers already encode positional information through decay terms, making RoPE on full attention layers not just redundant but actively harmful for long-context. Any team building hybrid attention models for long-context applications should immediately examine this Cohere paper, as it eliminates an entire class of RoPE parameter tuning problems.

"最早是这个Cohere团队的一个工作，主要讲了这样的一个因素...我在会议里边还审到理的片段，当时我直接给了一个Strong Accept，我感觉我这几年可能都没有给过几个Strong Accept，因为我觉得这个确实是一个很有效，也是一个很优雅的解决方案。" ("The original work was from the Cohere team... when I reviewed it, I immediately gave a Strong Accept — I think I've given very few Strong Accepts in recent years, because this is genuinely very effective and elegant.") 01:20:13

The RL Memory Optimization: Reference Model and Gradient Buffer Can Share Memory

Sun briefly mentions a specific engineering trick in K3's RL infrastructure that is easy to miss: the reference model (which is frozen and used for KL-divergence computation) and the gradient buffer share the same memory allocation. The key observation is that these two are never needed simultaneously — by the time gradients are being computed, the reference model's forward pass is complete. This is a non-obvious practical trick that could immediately reduce memory pressure in any team running RLHF/GRPO at scale.

"对于reference model而言，然后他们发现它可以跟gradient buffer来去合起来，因为首先reference model它是没有梯度的，然后在你求objective的过程中你是没有gradient的，等你有了gradient之后，然后你可能reference model就不会去用了。所以说，它这块用了一个gradient buffer use for non-policy model for forwarding。" ("They found the reference model can share memory with the gradient buffer — the reference model has no gradients, and during objective computation there are no gradients; once gradients exist, the reference model is no longer needed. So they use the gradient buffer for reference model forwarding.") 01:51:59