Neil Movva | Invest like the best Summary

Invest Like the Best, EP.488

1. Key Themes

The Background Agent Thesis: Latency Is the Wrong Optimization

Neil's core bet is that the AI industry has been building for the wrong use case. Real-time chatbot inference optimizes for latency, but the future is long-running background agents where latency is irrelevant and cost is everything.

"I want it to be proactive. I want it to be in the background. One way to say it is like the best latency is no latency at all. When you wake up in the morning, the work's already been done overnight. You didn't even have to ask for it. That's the dream." 00:06:46

"I see this going to 90-10 in favor of background." 00:08:29

The GPU Throughput/Latency Tradeoff Is Fundamental and Exploitable

There is a physical, unbreakable tradeoff inside every GPU between throughput and latency. The entire industry has chosen latency optimization because of chatbot demand. Neil is building the only major stack optimized for throughput — treating it as a structural arbitrage.

"Everyone has chosen latency optimization because the shape of usage was chatbot oriented. I believe that's the most profound change we're going to see in the next year. We're going to move away from chatbots to more proactive or background agents. In that world, it makes a lot more sense to build a stack around throughput." 00:20:05

"NVLink is mandatory for low latency inference. So NVIDIA is excellent at low latency inference. And I'm telling you that we don't really care that much about low latency inference." 00:23:30

The Scavenger Strategy: Arbitrage Chips and Power No One Else Wants

SAIL's supply-side strategy is deliberately anti-consensus: buy chips, power, and data center capacity that major labs find illegible or unattractive, extract more performance from them through superior software, and undercut everyone on price.

"We will buy any chip anywhere in the world for any duration of time. That is a level of flexibility and liquidity that no one else has right now. We're very aggressive about putting our money where our mouth is and we will take any capacity and find a way to make it work in our fleet." 00:52:45

"There's no bad chips. There's only bad pricing. I will make any chip work at the right price." 00:45:04

Small, Distributed, Imperfect Data Centers Are the Inference Infrastructure of the Future

The conventional wisdom is that data centers must be large, redundant, and centralized. Neil inverts this: inference workloads can run on geographically dispersed, low-redundancy, even intermittent-power data centers — creating a massive supply pool others won't touch.

"I won't be surprised if some of them get down to like 95% uptime. You'd have basically zero buyers for a data center that has 95% uptime. I'm that first buyer. I will buy 95% uptime." 00:53:38

"A megawatt of compute. You'd imagine this like massive data haul, like a huge warehouse basically. And now you can actually pack that into around like eight racks for the compute. Each rack is about the size of a refrigerator. You can just imagine eight of them lined up. That's a megawatt." 00:51:53

Open Source Intelligence Is Inevitable and Unstoppable

Neil argues that model capability diffusion cannot be stopped, especially because AI-generated artifacts (e.g., code on GitHub) are already latently retraining future models. The premium for being frontier is real but temporary.

"An increasingly large percentage of the artifacts we put out on the internet are AI generated. You just look at GitHub alone. What percentage of repos created in the last year do we think were created by Claude Code? Do we consider that to be distillation? Because it's probably all we need." 01:06:03 (quote continues from 00:06:03)

"I don't think there's a point in specializing... Fundamentally, I don't think open source ever goes away. If there's a vacuum because one leader steps out, a new leader will step in. There's too much incentive and too much tailwinds too. It gets easier every day to train a frontier class model." 00:41:20

Transformers Will Remain Dominant Because They Scale With the Two Things We Can Always Get More Of

Neil gives a nuanced defense of the transformer architecture — not because it's perfect, but because it is uniquely amenable to scaling with compute and data, the two resources that keep growing.

"You increase the compute available to a transformer by 10x and you'll get some log improvement somewhere. And so far, the scaling laws really work. They're really quite beautiful." 00:34:50

"Transformers represent any pairwise relationship that you want. Any token in the sequence can attend to any other token in the sequence. So if there's any relationship that's in the sequence at all, you're going to find it with a transformer." 00:34:50

The Internet Was a One-Time Data Subsidy; RL Gyms Are the Next Data Epoch

The era of learning from human-generated internet text is essentially over. The next phase of AI capability gains will come from self-improvement through reinforcement learning environments on verifiable tasks.

"I like the phrase that internet was a one-time subsidy on data. We got it for free. It's extremely high quality... Models have seen the entire internet many times over at this point. And there is not a whole lot more to be done on human data from the internet." 00:36:24

"The median model that we serve is so much more advanced than like a random human giving feedback that the signal you get from random human preference, unconditioned human preference, is not actually worth anything anymore. You want expert human preference at this point." 00:36:53

KV Cache Is the Most Underappreciated Bottleneck in All of AI Infrastructure

While everyone talks about compute and memory bandwidth, Neil identifies KV cache compression as the single most inefficient part of the modern inference stack — and the most promising area for a step-change improvement.

"Where we are not good is attention and its use of memory specifically. The KV cache is quite uncompressed right now. I think if you look at the entropy in a KV cache, it's not earning its keep. Like we're storing many kilobytes of data in the KV cache per token. And that's probably off by an order of magnitude or two." 01:00:10

Inference Spend Is Non-Speculative; Training Spend Was

This is a structural distinction that matters enormously for predicting chip demand. Training is a bet on future value. Inference is immediate value capture. This changes the risk profile of the whole market.

"People buy tokens because they're immediately valuable to them. You don't hoard tokens, you use them immediately... I do think inference spend monotonically increases. There's no speculation on inference." 00:49:02

2. Contrarian Perspectives

The TSMC Geopolitical Risk Is Vastly Overstated

Everyone in tech treats the loss of TSMC access as a civilization-ending event. Neil thinks the performance gap between TSMC and Western fabs is much smaller than the chip war narrative implies.

"What would happen if we lost access to TSMC for any reason? My contrarian take is that it wouldn't be that bad. Supply would take a shock for sure. But the best processes that we have in the West, like Intel, not that far behind. At worst, like maybe 2x, worst performance per watt. The gap is just far smaller than you would make it out to be if you follow like the chip war dialogue." 01:11:44

NVIDIA's Generational Gains Are Smaller Than People Think

The market prices NVIDIA as though each chip generation is a massive leap in efficiency. Neil argues the raw performance-per-watt improvement from Hopper to Blackwell to Rubin is actually quite modest.

"If you look at Hopper to Blackwell to Rubin and you compare like for like, what is the performance per watt of BFloat16 multiply? It hasn't improved all that much. Or you take that one step further and go to TSMC. If you look at TSMC 5 nanometer versus 4 versus 3 versus 2, the performance per watt on these chips doesn't change like a dramatic amount." 01:11:15

95% Uptime Data Centers Are a Viable Infrastructure Strategy

The entire data center industry treats high availability as non-negotiable. Neil argues this is an assumption made for training workloads, not inference — and being the first buyer of "bad" uptime is a structural cost advantage.

"You'd have basically zero buyers for a data center that has 95% uptime. I'm that first buyer. I will buy 95% uptime." 00:53:38

"We serve these long horizon agents because what happens when a request fails is that I'm going to have to go find a new GPU to put that request on... My argument is that my customers don't care because their agent was running for hours. They're sleeping." 00:54:05

The Frontier Labs' Three-to-Six-Month Premium May Not Hold

Conventional wisdom is that being at the frontier of model capability is a durable moat. Neil is skeptical — enterprises don't adopt at that pace, and capability diffuses fast.

"I don't know that the premium for being three to six months ahead is going to last that long. A lot of enterprises are probably still on like Opus 4.6 or Opus 4.7. They don't adopt the bleeding edge rapidly." 01:07:07

Intermittent Renewable Power Is Actually Fine for AI Inference Data Centers

The prevailing view is that solar and wind are unsuitable for data centers due to intermittency. Neil argues that long-running background agents make this a non-issue — you can predict outages and route workloads around them.

"I am totally capable of tolerating an outage from a data center that's measured in even days or weeks... It's in fact highly predictable. And I can just call in capacity in some other place of the world whenever that happens. I'll just model the weather and figure out when my data centers are going to be offline, move my work somewhere else, and it's fine." 00:55:47

3. Companies Identified

SAIL Research Inference company building a "token factory" optimized for long-running background agents. Differentiates on lowest cost per token, heterogeneous chip support, throughput-optimized software stack, and willingness to use low-uptime distributed data centers.

"Sail Research is a token factory. We have an API where anyone can send us requests where they can use large language models, open source language models for any task they want. We will serve those tokens to them at a price that is unbeatable in the market." 00:03:21

Parallel Web Systems A SAIL customer building an AI-powered index over the entire internet, monitoring it in real time for changes — a massive, long-horizon background workload.

"One of our customers Parallel Web Systems seeks to do. They want to build an index over the whole internet and they want to monitor the internet in real time for changes." 00:08:54

Cerebras Chip company building wafer-scale engines with massive on-chip SRAM (21 petabytes/second bandwidth) for ultra-low latency inference. Excellent for weight storage; challenged by KV cache capacity needs.

"Cerebras quits petabytes per second, 21 petabytes per second for their wafer scale engine 3. Compare that to HBM on an NVIDIA Blackwell is 10 terabytes per second or so in that range." 00:27:46

Groq Chip company with similar SRAM-focused architecture to Cerebras. Neil believes Groq and NVIDIA are already exploring a hybrid where Groq handles MLP layers and NVIDIA handles attention.

"I believe this is what's happening with NVIDIA and Groq." 00:32:30

DeepSeek Chinese AI lab praised specifically for publishing research on compressing KV cache — identified as the most important efficiency frontier in the current inference stack.

"DeepSeek certainly publishes really interesting work to compress that further and further. And they're making good progress." 01:00:10

NVIDIA Praised for NVLink, kernel software quality, long-term strategic thinking (not competing with customers), and engineer retention. Critiqued for performance-per-watt gains being overstated generationally.

"Jensen is really good at making his friends billionaires. He's made CoreWeave a many-billion-dollar company. There's no need for him to destroy that goodwill." 01:14:37

CoreWeave Called out as a prime beneficiary of Jensen Huang's deliberate strategy of making NeoCloud partners wealthy rather than competing with them.

"Jensen is really good at making his friends billionaires. He's made CoreWeave a billion-dollar company, a many-billion-dollar company." 01:14:37

Anthropic Cited as the first to hit 1 million context window length. Claude (Sonnet/Haiku/Opus) models referenced throughout as benchmarks. Identified as paying a premium to be 3-6 months ahead.

"Anthropic was, I think, the first to hit that 1 million context window length. I still use slash compact in my Claude Code well before 1 million context length." 00:30:51

Etched Chip startup called out as an example of the new wave of inference-focused silicon companies Neil is eager to evaluate.

"Net new companies are great, such as Etched or Samba Nova or Dmatrix." 00:46:01

Samba Nova Mentioned alongside Etched as a promising new chip company in the inference hardware space.

"Net new companies are great, such as Etched or Samba Nova or Dmatrix." 00:46:01

Dmatrix Mentioned as another emerging chip company worth watching.

"Net new companies are great, such as Etched or Samba Nova or Dmatrix." 00:46:01

AMD Discussed as an underrated chip option — perception that it's inferior to NVIDIA creates a buying opportunity for sophisticated operators like SAIL.

"There is alpha and just other people have this perception that AMD is not as good as NVIDIA. That's music to my ears. I'm very happy for them to sleep on this chip and for me to buy as much as I can." 00:45:33

Meta Mentioned as a major publicly confirmed buyer of AMD chips.

"I think publicly, Meta and OpenAI have bought a ton of AMD chips." 00:45:33

OpenAI Referenced as a major AMD buyer and mentioned in context of inference pricing benchmarks (GPT-5.5/5.6 at ~$5 per million tokens).

"At OpenAI pricing, that's at least $5 million at the very least for 5.5 or 5.6." 01:09:48

Cursor Cited as the single customer that pulled the entire inference industry toward low-latency optimization — the dominant force that shaped the wrong direction for everyone else.

"They all focus on low latency inference. And they were pulled in that direction by one very important customer, Cursor." 00:06:03

Base Ten, Fireworks, Together AI Mentioned as the existing class of inference providers all oriented around low-latency serving — the incumbents SAIL is differentiating against.

"You could take your pick, Base Ten, Fireworks, Together, they all focus on low latency inference." 00:06:03

xAI Mentioned as a cautionary example of poor GPU utilization at scale — and the implication is that underutilization is actually industry-wide and far worse elsewhere.

"We all make fun of xAI for having some challenges with total flop utilization on its clusters. The reality for the rest of the world is it's far worse." 01:01:06

Micron Called out as one of the irreplaceable HBM memory manufacturers; the HBM shortage is identified as the single most important bottleneck for chip startups to address.

"There's no easy way to bring on a lot more fabs of memory. So it's just going to be a while... The boys in Boise don't love huge CapEx for cyclical." 01:14:02

SK Hynix Named alongside Micron and Samsung as a critical DRAM/HBM manufacturer underpinning the entire AI chip stack.

"Not TSMC anymore, but now Micron, SK Hynix, Samsung. They build DRAM." 00:25:50

TSMC Discussed extensively as the bottleneck fab for advanced chips. Neil argues its geopolitical importance is overstated relative to Western alternatives.

"If you look at TSMC 5 nanometer versus 4 versus 3 versus 2, the performance per watt on these chips doesn't change like a dramatic amount." 01:11:15

Intel Named as the best Western alternative to TSMC, with Neil estimating it is at most 2x worse in performance per watt — far closer than the geopolitical narrative suggests.

"The best processes that we have in the West, like Intel, not that far behind. At worst, like maybe 2x, worst performance per watt." 01:11:44

4. People Identified

Tri Dao (Tridao) Creator of FlashAttention, the kernel that underpins virtually all modern transformer inference. Neil calls him foundational to the field and credits him with shifting the philosophy from hand-writing kernels to whiteboard-level design.

"There are great people like Tri Dao who write excellent kernels and they form the bedrock of all of our modern deep learning is built on FlashAttention. Modern transformers are built on FlashAttention." 00:39:04

Jensen Huang CEO of NVIDIA. Praised for strategic foresight in betting on ML in 2015-2016 and for deliberately building a partner ecosystem (NeoClouds) rather than competing with customers.

"Jensen is really good at making his friends billionaires. He's made CoreWeave a many-billion-dollar company. There's no need for him to destroy that goodwill. He wants to create a diverse community of NeoClouds and inference providers." 01:14:37

Andrej Karpathy Referenced for a foundational principle in ML model development: overfit first, then compress to generalize.

"One of the old Karpathy sayings is that if you have a new data set that you want to train a model for, your first goal should be to over-parameterize the model and try to overfit the data that you have." 00:35:48

Neil Movva Founder of SAIL Research. Former NVIDIA kernel engineer (started at NVIDIA while still in college). Deep expertise in GPU architecture, kernel development, and inference systems. Building a throughput-optimized inference company for the long-running agent era.

"NVIDIA was my first job while I was in college. I got to see how the Tensor cores got to earn their right to be on the chip." 00:15:08

5. Operating Insights

Chase "Speed of Light" as an Absolute Internal Standard, Not a Competitive Benchmark

Neil learned this directly from NVIDIA's engineering culture: define what the hardware is theoretically capable of, then relentlessly close the gap to that ceiling — never measure yourself against competitors.

"I tell all my engineers, we're chasing 100% speed of light. I don't care about relative numbers versus the competition. I only care about absolute numbers. What are we able to do on the chip? How do we achieve that?" 00:17:59

Hire for Unteachable Love of Performance, Not Credentials or Relevant Experience

Neil explicitly deprioritizes CUDA experience and AI background when hiring performance engineers. The only thing he cannot instill is the intrinsic obsession with squeezing every microsecond from a machine.

"I don't look for lots of AI experience. I don't look for CUDA experience at all. That's actually a huge red herring... I want to teach that, but I cannot teach the love for performance engineering. That is what I seek." 01:04:56

Write the System Design on the Whiteboard First; Let the Model Do the Implementation

For highly technical work like kernel engineering, the human advantage is conceptual architecture. Models can handle the execution. Reserving human effort for the "what and why" and delegating "how" to AI is the right division of labor now.

"We write kernels on the whiteboard. We go to the whiteboard, we describe what we think the machine should be doing. Then we succinctly describe that in natural language to a model. And then the model is able to do the execution." 00:40:30

Document Hardware Bottlenecks Explicitly and Feed Them Back to Vendors

Rather than accepting chip limitations as fixed, SAIL systematically characterizes performance ceilings, writes them down, and shares them with chip vendors — both to improve current deployments and to inform future hardware purchasing decisions.

"It's very important to me that we understand and characterize that very well and write it down so we can both A, tell NVIDIA about it, who are friends, and also to basically keep this in mind for future chips that we buy." 01:03:44

Design Business Architecture Around Asymmetric Tolerance for Messiness

SAIL's entire competitive advantage depends on being willing to accept what others reject: imperfect uptime, heterogeneous chips, intermittent power, small data centers. The operating insight is to explicitly identify the constraints your customers impose that you do not share — and build a business model around that gap.

"We tell our customers, look, our average throughput is going to be very competitive, but our P99, our 99% latency is not going to be controlled. It cannot be. And in return, I'll give you unbeatable economics." 00:54:34

6. Overlooked Insights

The "Reject Chips" Arbitrage: Fabs Are Over-Controlling Process Quality for Their Buyers' Actual Needs

Neil makes a brief but extraordinary observation: chip fabs spend enormous resources tightening process variation to deliver uniform chips — but for an inference company willing to use lower-quality dies, relaxing that process control could unlock dramatically cheaper chips and greater supply. No one is currently negotiating with fabs on these terms.

"Any fab has a lot of spread in their worst chip that comes out of the production line and the best chip that comes out of the production line... They are adding a lot of controls in the process that maybe I don't need. Maybe I'm actually willing to find a place for that worst chip. You don't need to tighten the process control as much, which takes more time and cost. Maybe I'm willing to take a lot more rejects." 01:02:35

This is a genuinely unexplored supply-side lever. A buyer who can underwrite "reject" or lower-spec chips creates a new market segment that doesn't currently exist, potentially at dramatically lower cost than any chip currently sold commercially. It implies a potential future where SAIL (or a similar company) negotiates directly with TSMC or others for below-spec wafers — a deal structure no one is apparently pursuing today.

Security Has Become Proof-of-Work: The Dollar Spent on AI Pen Testing Is the New Security Standard

Neil mentions almost in passing that the cybersecurity community has arrived at an extraordinary consensus: the best measure of how secure your software is is how many dollars you spent having AI models try to break it. This reframes the entire security industry's economics around inference spend.

"At some point, people started to make this joke that security has become proof of work. When you want secure software, it's really a question of how many dollars did you spend on Anthropic's APIs trying to break into your software. That is the best indication for how secure it is because that's the best tool in the world." 00:09:50

This is a profound reframing: software security is no longer a one-time audit or a headcount problem — it's an ongoing, metered compute expenditure. Every software company's security budget is now implicitly a long-running AI inference budget. This creates an enormous, recurring, price-sensitive demand pool for exactly the kind of cheap background inference SAIL is building — and it may be one of the largest near-term enterprise markets for the product, yet it was mentioned only in a single throwaway paragraph.