Reiner Pope – Chip design from the bottom up
- 01The Hidden Cost of Data Movement in Chips
- 02The Quadratic Scaling of Precision
- 03Compute-to-Communication Ratio as the Universal Design Principle
Dwarkesh Podcast | Participants: Dwarkesh Patel, Reiner Pope (CEO of Maddox)
1. Key Themes
The Hidden Cost of Data Movement in Chips
The central thesis of the entire conversation is that in chip design, the actual computation (the multiply-accumulate) is a small fraction of total circuit cost — most die area is consumed by moving data to the computation. This fundamental tension drives every major architectural decision.
"Seven-eighths of the cost is in the reading and writing the register file. And only a tiny fraction of the cost is in the logic unit itself. So this is the problem to solve." — Reiner Pope 00:25:28
This insight cascades upward: it explains why systolic arrays (tensor cores) were invented, why TPUs look different from GPUs, why scratchpad memory beats cache for deterministic workloads, and why Maddox is building what they call a "splittable systolic array."
The Quadratic Scaling of Precision — and Why Nvidia Has Been Underreporting It
The number of gates required to perform multiplication scales as P×Q (where P and Q are bit widths). This means cutting precision in half doesn't just double your throughput — it should quadruple it. Nvidia's own spec sheets historically reported only a 2X improvement when halving precision (e.g., FP8 to FP4), which Reiner identifies as technically wrong.
"That ratio is actually slightly wrong. It should be like an even bigger, you should get an even bigger speedup than you might otherwise think. Nvidia's product specs have sort of started acknowledging that in B300 and beyond, where the FP4 is three times faster than the FP8. Right. Though it should be 4X." — Reiner Pope 00:15:37
This is a significant signal: the industry is only now catching up to the physics of what lower precision actually enables, and companies building hardware around INT4/FP4 natively may have a larger structural advantage than the market currently appreciates.
Compute-to-Communication Ratio as the Universal Design Principle
From the gate level all the way up to multi-chip inference clusters, the same optimization principle applies: maximize actual computation relative to the cost of moving data. This isn't just a chip-design insight — it's a unifying framework.
"This shows up sort of all the way up and down the stack... There's like a square cube law or a squared versus linear term going on both in just purely the precision of this ALU, but then also in terms of the size of the matrix." — Reiner Pope 00:35:46
2. Contrarian Perspectives
Systolic Arrays Are the Only Efficient Way to Do Matrix Multiply — Everything Else Is Overhead
Before tensor cores (pre-Volta GPUs), nearly all die area was wasted on register file MUXes rather than actual math. Most people think of CUDA cores as "the GPU" — Reiner's framing is that CUDA cores were fundamentally wasteful and the move to tensor cores was an admission of that.
"This is what was inside the CUDA cores. And this problem statement is what motivated introduction of tensor cores, which are more generically called systolic arrays... We're spending almost all of our circuit area on something that we just really don't care about." — Reiner Pope 00:25:42
GPUs Are Just "Many Tiny TPUs" — And That Architecture Has a Hidden Bandwidth Tax
The conventional framing is that GPUs and TPUs are fundamentally different architectures. Reiner reframes it: a GPU SM is essentially a miniaturized TPU, and having many small ones versus a few large ones is a tradeoff — not an unambiguous win.
"At a very high level point of view, the GPU has a lot of tiny, tiny TPUs, sort of tiled across the whole chip... You need to move a lot of data from the vector unit to the matrix units through sort of two lines of perimeter. Whereas in a GPU, you're moving all this data through like 16 lines." — Reiner Pope 00:17:26 / 00:18:54
The implication: GPUs have higher internal bandwidth between vector and matrix units because they're fragmented, but this comes at the cost of being constrained to small systolic arrays that can't amortize register file costs as well.
FPGA vs. ASIC Is a 10X Efficiency Gap — And It's Entirely Explained by MUX Overhead
The widely-cited rule that FPGAs are ~10X less efficient than ASICs is often treated as a vague heuristic. Reiner derives it from first principles: an FPGA lookup table requires ~32 gates to implement what an ASIC does in 3 gates, because the LUT is fundamentally a MUX selecting among all possible truth-table entries.
"If you look at the cost of building this lookup table, it's like 32 gates. And then it can give me the equivalent of like a four-way AND gate. I could implement it in an ASIC directly using these three AND gates. But using a LUT, it's going to take like these 32 gates instead of three." — Reiner Pope 01:02:19
Deterministic Latency Is Actually the Simpler Design — CPUs Were Deliberately Made Non-Deterministic
The common assumption is that determinism is hard to achieve and non-determinism is the natural state. Reiner inverts this: determinism is the default; caches and branch predictors are features that CPU designers added which introduced non-determinism as a side effect.
"It's actually possible to remove those design choices and make a CPU that has deterministic latency. Those are not very attractive in the market and so people don't make those CPUs anymore. Deterministic latency is maybe a simpler design starting point, and then some chip designers have added things into it to be non-deterministic." — Reiner Pope 01:04:06
3. Companies Identified
Maddox
- New AI chip startup. CEO is Reiner Pope.
- Why mentioned: Dwarkesh is an investor. Reiner describes their novel "splittable systolic array" architecture — large systolic arrays that can also operate as small systolic arrays, attempting to get the efficiency of TPU-style coarse-grained compute with the flexibility of GPU-style fine-grained parallelism.
"We've talked publicly about something which we call a splittable systolic array, which is in some sense you can think of as big systolic arrays that can be small systolic arrays as well." — Reiner Pope 01:20:13
- GPU cloud provider.
- Why mentioned: Cited as one of only five gold-tier providers in Semi Analysis's ranking of ~100 GPU clouds. Gold-tier providers had 5-15% lower total cost of ownership even at identical GPU pricing, driven by fault detection, uptime, and rapid node replacement (<10 minutes).
"Crusoe was one of the first clouds to adopt Envy Sentinel, NVIDIA's own GPU monitoring and self-healing software for enhanced GPU uptime utilization and reliability... Crusoe can swap in a healthy node in less than 10 minutes." — Dwarkesh Patel 00:22:09
- AI inference chip company.
- Why mentioned: Cited as a public example of a chip with deterministic latency, alongside TPUs — validating that the scratchpad/deterministic design philosophy is commercially viable.
"The processes that are inside a lot of AI chips actually also have deterministic latency too. Groq has advertised this, TPUs have that in the core as well." — Reiner Pope 01:03:51
- Quantitative trading firm.
- Why mentioned: Used as a concrete example of a sophisticated FPGA user who values deterministic clock cycles for high-frequency trading. Also mentioned in the context of AI threatening (or not) quantitative trading.
"Trading in particular feels to me like kind of AGI complete... Jane Street isn't betting against AI. They just signed a $6 million compute deal." — Ron Minsky 00:51:01
4. People Identified
Reiner Pope
- CEO of Maddox, former chip architect (likely Google TPU background given depth of TPU knowledge).
- Why mentioned: Exceptional ability to explain chip architecture from first principles. Identifies non-obvious inefficiencies (quadratic precision scaling, MUX overhead, data movement costs) that have real investment implications.
"This is the most efficient known mechanism for implementing a matrix multiply." — Reiner Pope 00:36:31
- Senior person at Jane Street.
- Why mentioned: Makes the provocative claim that trading is "AGI complete" — meaning if AI can trade, it can do almost anything cognitively. Yet Jane Street is aggressively hiring and investing in compute, suggesting the edge keeps moving rather than disappearing.
"I have never been more desperate to hire more engineers and more traders than I am today. You have the usual thing of like the other hard parts that we don't yet know how to automate — well, that ends up being where the competitive edge lies." — Ron Minsky 00:51:31
5. Operating Insights
Budget Die Area Explicitly — It's the Core Chip Design Discipline
Reiner describes chip design as fundamentally a series of area-allocation decisions. The explicit framing of "what percentage of my chip area do I want to spend on data movement vs. compute?" is directly portable to any system design problem — how much overhead (management, tooling, communication) vs. direct value creation are you funding?
"One way to think of it is to say, I'm gonna have a budget for how much of my chip area I want to spend on data movement. So maybe I just say that I want this to be 10% and the systolic array to be 90%. And then I can size my register file." — Reiner Pope 00:37:42
Load Weights Slowly, Reuse Them Often — The Right Amortization Strategy
The systolic array loads weight matrices slowly (trickling in over many clock cycles via a daisy chain) but then reuses them across many input vectors. This minimizes bandwidth (die area) while maximizing utilization. The operating analog: invest heavily upfront in fixed assets (models, infrastructure, knowledge) that can be amortized across high volumes of usage.
"The trick is just we do it very slowly. We very slowly trickle feed it in into the systolic array... We bring in a number into the top row of the systolic array, and then for Y consecutive clock cycles we shift all the other rows down by one. That keeps the wiring that needs to come from this expensive register file only down to a factor of X." — Reiner Pope 00:33:28
Clock Speed vs. Throughput Is a False Optimization — Don't Confuse Latency with Output
Maximizing clock speed by over-inserting pipeline registers actually hurts throughput because you're spending die area on synchronization rather than computation. The analogy to batch size in inference (low batch = fast response per user, lower total throughput) is explicit in the conversation.
"The throughput of your chip, you can think of as the product of how much I can get done per clock cycle, which is based on this area efficiency thing, times how many clocks I get per second... You get less parallelism out if you drive your clock speed up really high." — Reiner Pope 00:50:13
6. Overlooked Insights
FP4 vs. FP8 Fungibility Is a Chip Design Choice — Not a Physical Law
Dwarkesh raises and Reiner confirms a point that most GPU buyers completely miss: FP4 and FP8 compute units are not fungible on current hardware. The ratio of FP4 to FP8 performance is a deliberate design decision made at tape-out time, based on the designer's assumptions about future workload mix. This means if the industry shifts faster to FP4 than Nvidia assumed when designing Blackwell, those chips are suboptimally configured — and a new entrant who designs around FP4-native workloads from the start has a structural advantage that can't be patched in software.
"As drawn, they're actually not particularly fungible. This is actually one of the main choices you have to make when designing a chip, which is how much FP4, how much FP8 do I have? And then sometimes I'll make that consideration from the point of view of what I think is the customer requirement." — Reiner Pope 00:13:43
Combined with the quadratic scaling insight (FP4 should be 4X faster than FP8, not 3X), this suggests that the first chip company to fully commit to FP4-native architecture — with the systolic array, memory buses, and register files all sized for 4-bit — may unlock a step-change in efficiency that the incumbents, constrained by backward compatibility, cannot match.
The Brain Runs at "Batch Size 1" — And That's Why Its Energy Efficiency Argument Needs Reexamination
Reiner briefly makes a profound point: when people compare GPU energy consumption to the brain, they're comparing the GPU running at batch size 1000 to a brain running at batch size 1. A fair comparison would be a GPU running at 1 MHz processing a single stream — and in silicon, that does reduce energy proportionally (dynamic power scales with switching frequency). The brain's efficiency advantage may be much smaller than commonly assumed, and the gap is primarily about parallelism utilization, not architectural superiority.
"When we compare a GPU running some workload, it's running batch size 1000 or something like that. Whereas the brain is not running batch size 1000. There's only one of me... If you just end up running this circuit once to stabilization and then it'll sit idle for a long period of time, it doesn't consume a lot of energy while it's sitting idle because most of the energy is consumed in toggling bits from zero to one and back." — Reiner Pope 01:13:42 / 01:14:37