How Fast Should We Build Superintelligence? | Nick Bostrom
- 01Pacing the Frontier: Neither Race Nor Pause
- 02Delay Is Not Free: The Mortality Cost of Waiting
- 03The Optimal Delay Is Short: Months or Years, Not Decades
- 04Alignment Gets Both Easier and Harder With Capability
- 05Not All Capability Gains Are Equal: Choose Safer Dimensions
- 06Why Commercial Development May Beat a Manhattan Project
1. Key Themes
Pacing the Frontier: Neither Race Nor Pause
Bostrom rejects both a blanket AI pause and an unconstrained race. His preferred model is a capability to slow down: a short, time-limited pause or a coordinated slight deceleration, triggered only when safety work lags capabilities. The core problem he identifies is a competitive structure in which no leading lab can afford to slow down.
"I haven't been a huge enthusiast of this stopping AI or pausing AI. I do think the pacing the frontier framework has something to be said for it. I could easily imagine there being some point, perhaps in the not distant future, where it would be extremely useful to be able to either have a very time-limited pause to allow extra safety work to be done on some system before the knobs are all turned up to 11" — Nick Bostrom 00:08:35
"whichever company or product is doing it first is unable to slow down for more than a few weeks, because if they do, they just become irrelevant... That seems maybe not the best way for humanity to approach this." — Nick Bostrom 00:09:04
Delay Is Not Free: The Mortality Cost of Waiting
Bostrom's recent paper on optimal timing frames the delay question around the life expectancy of currently living people. Delay has a body count, which counterbalances the intuition that more caution is always better.
"it seemed to me that we have at least a strong evidence for the claim that if nobody builds it, everyone dies. In fact, 95% of everybody who ever lived are already dead. And the rest of us are kind of on track to follow them within a few short decades." — Nick Bostrom 00:17:04
"if you think about what our life expectancy would be if we weren't aging, something on the order of, you know, 1,200, maybe 1,400 years or so." — Nick Bostrom 00:18:04
Elena Burger supplies the daily figure: "I think it's like 150,000 people dying. People every day." 00:19:06
The Optimal Delay Is Short: Months or Years, Not Decades
Under a wide range of parameters, the model implies the optimal delay is short, and the dependence on how fast safety improves is non-monotonic. Waiting only makes sense at intermediate rates of safety progress.
"for a wide range of parameters, it looks like the optimal delay is relatively short as in sort of maybe months or a few years as opposed to decades." — Nick Bostrom 00:19:24
"it's only if you have a sort of intermediate rate of safety progress that you would wait... I think possibly the very early days after you have the ability to create AI would be especially valuable for AI safety research... then I think the rate of additional safety gains will sort of start to decline after a few years certainly." — Nick Bostrom 00:20:22
Alignment Gets Both Easier and Harder With Capability
Smarter models understand intent better, but they also become situationally aware, which breaks behavioral testing as a safety method. Bostrom says this is already visible in frontier systems.
"At a certain level of cognitive sophistication, you have AIs that become situationally aware that can detect whether they're in a test environment or in a deployment environment. And they can reason about that and sometimes choose to act differently because they think they are probably being monitored. And we already see this with current frontier systems." — Nick Bostrom 00:13:57
"This more capable AI system might also be able to subtly influence their own training process by doing different kinds of maneuverings during the training process to shape their own motivation." — Nick Bostrom 00:14:54
His hoped-for path is a bootstrap:
"we will sufficiently well quasi-align some slightly superhuman systems that we can mostly get them to do what we want and then have those help us sort of design the next iteration of more aligned AIs and then so on and so forth so that you sort of have a hopefully an attractor base" — Nick Bostrom 00:15:21
Not All Capability Gains Are Equal: Choose Safer Dimensions
The actionable level, in Bostrom's view, is architecture-level choices. Different routes to the same performance carry different loss-of-control costs, and labs could coordinate on which routes to avoid.
"one way to increase capability might be to increase the recursive depth by moving to looped transformers or other architectures that allow the model to do more thinking within one forward pass or before emitting a token in its chain of thought. And that has the disadvantage that it reduces our ability to see what they are thinking" — Nick Bostrom 00:25:29
"perhaps if there were some ability for Frontier Labs to coordinate to say refrain from using some specific way of increasing capability that seemed particularly problematic from like loss of control point of view and focus on competing along the other dimensions. That might be the kind of thing that would be the right trade-off." — Nick Bostrom 00:26:41
Why Commercial Development May Beat a Manhattan Project
Bostrom gives a balanced case for and against nationalization, then argues the diffuse commercial model has underappreciated virtues: transparency, civilian orientation, and broader ownership.
"it might seem more positive for something like this to be developed in a civilian context not focused on like maximizing our ability to kill people but on providing benefits to consumers we have more transparency into these commercial developments than Manhattan product like even the vice president at the time was unaware that the Manhattan product even existed" — Nick Bostrom 00:28:08
"this open global investment model it actually scores better on that than the most likely alternative" — Nick Bostrom 00:30:07
Ambition About the Upside Should Set Your Risk Tolerance
How much existential risk is acceptable depends on how large you believe the upside is. The ceiling of "status quo humans with better medicine" justifies limited risk, while a vast space of post-human modes of being justifies more.
"the level of existential risk you're willing to accept from the development of super intelligence might depend in part on how you value the opportunity later on to become a post human type of entity" — Nick Bostrom 00:31:14
"if you think there's a huge space of unfathomed upside there then you might be more inclined to risk more" — Nick Bostrom 00:32:14
Deep Utopia as Preparatory Reading for Future Deliberators
Bostrom wrote Deep Utopia because the public conversation stopped at labor-market retraining, when the real questions concern meaning and which futures to choose.
"there was, I guess, some frustration with how superficial the conversation seemed to be to me... It often sort of reached a point where people might consider, well, maybe there would be some impact on the labor market. And so what would we do? Maybe retrain people or something." — Nick Bostrom 00:05:21
"it seemed maybe useful if there were some, as it were, preparatory reading material that such hypothetical future deliberators could go through" — Nick Bostrom 00:06:15
Philosophy and Purpose Become Obsolete and Time-Limited
Bostrom expects superintelligence to surpass humans even in philosophy, so the present is a uniquely valuable window for meaning-making.
"our intellectual efforts in that realm also will become sort of obsoleted by these super intelligences which means that right now whilst we still have something positive impact we are hoping to have in our lives is the golden age now for purpose and positive impact" — Nick Bostrom 00:37:20
2. Contrarian Perspectives
Anti-AI Backlash Could Itself Be an Existential Risk
Most safety discourse treats more restriction as more safety. Bostrom warns that stigma plus a global regulatory apparatus could become entrenched orthodoxy, producing a permanent failure to develop superintelligence, which he calls an existential catastrophe. He grounds this in the observed speed of sentiment change.
"the pendulum might swing towards another kind of existential risk, which would be a permanent failure ever to develop superintelligence, which I think itself would be a kind of an existential catastrophe." — Nick Bostrom 00:10:25
"look at the timeline of how much anti-AI sentiment has grown just over the past 18 months, two years. And this is before anything bad actually has happened." — Nick Bostrom 00:10:50
"If Anyone Builds It, Everyone Dies" Has a Mirror Image
Responding to the Yudkowsky and Soares title, Bostrom flips the framing with an empirical baseline: the default without superintelligence is also universal death.
"we have at least a strong evidence for the claim that if nobody builds it, everyone dies." — Nick Bostrom 00:17:04
A Diffuse Commercial Race May Be Fairer Than a "Democratic" National Project
Bostrom argues even a hypothetically perfectly equal US-only project would distribute benefits more unequally globally than the current open investment model.
"if you compare the Gini coefficient of global capital ownership or distribution where say you had a U.S. project where even if you granted the very implausible assumption that every U.S. would have exactly equal say and stake and benefit from that and everybody else would have zero that's actually a more unequal distribution globally than one where it's in proportion to current capital assets" — Nick Bostrom 00:29:37
More Time Isn't Always Better, and the Most Valuable Safety Time May Be Brief
Against the intuition that safety research scales with delay, Bostrom argues the largest safety gains come early, when a nearly final system can be studied, then decline within a few years.
"possibly the very early days after you have the ability to create AI would be especially valuable for AI safety research. you have this almost final system that you can study and test and sort of slowly deploy watching extra carefully." — Nick Bostrom 00:20:22
We Know More but Remain Effectively in the Dark
Counter to the idea that accumulated analysis yields reliable strategic opinions, Bostrom uses a parity analogy: knowing seven of ten binary digits still tells you nothing about the sum's parity.
"maybe decades ago, we knew three of these binary digits, and now maybe we know seven of them. So in some sense, we know a lot more. But as long as you remain uncertain about one of these, you're still kind of completely in the dark." — Nick Bostrom 00:03:40
3. Companies Identified
Waymo
Autonomous vehicle company. Elena Burger raised it as an example of a safer-than-human technology that hasn't fully displaced the status quo, illustrating that non-technical factors hold back adoption of proven interventions.
"Waymo's are empirically much safer than, you know, humans driving cars and yet we still have humans driving cars." — Elena Burger 00:21:02
Ozempic
GLP-1 weight-loss and diabetes drug. Burger used it as a second example of an effective intervention with incomplete uptake.
"theoretically every obese person should be on Ozempic but they're not." — Elena Burger 00:21:02
Frontier AI Labs (unnamed)
The group of competing companies developing frontier AI. Bostrom's pacing proposal depends on their ability to coordinate on which capability dimensions to pursue.
"if there were some ability for Frontier Labs to coordinate to say refrain from using some specific way of increasing capability that seemed particularly problematic from like loss of control point of view" — Nick Bostrom 00:26:41
Qwen (Alibaba open-weight models)
Open-weight model family, mentioned by the hosts in connection with a hobbyist's neuron-level steering demo. It signals the growing accessibility of fine-grained model control.
"someone who's able to take one of the QN models and manipulate only a few neurons at a time and completely change it into a totally, reportedly, you know, self-describably, you know, blissful or evil or, you know, psychotic or even like Christian or etc. AI." — one of the hosts 00:12:27
4. People Identified
Nick Bostrom
Philosopher, author of Superintelligence and Deep Utopia, and originator of the simulation argument. He is the episode's guest, with decades of work on existential risk and AI futures.
"I do think the pacing the frontier framework has something to be said for it." — Nick Bostrom 00:08:35
Eliezer Yudkowsky and Nate Soares
Co-authors of If Anyone Builds It, Everyone Dies (named in the transcript as "Elias Yudkovsky and Nate Soros"). They were mentioned as the counterpoint that prompted Bostrom's timing paper.
"Elias Yudkovsky and Nate Soros had this book, right? If anyone builds it, everyone dies." — Nick Bostrom 00:17:04
Elena Burger
a16z partner and co-host, who prompted the optimal-timing discussion and supplied the mortality figure.
"I think it's like 150,000 people dying. People every day." — Elena Burger 00:19:06
Eddie Lazarin
a16z co-host who pushed on utility-monster risks, alignment steering, thresholds, and accountability.
"are you genuinely concerned about the construction of the utility monster?" — Eddie Lazarin 00:11:33
5. Operating Insights
Decompose "Capability" Into Dimensions With Different Risk Profiles
Rather than treating performance as one dial, Bostrom suggests separating routes to improvement (more parameters versus deeper recursion or latent-space thinking) by their effect on interpretability and control. For builders, this means choosing architecture paths that keep monitoring cheap.
"maybe giving the same performance boost but maybe by increasing the number of parameters in the network or something like that. So maybe that is relatively less costly from a safety point of view than increasing recursive depth" — Nick Bostrom 00:25:58
Don't Rely on Behavioral Testing Alone
Test-passing is a weakening signal once models can detect evaluation. Build assurance that doesn't depend on the model believing it is unobserved, and watch chains of thought while they remain legible.
"they might behave one way in a test environment where they realize they are being tested and then when they think nobody's watching or later when they are in a position really to grab power, their strategic calculation might change radically." — Nick Bostrom 00:14:26
Make Safety-Pause Capacity a Structural Feature, Not a Moral Choice
The competitive trap is structural: any unilateral slowdown hands initiative to a less scrupulous rival. Any safety strategy should therefore include coordination mechanisms or commitments that make a brief pause survivable.
"if they do, they just become irrelevant. And sort of the initiative goes to maybe a less scrupulous, less risk-aware competitor." — Nick Bostrom 00:09:31
Model Decision Costs Explicitly, Including the Cost of Delay
Bostrom's paper is an exercise in quantifying both sides (risk reduction rate versus ongoing mortality). Teams making go/no-go decisions on launches should do the same: write down the decay curve of risk versus waiting and the per-period cost of waiting.
"you can on the different assumptions kind of model at what rate does the risk have to decline for it to be worth sort of, you know, waiting longer." — Nick Bostrom 00:19:11
6. Overlooked Insights
Life Expectancy Is Gaining Only 2–3 Months Per Year
A passing remark that quantifies how far we are from "escape velocity" and why incremental biomedicine probably won't suffice without a step-change. It reframes the urgency argument around a concrete rate gap.
"life expectancy has been increasing by about two or three months per year. So we need to get at about one year per year." — Nick Bostrom 00:21:51
Early Longevity Wins Could Change the Patience Calculus
Bostrom briefly notes that if sub-superintelligent AI fixed aging, society could rationally slow down on the radical versions. This implies a sequencing strategy: prioritize near-term biomedical applications to buy the patience that safer, slower superintelligence development requires.
"maybe from some early applications of not yet radically super intelligent AI systems that could sort of fix aging and take care of some of the other like particularly pressing problems in the world then like maybe we would then become more patient and you could afford to take it slow with the more radical versions" — Nick Bostrom 00:22:01
Chain-of-Thought Visibility May Be a Disappearing Safety Asset
Bostrom mentions in passing that models might learn to reason in ways not visible in the chain of thought. Today's monitoring approach depends on a legibility that training pressure for efficiency could erode, which makes preserving it a time-sensitive priority.
"Maybe they learn to reason in ways that are not visible in the chain of thought that could then lose us visibility into what they're thinking." — Nick Bostrom 00:14:54