Import AI 467: Self-sus… | Jack Clark from Import AI Summary

1. Key Themes

Theme 1: AI-Powered Cyberattacks Are Now Operational, Not Theoretical

Researchers have demonstrated a working, self-replicating AI worm that uses stolen GPU compute to run local LLMs, plan attacks, and spread autonomously — with no dependency on external APIs.

"We must prepare for autonomous generative adversaries... a worm that generates tailored attack strategies to each target it encounters. The worm parasitically uses compromised machines to run open-weight large language models (LLMs) to sustain its reasoning, or extend its reach for further attacks."

The worm achieves an overall end-to-end attack success rate of ~37%, with 88% self-replication success once a foothold is established.

"The proof-of-concept operates using only an open-weight LLM running on a single, local GPU, with no reliance on vendor APIs that could be monitored or revoked."

Theme 2: Compute Is Not a Commodity — It Will Reprice Dramatically Upward as AI Capability Grows

Dwarkesh Patel argues that GPU pricing is currently distorted downward because AI can't yet do what top humans do. As that changes, pricing will normalize to the economic value AI delivers.

"If a true human-level software engineer that could run on an H100 equivalent, at current market rates for software engineers, that H100 should rent for over $250k a year. That's 15x today's spot prices."

"The reason AI is relatively cheap right now, at least in comparison to human labor, is partly that it can't do a lot of things that top humans can do. At some point that will no longer be the case. And so using GPUs to make short-form video slop will just get priced out."

Theme 3: The AI Industry Is Asking Governments to Help It Slow Itself Down

~1,337 employees and senior leaders across every major Western AI lab have jointly requested U.S. government support for internationally pacing frontier AI development — a remarkable collective action signal.

"Each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress."

"We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development."

Theme 4: AI Is an Extraordinary Engineer but a Poor Researcher — and This Matters for Recursive Self-Improvement Timelines

A multi-institution study used "shadow evaluation" — testing AI against unpublished NeurIPS 2026 submissions — and found that frontier AI agents could execute engineering tasks but failed to produce creative, original research contributions.

"While agents could solve the engineering problems necessary to do the research, they failed to produce original research at the caliber of a top ML conference."

"Both reviews highlighted the same failures: poorly motivated data and experiments, no novel contribution, and impenetrable prose."

Theme 5: AI Is Solving Open Mathematical Problems — Blurring the Engineer vs. Researcher Distinction

In apparent tension with Theme 4, OpenAI's unreleased "Astra" model solved ten open problems across high-dimensional geometry, coding theory, group theory, quantum complexity, and more — domains where creativity was assumed to be essential.

"New circuit lower bounds? A simple, easy-to-describe non-sofic group? Hardness of approximation for CVP without needing a unique games-like conjecture? I didn't just hear about these problems from my friends or from seminars. I feel their importance in my bones." — Henry Yuen, Columbia University

Clark frames this as an open question: "It remains to be seen whether AI systems can generate their own questions to ask which can expand the frontiers of human knowledge, but it certainly feels like we're getting close to this."

2. Contrarian Perspectives

Perspective 1: The AI Creativity Gap May Be the Key Brake on the Singularity — Not Safety Regulation

The consensus worry about recursive self-improvement (RSI) assumes AI will rapidly automate AI research. But empirical evidence suggests current AI systems are formulaic thinkers, not creative ones — which could dramatically slow RSI timelines regardless of governance interventions.

"Research papers like this continue to show that there's a certain absence of valuable, intuitive creativity in today's AI systems, and though they're extraordinarily capable engineers they seem to have a certain property of rote, formulaic thinking that might prevent them being good researchers."

Supporting evidence: When Anthropic attempted to automate scalable oversight research, human researchers had to "prime some agents with particularly good research directions to pursue, otherwise though they made some progress they failed to explore sufficiently creative ideas to dramatically improve performance."

Perspective 2: Cheap AI Compute Is a Temporary Market Anomaly, Not a Structural Feature

Most AI business models assume perpetually falling inference costs. Patel's argument inverts this: as AI becomes more capable, demand will bid up compute prices faster than supply can respond, disrupting any company whose unit economics depend on cheap GPU access.

"As AI models become smarter, they'll better monetize the same amount of compute... That's 15x today's spot prices... using GPUs to make short-form video slop will just get priced out."

Note: Patel acknowledges this is temporary — roboticization of the supply chain will eventually push prices back toward raw input costs — but the intervening period could be economically disruptive for low-margin AI applications.

Perspective 3: The Internet's Future Ecology Is AI Agents Fighting AI Agents, With Humans as Bystanders

The conventional framing of AI cybersecurity is "AI tools helping human defenders." Clark argues the more accurate frame is an autonomous ecosystem of adversarial and defensive agents, with humans designing the agents but not controlling the battles.

"The future internet is going to be more like a complex ecology full of attacker and defender AI agents than anything else... This may mean that humans need to create their own AI agents which they release onto the internet to serve as kinds of white blood cells against the adversary models."

"The worm operates in a fully decentralized manner, and no single point of control can be taken offline to interrupt its spread."

3. Companies Identified

OpenAI Description: Leading U.S. AI lab Why mentioned: Used an internal version of its unreleased "Astra" model to solve ten open problems in math and CS; also a signatory to the AI pacing statement Quotes: "These problems span high-dimensional geometry, coding theory, arithmetic circuit complexity, group theory, operator algebras, quantum complexity, lattice cryptography and extremal combinatorics."

Anthropic Description: AI safety-focused lab; CEO Dario Amodei and chief scientists are signatories to the pacing statement Why mentioned: Referenced for a prior internal experiment where automating scalable oversight research required human researchers to "prime" agents with good directions; also a co-signatory on the pacing statement Quotes: "A human researcher needed to prime some agents with particularly good research directions to pursue, otherwise though they made some progress they failed to explore sufficiently creative ideas to dramatically improve performance."

Google DeepMind Description: Google's AI research division Why mentioned: Signatory to the AI pacing statement; cited as one of the major Western AI labs whose co-founders and chief scientists signed Quotes: "Signatories include chief scientists and cofounders of Anthropic, Google, and OpenAI."

Meta Description: Social media and AI company Why mentioned: Signatory to the AI pacing statement Quotes: "A new statement is out with senior representation from all the major Western AI labs - OpenAI, Anthropic, Google DeepMind, Thinking Machines, Meta, and Safe Superintelligence Inc, among others."

Safe Superintelligence Inc (SSI) Description: AI safety startup co-founded by Ilya Sutskever Why mentioned: CEO is a signatory to the pacing statement; notable given SSI's stated mission of building safe superintelligence Quotes: "Signatories include...the CEOs of Safe Superintelligence and Anthropic."

Thinking Machines Description: AI lab (less publicly prominent) Why mentioned: Listed among major Western AI lab signatories to the pacing statement Quotes: "Senior representation from all the major Western AI labs - OpenAI, Anthropic, Google DeepMind, Thinking Machines, Meta, and Safe Superintelligence Inc, among others."

ServiceNow Description: Enterprise software and workflow automation company Why mentioned: One of the institutional affiliations of researchers who built the self-replicating AI worm proof-of-concept Quotes: "The results were achieved by researchers from the University of Toronto, the Vector Institute, the University of Cambridge, and ServiceNow."

4. People Identified

Dwarkesh Patel Description: Podcaster and technology writer Why mentioned: Authored the thesis that GPU compute will reprice 10-15x upward as AI capabilities improve to human-level, disrupting businesses built on cheap inference Quotes: "If a true human-level software engineer that could run on an H100 equivalent, at current market rates for software engineers, that H100 should rent for over $250k a year. That's 15x today's spot prices."

Henry Yuen Description: Associate Professor of Computer Science, Columbia University Why mentioned: Expert validator of OpenAI's ten open math/CS problem solutions; his emotional endorsement signals these aren't trivial results Quotes: "I didn't just hear about these problems from my friends or from seminars. I feel their importance in my bones; I deeply care about the answers to these questions."

Jack Clark Description: Author of Import AI; co-founder of Anthropic Why mentioned: Provides framing and editorial perspective throughout; authored the piece and the accompanying short fiction Quotes: "The future internet is going to be more like a complex ecology full of attacker and defender AI agents than anything else."

5. Operating Insights

Insight 1: Structured Reasoning Graphs Dramatically Improve Agent Reliability The AI worm's success was not due to raw model capability alone — it used a directed reasoning graph that decomposed tasks into scoped nodes (Plan, Judge, Action, Summary, Progress), preventing context bloat and keeping the agent focused. Operators building agentic systems should adopt similar architectures.

"By decomposing the agent's reasoning into these scoped steps, the graph controls what the LLM attends to at each decision point, and limits context growth to information relevant for the current sub-goal."

Insight 2: Don't Build Business Models That Require Cheap Compute Indefinitely Patel's analysis implies that any AI product relying on low-cost inference as a core margin driver is exposed to a structural repricing risk as model capability grows and demand intensifies. Entrepreneurs should pressure-test unit economics under a 10-15x compute cost scenario.

"Using GPUs to make short-form video slop will just get priced out."

Insight 3: Human "Priming" Remains Essential for AI Research Workflows Even the most capable frontier agents fail at open-ended research without human direction-setting. For AI-augmented R&D teams, the highest-leverage human role is not execution — it's identifying and seeding the right research directions for agents to pursue.

"A human researcher needed to prime some agents with particularly good research directions to pursue, otherwise though they made some progress they failed to explore sufficiently creative ideas to dramatically improve performance."

6. Overlooked Insights

Insight 1: Open-Weight Models Are the Enabling Infrastructure for the Most Dangerous AI Threats The AI worm's most threatening property — independence from any monitorable API — is made possible specifically by open-weight models. This is a concrete, operational argument in the open vs. closed model policy debate that goes beyond abstract safety concerns.

"The proof-of-concept operates using only an open-weight LLM running on a single, local GPU, with no reliance on vendor APIs that could be monitored or revoked."

Insight 2: "Shadow Evaluation" Is a New, More Rigorous Benchmark Methodology The researchers introduced a novel evaluation approach — testing AI against genuinely unpublished research — that sidesteps the benchmark contamination problem endemic to AI capability assessments. This methodology could become a new standard for measuring frontier AI research capability, with significant implications for how labs and investors assess true model capability vs. benchmark gaming.

"Shadow evaluation works by 'taking the central research question from a high-quality research paper that is not yet public, tasking a well-resourced frontier agent with answering it, and asking the paper's original authors to grade the agent's output as they would a conference submission.'"