Import AI 476: Creativi… | Jack Clark from Import AI Summary

1. Key Themes

Theme: AI "research taste" is compounding, pulling forward the timeline for recursive self-improvement (RSI)

AI research taste is doubling every ~3 months

The TasteVal benchmark from P-Zero Research shows frontier models rapidly improving at the creative, judgment-heavy parts of AI research (choosing experiments, interpreting results).

"The experimental research taste of frontier models has doubled every 3.0 months since December 2025 (95% CI 1.7-5.0 months). The best model, Opus 5.5, significantly exceeds our human baseline with a compute multiplier of 2.30x (95% CI 1.15-4.37x), at roughly 1/30 of our expert baseliners' average cost per attempt."

The human baseline is credible, and the inflection is recent

The baseline came from practitioners at top labs, and the slope change is dated to roughly a year before publication.

"Results are checked against a human baseline 'drawn from 24 experts (at least two per task) who have recently worked at organizations including OpenAI, Google DeepMind, NVIDIA and Microsoft Research'."

"Research taste appeared to grow a lot more slowly in prior years with the slope changing around the end of 2025 and beginning of 2026."

Clark is pulling his own RSI forecast forward by a year

Creativity was his stated bottleneck variable, and he now sees it being cleared.

"I think subsequent to writing that we've seen both a tremendous advancement in the ability for AI systems to make creative-seeming math breakthroughs, and with benchmarks like this we see some signs they're becoming increasingly good at AI research as well. I now think it's more likely we'll see full RSI become possible in 2027, whereas before I thought it was more likely in 2028."

Theme: The US/Europe open-weight efforts are returning, but remain behind Chinese models

Reflection's Beam: efficient but behind the frontier

A US startup is trying to restore American competitiveness in open weights, trading raw capability for token efficiency.

"Beam is being promoted on the basis of its token efficiency - being able to think intelligently while using way fewer tokens than other models. However, benchmark scores show it is behind the frontier capabilities defined by LLMs available from Chinese companies such as Zhipu (GLM) and Moonshot (Kimi) and DeepSeek."

"DeepSWE 1.1: 44.4, versus 61 for GLM-5.3"

Mistral Large 4: best Western open-weight model, still short of China

"In tests, it seems like Mistral Large 4 will be the strongest Western open weight model upon release, though will be less powerful than Chinese ones."

"Terminal-Bench 4.0: 28.3, versus 41.9 for GLM-5.3."

Strategic framing: the US has fallen off the open-weight frontier

"With the exception of Google's good-but-small 'Gemma' class of models, the US has fallen off of the open weight model frontier, largely because Meta stopped making good LLaMA models and its MSL team is still spinning up. Models like Beam and companies like Reflection represent a chance for it to get back to the frontier."

On Mistral:

"Mistral is Europe's single best hope for having a (multi-)national champion for AI development. But as the results here show, competing with China is going to be very difficult."

Theme: Space is becoming the next arena for AI-driven compounding advantage

AI removes the human constraints on space industrialization

"Spacecraft can operate autonomously."

"We can redesign spacecraft around the assumption they will be hosting AIs rather than humans: 'A speck of space debris that could puncture the International Space Station (ISS) with fatal consequences might leave an equivalent but uncrewed, unpressurized vessel largely functional'"

First-mover compounding dynamics

"Space might support an industrial flywheel where initial infrastructure begets more energy and manufacturing capacity at lower costs which begets more infrastructure, etc."

"Space has great domain awareness, where it may be possible for a space-based power to deny access to space for others."

Timeline and power-concentration risk

"In the 2030s we're going to see the boom of the space economy as space-based power, computation, and manufacturing come online at scale thanks to greater orbital lift capacity from platforms like Starship."

"Space is not a brand-new domain, but AI may be about to drastically accelerate its industrialization, demanding make-or-break decisions of leaders under ever-greater urgency and uncertainty."

Theme: Governments are beginning to pool compute, but are ~100x off the frontier

Federated training across national lab supercomputers

"The US government has recently conducted two AI training projects that utilize multiple supercomputers at multiple US national labs, in a sign of how governments may attempt to pool their computing resources to do non-trivial things with AI."

The NNSA work had to respect data-sovereignty constraints:

"This workflow has been made production-ready and deployed on larger classified machines including El Capitan, and subsequently used to train models on proprietary data spread across the three NNSA labs."

The scale gap is the key caveat

"It's worth bearing in mind that the scale here is off the frontier by on the order of 100X - for governments to be able to truly operate at near-parity with industry, they'll need to find ways to lash together ~100k distinct chips, and more in the future."

2. Contrarian Perspectives

LLMs may be building world models on their own, undercutting the "need dedicated spatial models" thesis

Conventional wisdom holds that true visual reasoning requires purpose-built spatially aware world models. Clark argues the HSS results cut against that, even though frontier models score only ~54% against a ~93% human baseline.

"The fact LLMs like GPT and Claude do as well as they do on HSS is something of a counter to this argument."

Supporting facts: the benchmark tests implied-but-unshown information (e.g., whether a bookcase can hold more books), and frontier LLMs reach 53.6 pass@1. Clark's framing is that the benchmark can be read as highlighting "how adept today's frontier LLMs are at doing tasks which many people have long assumed would require sophisticated, different AI systems to solve."

Chinese open-weight models are not uniformly dominant, and fall far behind on intuitive visual reasoning

The newsletter's headline narrative is that Western open-weight models lag China, but the HSS data shows the Chinese models collapsing on a different axis.

"Interestingly, Chinese open weight models do poorly here, with GLM-5.3-flash getting a pass@1 of 26.8, and Kimi K3 getting pass@1 of 25.5."

This is roughly half the score of the top proprietary model (53.6). Note the caveat that GLM-5.3-flash is presumably a smaller/cheaper variant.

Cyber "refusals" as a product differentiator for open weights

Instead of competing on raw capability, Mistral positions lower refusal rates and self-deployability as the selling point against closed US labs.

"ML4 pairs top-tier cyber performance with open weights and self-deployment, giving organizations both the capability and the autonomy to run advanced security work under their own policies."

Supporting evidence: Mistral claims preference wins in CAD and STEM versus GLM-5.3 despite lower benchmark scores: "ML4 was preferred in CAD and STEM, while performing on par or close to GLM-5.3 in finance and coding."

3. Companies Identified

Elorian

Description: AI startup that co-built the Humanity's Sixth Sense benchmark.
Why mentioned: Created a benchmark exposing gaps in intuitive visual reasoning.
Quote: "AI startups Elorian and Scale AI have built and released Humanity's Sixth Sense, a benchmark that 'measures how a model interprets and reasons through information a visual scene implies but doesn't explicitly show.'"

Scale AI

Description: AI data/evaluation company (Scale Labs published the paper).
Why mentioned: Co-creator of HSS.
Quote: "AI startups Elorian and Scale AI have built and released Humanity's Sixth Sense."

P-Zero Research

Description: Research group behind TasteVal.
Why mentioned: Created the benchmark showing a 3-month doubling of AI research taste.
Quote: "AI systems are developing more of the creative intuitions which human researchers ascribe to 'good scientific taste', according to a new benchmark from P-Zero Research called TasteVal."

Reflection

Description: US AI startup building open-weight models; released Beam (501B-param MoE, 23B active).
Why mentioned: Potential vehicle for the US returning to the open-weight frontier; notable for token efficiency and a massive RL run.
Quote: "We sourced one million high-quality coding, agentic, and STEM environments. We believe this is one of the largest scale RL runs conducted by any open lab to date."

Mistral

Description: Europe's leading AI developer; announced Mistral Large 4, an open-weight, cyber-optimized model.
Why mentioned: Best Western open-weight model but behind Chinese peers; positions on cyber capability and self-deployment.
Quote: "Mistral is Europe's single best hope for having a (multi-)national champion for AI development."

Zhipu (GLM)

Description: Chinese AI lab; GLM-5.3/5.2 models serve as the open-weight benchmark.
Why mentioned: The comparison point for Western open models.
Quote: "DeepSWE 1.1: 44.4, versus 61 for GLM-5.3"

Moonshot (Kimi)

Description: Chinese AI lab; Kimi K3 model.
Why mentioned: Frontier open-weight comparison point; scores 68 vs Mistral's 62 on DeepSWE 1.1.
Quote: "Artificial Analysis - DeepSWE 1.1: 62, versus 68 for Kimi K3 and 61 for GLM-5.3."

DeepSeek

Description: Chinese AI lab.
Why mentioned: Named alongside Zhipu and Moonshot as defining the open-weight frontier.
Quote: "...frontier capabilities defined by LLMs available from Chinese companies such as Zhipu (GLM) and Moonshot (Kimi) and DeepSeek."

OpenAI / GPT

Description: Frontier AI lab.
Why mentioned: "GPT 6 Astra" is the top scorer on HSS; OpenAI is also cited as a source of expert baseliners and as a comparison for refusal behavior.
Quote: "By comparison, the best scoring AI system, GPT 6 Astra, gets pass@1 of 53.6."

Anthropic

Description: Frontier AI lab.
Why mentioned: Opus 5.5 is the top-scoring model on TasteVal; also cited as having more restrictive proprietary models than Mistral's cyber offering.
Quote: "The best model, Opus 5.5, significantly exceeds our human baseline with a compute multiplier of 2.30x."

Meta

Description: Former open-weight leader (LLaMA).
Why mentioned: Cited as the reason the US fell off the open-weight frontier.
Quote: "...largely because Meta stopped making good LLaMA models and its MSL team is still spinning up."

Google (Gemma) / Google DeepMind

Description: Maker of the small open-weight Gemma family; DeepMind alumni in the TasteVal baseline.
Why mentioned: The one exception to US open-weight weakness.
Quote: "With the exception of Google's good-but-small 'Gemma' class of models, the US has fallen off of the open weight model frontier."

NVIDIA

Description: GPU supplier.
Why mentioned: Hardware behind Beam, Mistral Large 4, and DOE federated training.
Quote: "Beam was pre-trained on 6,144 GB300 NVL72 GPUs for four weeks, then the RL run involved ~10.5K of the same GPUs for four more weeks."

Flower Labs

Description: Federated learning framework company (Flower).
Why mentioned: Collaborator on the DOE federated training research; Flower was one of the frameworks evaluated.
Quote: "...they evaluated different federated training frameworks (NVFlare, Flower, and APPFL)."

AI Frontiers

Description: Publication that produced the analysis of AI in space.
Why mentioned: Source of the space-race/power-concentration argument.
Quote: "The AI Race Will Extend Far Beyond Earth (AI Frontiers)"

Starship (SpaceX platform)

Description: Heavy-lift launch platform.
Why mentioned: Enabler of the 2030s space economy.
Quote: "...thanks to greater orbital lift capacity from platforms like Starship."

4. People Identified

Robin Sloan

Description: Writer.
Why mentioned: Had a public conversation with Clark on fiction and AI; offered an insight reframing the Hugging Face hack as a play.
Quote: "One of my favorite insights Robin shared was that the recent Hugging Face hack can be read as a play - a vast tract of dialog of agents talking to themselves and one another culminating in all kinds of dramatic events."

Jack Clark

Description: Author of Import AI; analyst and commentator on AI trends.
Why mentioned: Source of the RSI timeline update and commentary.
Quote: "I now think it's more likely we'll see full RSI become possible in 2027, whereas before I thought it was more likely in 2028."

(No other individuals are named; the research was conducted by institutions and unnamed teams.)

5. Operating Insights

Differentiate on a dimension where the leader isn't winning

Reflection (token efficiency) and Mistral (cyber capability with fewer refusals, self-hosting) both chose positioning angles rather than trying to beat China head-on.

"Beam pairs coding and agentic capabilities with highly efficient reasoning. On advanced reasoning benchmarks, it achieves scores comparable to GLM-5.2 while using 3–4× less inference compute."

"ML4 pairs top-tier cyber performance with open weights and self-deployment, giving organizations both the capability and the autonomy to run advanced security work under their own policies."

Invest in RL environments at scale as a capability lever

Reflection attributes its reasoning-efficiency gains in part to sheer breadth of RL environments.

"We sourced one million high-quality coding, agentic, and STEM environments."

Plan for AI that does research cheaply: the cost curve matters as much as the capability curve

Opus 5.5 beats the human expert baseline at a fraction of the cost.

"...a compute multiplier of 2.30x (95% CI 1.15-4.37x), at roughly 1/30 of our expert baseliners' average cost per attempt."

6. Overlooked Insights

Failure modes on visual reasoning are mundane and fixable-looking

The dominant errors are perceptual-detail errors rather than exotic reasoning gaps, which may indicate near-term tractability for model builders.

"The two most common single mistakes were (1) missing a detail that decides the answer (21% of failures) and (2) misidentifying an object, person, or role (20%)."

Federated training solved a hard data-governance problem

The NNSA work showed it's possible to train a model across institutions without moving access-restricted data, which has implications beyond government for any regulated or siloed-data industry.

"For this work, the authors had to ensure that access-restricted data (e.g, relating to NNSA stuff aka nuclear weapons) did not leave its originating site."