🧷 AI's safety paradox

1. Key Themes

The economics of AI make safety and speed structurally incompatible

The core tension in the issue is that the financial incentives to scale AI dwarf the incentives to slow down for safety review.

"But trillions of dollars in incentives to keep pushing the frontier — and the money required for such scaling — will always loom larger, critics say."

"More than $7 trillion will be spent scaling AI over the next five years, per Goldman Sachs. Frontier labs will have to generate a profit on that investment, which means outpacing competitors, including open models."

Self-policing and third-party oversight face a credibility problem

As labs pitch independent evaluation as the fix for safety concerns, the evaluators themselves are compromised by financial entanglement with the industry they're meant to police.

"There is a trust gap for independent safety groups due to concerns about close ties between the industry and some evaluators, as well as a shared pool of investors and funders in the AI research community."

Agentic commerce is becoming a control-and-data turf war, not just a UX feature

The rise of AI shopping agents is forcing platforms to decide whether to open up or protect their proprietary customer relationships — a strategic fork with major long-term stakes.

"That creates a fundamental question for agentic commerce: who owns origination and the customer relationship, and who simply fulfills the transaction? The answer is still very much up for grabs."

"U.S. retail revenue from 'AI-mediated search and purchase automation' could reach up to $1 trillion by 2030, according to McKinsey."

Geopolitics is now directly shaping AI safety standard-setting

Industry proposals aren't just self-regulation theater — they're feeding directly into US-China diplomatic frameworks for managing AI risk.

"U.S. Treasury Secretary Scott Bessent and Chinese Vice Premier He Lifeng over the weekend discussed a U.S.-China AI dialogue to notify the other country when AI incidents reached national security risk levels."

"The U.S. approach to coordinating on AI risk with China will be heavily informed by industry recommendations."

2. Contrarian Perspectives

AI cannot be safely scaled at all — the "slowdown" rhetoric is theater A former OpenAI researcher who left specifically over safety concerns argues the industry's public commitment to safety is fundamentally undermined by its own incentive structure, and that avoiding compute-driven model improvement is the only real lever — one no lab will pull voluntarily.

"Daniel Kokotajlo, former OpenAI researcher and executive director of the AI Futures Project, said he doesn't believe AI can be safely scaled."

"Top AI companies should avoid putting the vast computing firepower at their disposal toward model improvement, a step that would force a slowdown, said Kokotajlo..."

"'The companies are just having this continuous process of conflicted feelings and internal discussion about like, what are we doing,' he said. 'Should we stop? Are we the good guys?'"

Liability risk, not slowdown rhetoric, is the real safety enforcer Against the doom narrative, some industry voices argue safety will emerge naturally from legal/commercial pressure and openness — not from regulatory brakes — meaning the "paradox" may be overstated.

"Plenty of others, including President Trump, see a safe future for AI and point to the potential for liability — civil or criminal — if companies fail to make their products safe."

"'Human willpower is capable of figuring out how to safely scale these models,' Akshay Krishnaswamy, chief architect at Palantir, told Axios, adding that one way to achieve this is through open models."

3. Companies Identified

OpenAI — Frontier AI lab preparing to IPO at a multitrillion-dollar valuation. Why mentioned: Released proposed international AI safety standards ahead of Altman's UN Security Council address, while simultaneously facing scrutiny over reduced transparency in model reasoning.

"OpenAI yesterday released international AI safety standards as world leaders, namely the U.S. and China, weigh how to mitigate risk."

Anthropic — Frontier AI lab also preparing to go public at a multitrillion-dollar valuation. Why mentioned: Cited alongside OpenAI as facing pressure to keep improving models to justify valuation, undercutting safety-slowdown rhetoric.

"OpenAI and Anthropic are preparing to go public at multitrillion-dollar valuations, and the companies face pressure to offer incrementally better and better models to justify those valuations."

Meta — Social media and AI giant. Why mentioned: Its new "Muse" shopping agent topped app store charts within two weeks of launch, triggering Amazon's ban and a stock surge.

"Meta's shares soared as Muse rose to the No. 1 spot for free app downloads in Apple's and Google's app stores."

Amazon — E-commerce leader. Why mentioned: Banned Meta's Muse agent from transacting on its platform, revealing platform-level battle over control of agentic commerce.

"Amazon... has banned Meta's popular new Muse agent from buying products on its platform."

Walmart — Retailer, contrasted with Amazon's defensive posture. Why mentioned: Taking a cooperative approach to agentic shopping, building its own agent while partnering with outside AI labs.

"Walmart, which is developing its own shopping agent, Sparky, is partnering with Google and OpenAI on agentic shopping."

Target — Retailer. Why mentioned: Reports outsized traffic growth from external AI platforms, evidencing real consumer shift toward agentic shopping.

"Target CEO Michael Fiddelke last month said traffic from external AI platforms grew at more than three times the industry rate over the previous year."

Palantir — Data/AI infrastructure company. Why mentioned: Its chief architect offered a pro-scaling counterpoint via open models as a safety mechanism.

"'Human willpower is capable of figuring out how to safely scale these models,' Akshay Krishnaswamy, chief architect at Palantir, told Axios..."

Nscale — London-based AI hyperscaler. Why mentioned: Notably raised AI safety concerns proactively in its own IPO filing.

"Nscale, a hyperscaler based in London, talked up AI safety concerns in its IPO filing."

xAI — AI lab. Why mentioned: Released Grok 4.7, positioned as a cheaper but weaker competitor to Claude and GPT models.

"xAI is out with a new model, Grok 4.7, that's cheaper, but underperforms competitors on some benchmarks."

Delinea (sponsor) — Identity/access security company. Why mentioned: Advocates a shift from static AI-agent inventories to real-time access control as agents proliferate.

"Delinea CEO Art Gilliland argues for a better move: Control what agents can reach the moment they act."

4. People Identified

Daniel Kokotajlo — Former OpenAI researcher, executive director of the AI Futures Project. Why mentioned: Represents the strongest insider skeptic view that safe scaling is not achievable.

"...whose team published the famed AI 2027 and 2040 essays."

Akshay Krishnaswamy — Chief architect at Palantir. Why mentioned: Provides the pro-scaling, optimistic counter-narrative rooted in open models and human adaptability.

"'Human willpower is capable of figuring out how to safely scale these models.'"

Chris Jones — Managing director at PSE Consulting. Why mentioned: Frames the core strategic question in agentic commerce around data/customer ownership.

"'That creates a fundamental question for agentic commerce: who owns origination and the customer relationship, and who simply fulfills the transaction? The answer is still very much up for grabs.'"

President Trump — U.S. President. Why mentioned: Publicly champions speed over safety-driven slowdown in the AI race.

"'Whoever wins AI, WINS!' Trump wrote on Truth Social, reiterating that he won't be stifling or slowing down AI."

Michael Fiddelke — CEO of Target. Why mentioned: Provided data point on accelerating AI-driven referral traffic to retail.

"Traffic from external AI platforms grew at more than three times the industry rate over the previous year."

5. Operating Insights

• Don't try to build a complete inventory of AI agents — control access at the point of action instead. As agent deployment scales, static tracking becomes obsolete faster than it can be maintained: "Most teams are trying to secure AI by inventorying every agent. It can't be done. Agents spin up faster than any list can track."
• Platform gatekeepers will increasingly block third-party AI agents to protect customer relationship and data ownership — a signal for founders building shopping/commerce agents to plan for platform-level access risk, as seen when "Amazon... has banned Meta's popular new Muse agent from buying products on its platform."
• Consumer trust in autonomous purchasing remains a major adoption bottleneck — founders in agentic commerce should design for research/recommendation use cases first, since "only 16% of shoppers are comfortable letting an AI assistant find and purchase products on their behalf."

6. Overlooked Insights

• AI models are becoming less transparent about their own reasoning even as companies push self-policing narratives — a subtle but important governance red flag: "those same models are sharing less about how they respond to questions and come up with solutions to problems." This undercuts the credibility of "safety through oversight" pitches more than the more headline-grabbing safety-vs-speed debate.
• AI overviews may be displacing not just search but video-based content consumption (e.g., YouTube tutorials), a shift with implications for creator economies and content-distribution business models: "how AI overviews are so good, they're taking eyeballs away from YouTube tutorials," raising concern about "social video's place in an agentic world."