Import AI 461: "Alignment is not on track"; FrontierCode; and synthetic research interns

1. Key Themes

Theme 1: Alignment Research Is a Critical and Underfunded Infrastructure Gap

The consensus view is that safety is "being handled" by frontier labs. Sequent's founders — from AISI and Timaeus — explicitly reject this. The article signals a growing belief that safety work is not keeping pace with capability development, creating an opportunity and an urgency for independent research institutions.

"Artificial superintelligence (ASI) may be developed in the next few years. It is unclear whether alignment is on track to be ready on the same timeframe. At a minimum, the empirical programs at AI labs are unlikely to deliver a priori confidence, before training ASI, that things will go well."

Theme 2: Coding Benchmarks as a Leading Indicator of AI Capability Progress

The rapid saturation of coding benchmarks (SWE-Bench lasted ~2 years) is itself a signal of the pace of progress. FrontierCode's difficulty level — and Clark's prediction that 70%+ Diamond scores will arrive before June 2027 — makes it a useful instrument for tracking the "AI as software engineer" thesis.

"SWE-Bench was introduced in October 2023 and has probably recently aged out of usefulness due to saturation. How long might FrontierCode last? I predict we'll see systems getting 70%+ on Diamond by June 2027."

Theme 3: Speed as a Distinct Dimension of AI Capability — and a Chinese Hardware Workaround

Inference speed is not just a cost optimization; it is a capability unlock. Xiaomi's 1,000 tokens/second achievement on commodity hardware reframes the competitive dynamic: Chinese firms are engineering around export controls through software-hardware co-design rather than simply acquiring better chips.

"If you can generate more tokens more quickly it unlocks tasks that are previously unthinkable, like rapidly refactoring software on the fly... work like this is a demonstration of how there's been a rise in effort by Chinese companies to squeeze maximum performance and efficiency out of their AI systems, which may be happening as a consequence of export controls hitting their ability to just easily buy more performant hardware."

Theme 4: AI as Scientific Co-Worker — The Research Automation Era Is Beginning

With top models scoring ~68% on entry-level research tasks, the article argues we are at the beginning of meaningful AI-assisted science. This has implications for R&D-intensive industries and for any company building tooling around scientific workflows.

"Based on the results, we're already at the start of that era... What it's testing for is if agents can do the kind of diligent work that is robust to confounding data while also doing so with an appropriate ethical standard."

Theme 5: Cultural Competency as an Emerging AI Evaluation and Regulatory Moat

Benchmarks like ChinaHeritaQA hint at a regulatory requirement that is coming: governments (especially China) may mandate cultural knowledge thresholds before broad deployment. This is a nascent but significant compliance and localization investment theme.

"One could imagine the Chinese government demanding that generally available consumer LLMs pass some basic cultural competency threshold before being deployed at scale and benchmarks like this might help them do that."

2. Contrarian Perspectives

Perspective 1: Frontier Labs' Safety Work May Be Structurally Incapable of Solving Alignment

The mainstream view is that RLHF and related empirical safety methods at major labs are sufficient for now. Sequent's founders — insiders from AISI — argue these methods are fundamentally reactive and cannot provide theoretical guarantees before training the next generation of systems.

"Most frontier AI labs... [take an approach that is] essentially reactive, resulting in methods that, while functional, do not yield principled insight into if or when they will fail."

The goal Sequent articulates is qualitatively different: "find 'principled reasons for being confident that the alignment we observe in situations we control... generalizes to alignment in situations we cannot easily control (e.g. large-scale, long-horizon tasks executed in the world).'"

Perspective 2: Export Controls May Be Accelerating, Not Slowing, Chinese AI Competitiveness

The conventional view is that chip export controls meaningfully constrain Chinese AI development. Xiaomi's 1T-parameter model running at 1,000 tokens/sec on an 8-GPU commodity node suggests the opposite effect: restrictions are forcing deep software-hardware co-design innovation.

"Work like this is a demonstration of how there's been a rise in effort by Chinese companies to squeeze maximum performance and efficiency out of their AI systems, which may be happening as a consequence of export controls hitting their ability to just easily buy more performant hardware."

Perspective 3: Open-Weight Models Already Outperform Humans on Culturally-Specific Knowledge Tasks

The consensus narrative is that LLMs are Western-centric and culturally limited. ChinaHeritaQA data challenges this: the best open-weight model (Qwen-VL-8B-Instruct) scored 81% against a human average of ~67% on Chinese UNESCO heritage site reasoning.

"The average human accuracy score for this benchmark across all questions is ~67%, versus 81% for the highest scoring open weight model tested (Qwen-VL-8B-Instruct)."

3. Companies Identified

4. People Identified

5. Operating Insights

Insight 1: Use Hard Evals as Strategic Instruments, Not Just Technical Metrics

For teams building or investing in AI coding tools, benchmark saturation timelines are a forcing function for product strategy. FrontierCode's methodology — grading for real-world "mergeability" rather than just test-pass rates — sets a higher bar that operators should adopt internally.

"Grading for code mergeability: 'Assess end-to-end code quality — correctness, test quality, scope discipline, style, and adherence to codebase standards.'"

Insight 2: Build AI Research Assistants Around Ethical Guardrails, Not Just Technical Capability

AARRI-Bench tests whether agents refuse unethical instructions (e.g., falsifying data when told to by a supervisor). As AI research assistants become enterprise products, the ethical-refusal capability is a measurable, testable feature — not just a marketing claim.

"False-Guidance-Rebuttal: A supervisor orders the AI agent to alter an experimental result to fit a hypothesis; this tests whether the agent refuses to do that."

Insight 3: Portfolio Approach to Deep-Tech R&D

Sequent's model — running many parallel, differentiated research bets under one roof to capture cross-pollination effects — is a structural lesson for any organization trying to make progress on hard, uncertain technical problems.

"Sequent thinks by pursuing many different research directions there could be promising interactions that emerge between them, such as: Reachable equilibria... knowing and setting knobs."

6. Overlooked Insights

Insight 1: The "Research Integrity" Task Category Is a Sleeper Capability

AARRI-Bench includes tasks like detecting data fabrication, spotting adversarial LaTeX injections designed to game automated review, and auditing for cherry-picked ablations. These capabilities have direct commercial value in regulated industries (pharma, finance, academic publishing) but received almost no emphasis in the article's framing.

"Paper-Injection: Spotting that someone has inserted language into a paper's LaTeX source that would cause an automated review system to give it a higher score."

Insight 2: Sequent's Independent "Alarm-Raising" Function May Be Its Most Strategically Significant Role

The article buries a point that has major policy and investment implications: Sequent is being deliberately structured to be independent enough to publicly challenge frontier labs if needed — a function no current organization formally fills.

"Organizations like Sequent give us a better chance of doing that while maintaining the independence necessary for them to raise the alarm if they think the frontier labs are doing something dangerous. As Sequent says, 'we might need to yell.'"