LLM Inference Infrastructure

Open-source inference engines commercializing into production platforms

The most compelling capital formation in LLM inference infrastructure is now happening around teams that built foundational open-source engines and are now commercializing them. RadixArk, founded by former xAI and Nvidia engineers, raised a landmark $100M seed round to commercialize SGLang — already deployed across 400,000+ GPUs for Google, Microsoft, and Nvidia. Similarly, Inferact was built by the core vLLM team (vLLM now powers a significant share of production inference globally) and is positioning itself as a universal inference layer. This pattern — open-source credibility converted into enterprise infrastructure businesses — is defining the category's top-tier deals and suggests that 'build in public, monetize in private' is the dominant go-to-market playbook for inference infrastructure in 2026.

Token spend is becoming a board-level line item — Uber's CTO reportedly burned through the full 2026 AI budget on token costs alone, and signal [1] captures the broader claim that AI token spend may soon rival headcount costs. This is spawning a dense middleware ecosystem: OpenRouter (routing across multiple AI models), Respan (unified gateway across 1,000+ models with observability and cost controls), Tokenwise (LLM proxy that recommends model substitutions based on actual usage patterns), Auriko (~30% cost reduction via inference path arbitrage), Shiba Code Labs (EU-friendly router with on-device data redaction), and Opper AI (EU-hosted gateway across 300+ models). Crucially, signal [4] reveals that on OpenRouter the top six most-used models are open Chinese models from Tencent, Xiaomi, DeepSeek, MiniMax, and Z.ai — underscoring that routing infrastructure has become the neutral layer enterprises trust precisely because it abstracts away model provenance.

Why it matters · Any platform sitting between enterprise spend and model providers on a per-token basis commands outsized margin potential and deep switching costs once observability data accumulates.

Below the routing layer, a new cluster of companies is attacking inference inefficiency at the silicon-software boundary. Standard Kernel automatically generates optimized GPU software kernels for AI workloads. Tensormesh uses KV-caching to eliminate redundant computation, and its $20M seed was backed by AMD Ventures, CoreWeave, and NVentures (Nvidia's venture arm) — a rare tri-party strategic syndicate signaling that chipmakers are hedging across the stack. Tile AI's TileRT software enabled Xiaomi to achieve 1,000 tokens/second on commodity hardware, and ZeroGPU claims 10x speed and 50% cost reduction by routing production tasks to efficient edge models. Amazon's Trainium 3 chip (signal [29]) is described as a step-change over Trainium 2, and Etched is baking neural network weights directly into custom silicon for maximum throughput.

Why it matters · Operators who win at the kernel and chip layer can offer structurally lower inference costs, making efficiency a durable competitive moat rather than a temporary pricing lever.

Signal [4] and [12] together document a significant shift: the top six most-popular models on OpenRouter are open-weight models from Chinese firms — Tencent, Xiaomi, DeepSeek, MiniMax, and Z.ai — and Chinese open-weight models are now capturing developer mindshare at a scale that directly challenges OpenAI and Anthropic. This is not merely a model-layer story: it is an inference infrastructure story, because serving these models efficiently at scale requires the same routing, caching, and orchestration tooling being built across this theme. Oxlo.ai's multi-model API already lists DeepSeek and Kimi as headline models in its 35+ frontier model catalogue.

Why it matters · Infrastructure vendors that remain model-agnostic and cost-optimize across Chinese open-weight models alongside Western frontier models will capture the largest addressable developer base.

The theme's velocity metric of -0.40 reflects a genuine deceleration in deal cadence — the week of July 13 saw only $400M across 2 deals versus $25.4B across 20 deals in the week of June 15. However, the stage mix tells a more nuanced story: Series D+ deals (9 rounds) have already absorbed $26.4B, strategic rounds (5 deals) account for $25B, and 'unknown' stage rounds represent $52.4B across 78 deals — suggesting that the largest capital pools are moving into late-stage and strategic instruments rather than drying up. Hydra Host's $100M Series A and RadixArk's $100M seed both closed in this environment, confirming that conviction capital continues to flow to infrastructure even as spray-and-pray seed activity cools.

Why it matters · For investors, the cooling velocity signals a barbell market: only companies with clear production deployments or proprietary efficiency claims will command premium valuations in the next 12 months.