“A mixture-of-experts model with roughly 32B total parameters, 32x sparsity was trained across two clusters about 9,000 miles apart.”
AI-extracted from podcast / newsletter / paper summaries. May contain errors.
“Startup co-founded by former Google distinguished engineer Rohan Anil.”
Source→“Core Automation is a lab created to build models that continuously learn from deployment. We believe that Transformers are incapable of continual learning. There's no way how to put continual learning on Transformers. So we know we have to find a different architecture.”
Source→“Co-founder of Core Automation; former VP at OpenAI where he led the Strawberry and reasoning teams and was central to scaling reinforcement learning.”
Source→“Core Automation is a lab created to build models that continuously learn from deployment... Our quest is to find that new architecture, find that Transformer replacement. And we want to build the most automated lab to do it.”
Source→“An example of this is our QR kernel competition that we hosted with GPU Mode... If we use the competition winner's QR, you get 60x faster.”
Source→“Co-founder of Core Automation; one of four pre-training leads on Gemini; led fundamental AI research at Google Brain; co-created the Shampoo optimizer; was at Anthropic before founding Core Automation.”
Source→“"Core Automation's founder Jerry Torik led the OpenAI o-series — he made many contributions in the reasoning domain." [00:41:55]”
Source→