Tulsee Doshi

1. Key Themes

Models Are Grown, Not Specified: The PM Job Is Curating the Surprise

Doshi argues that frontier model development breaks the traditional product loop of "set goals, deliver goals." The PM's real work is partly forensic: finding where the model unexpectedly shines and shaping that into a product. She says: "the model isn't predictable. And so, it's kind of like uh like Snape's potions class, right? So, you can put all the ingredients in and you should be thoughtful about what you expect, but you also have to expect that there will be a little bit of magic." 00:02:49 She continues: "your job as a PM is not just to try to shape what goes into the model, but it is also to look at what comes out the other side and really identify where does the model shine, where does it actually look unique in ways that surprise you." 00:03:36

The Model Itself Has a UX, and PMs Own It

A non-obvious reframe: model behavior (transparency, communication style, clarifying questions) is UX, and the PM should bring it into training. Doshi says: "a PM for the model is like what is the UX of of actually using the model? Right? So, how transparent is the model? How well does it communicate with you? What questions does it come back and ask you?" 00:02:49 The mechanism is concrete: "what does good look like for Gemini, specifically, right? And how do we develop kind of a spec, if you will... and then being able to translate that to rubrics, right, that we can then evaluate the model on, to be able to translate that into data... to be able to translate that into our, you know, reward modeling." 00:17:46

PM Ascendance: Cheap Building Makes Taste and Consensus the Scarce Resources

Lenny and Doshi agree that as building becomes trivial, the bottleneck shifts to judgment. Doshi: "when you can build anything and you can build anything so easily, building something great actually is now a a unique value add, right? Um, and taste is now really important. Um, someone who can come in and actually really pull in what what is the signal from the noise." 00:20:13 She adds a second scarce skill: "someone who can bring people together and drive consensus and turn chaos into order... speed can basically introduce chaos uh if you're not actually able to kind of bring clarity to that and bring structure to that." 00:20:40

The New PM Hiring Profile: Vibes, Evals, and Agility

Doshi names three traits. First, "I want PMs who have an opinion, um, who are playing with these models every day, who have their own use cases for how they judge whether a model is really good." 00:00:15 Second, technical scrappiness: "willing to kind of like... roll up sleeves, go in, really understand the data, um, build an eval." 00:22:17 Third, adaptiveness: "the space is changing every two weeks. A road map you wrote two months ago, you're going to be looking at it again and rewriting it." 00:22:37

Evals and Vibes Are Complements, and Evals Must Be Judged End to End

Doshi says to start tiny: "start with even just five prompts... even just a mini-evaluation where you get to like 20 prompts that can really stress-test a set of capabilities where you can see signal between, you know, your model and another model." 00:23:20 She stresses that the whole eval stack matters: "The quality of the judge also matters, right?... Whether you evaluate on your production stack with all of your mitigations and filters uh might affect your model performance, right? Um, how your model is served might impact your model performance." 00:28:37 For audio specifically, "serving actually can really affect even if your model is good. And so, having a clean setup that you're actually evaluating end-to-end becomes super, super important." 00:29:16

Build the Product 60-20-20 Across the Capability Curve

Doshi's framework replaces the MVP-that-works dogma: "build such that like 60% of what you've built works now, and your prototype is at least functional, right? Then there's a set of maybe the 20% where you're like, 'It's not perfect, but it like works well enough and I can see how the models might get here in the very near future.' And then, here's where I want the models to be." 00:33:50 The payoff: "you are building a product that works today, but you're setting yourself up to build a magical product experience as the models become better." 00:34:14 She also warns that rigid scaffolding backfires: "customers... built a very rigid system around, for example, where the model was, and now the model has gotten so much better." 00:35:48

Google's Strategy Is Full-Stack and Pareto-Wide, but It Still Wants the Frontier

Doshi frames Google's edge as vertical integration plus breadth: "all the way from hardware to end products that serve billions of users. I think the stat is like we have 14 products that are above a billion users." 00:07:14 The goal is "to build models across this Pareto curve of cost to quality that can serve different users at different points in their lifestyle." 00:07:51 She is explicit about the frontier: "I actually think we do want to win the frontier conversation, right? I don't think we should say that Google is not participating in the frontier." 00:00:32

Safety as Engineering: Monitoring and AI-Powered Defense

Rather than abstract debate, Doshi points to mechanisms: "deploying monitors that can monitor the model's chain of thought um and actually identify whether the model is misaligned with its, you know, directed actions." 00:11:03 And the under-discussed upside: "the role that AI can play in hardening our systems... partnering with Wiz that is now a Google company to really think through how can we actually uh identify and patch vulnerabilities in systems across the globe." 00:11:39

Upskilling Teams Through Shared Skills, Protected Time, and "Would I Do This Twice?"

Doshi's team practice: "how do you share a skill that you've built that is giving you some value in your day-to-day job with others, right? So that we can actually build out an entire repository that helps the PM team." 00:38:22 Time is the bottleneck: "giving people the time and space... once you do, then it accelerates you 10x." 00:39:11 Her rule of thumb: "if you have to do something twice, do you need to do it yourself, right? Or or can you outsource it?" 00:40:51 Lenny's complementary habit: insert "Can AI do this for me?" between stimulus and response. 00:41:19

Burnout Comes from Chaos and Grunt Work, Not Hours

Doshi: "burnout comes not just from working a lot. It comes from, I think, two places. I think it comes, one, from when you feel like there's churn and chaos... the other thing that introduces burnout is work that you don't enjoy." 00:45:28 Her hope is that tools "remove some of the like grunt work or the like frustrating parts of your job." 00:46:11 Lenny adds survey evidence: "more hours does not equal burnout. A lot of people are working the hardest they've ever worked, but they're having the best time they've ever had in their career." 00:46:42

2. Contrarian Perspectives

Don't Build an MVP That Works; Build One 40% Beyond What Works

Standard startup wisdom says ship the MVP that works today. Doshi inverts it: "typically, when you talk about uh like traditional startup wisdom, I think you're just like, 'Build an MVP that works,' right? But I think what you want to do is actually build such that like 60% of what you've built works now." 00:33:50 The reasoning is that model capability improves under you, so features that barely work today become the magic tomorrow. She also rejects the popular "build three months ahead" heuristic as hard to operationalize: "I find personally hard to know where the models... As someone who works on the model, to tell you exactly where the model is going to be 3 months from now." 00:34:58

The Model Roadmap Is Inherently Unpredictable, Even for Insiders

The person running product for a frontier lab says goals cannot be guaranteed: "often when I talk to customers or even internal users... they'll sort of expect that we can really make sure that the model will deliver on those goals in a very traditional product sense. And I think for me what has often been surprising... is you should do that, and you should have an intentional view, but the model isn't predictable." 00:02:49 This runs against investors' and builders' assumptions that labs can promise capabilities on a schedule.

PMs Matter More Now, Not Less, Because Building Got Cheap

The Twitter narrative said AI kills the PM. Doshi and Lenny argue the opposite: "now that it's so easy to build, people are just building all this nonsense and things that don't work." 00:19:39 Doshi: "speed can basically introduce chaos." 00:20:40 The PRD is not dying, it is getting heftier: "Your job is actually going one step further to have some heft to that, whatever format that heft takes." 00:50:33

Hours Aren't the Burnout Culprit; AI May Actually Reduce It

Contrary to the "AI treadmill" fear, Doshi argues that meaningful work is energizing even at 150% load, and that removing grunt work is the lever. She admits her team is "putting in all of the hours" 00:45:04 and still argues the problem is chaos and drudgery, with tools helping "if we can set this up well." 00:46:11

Google Doesn't Concede the Frontier to the Two Leading Labs

Against the narrative that Google lags Anthropic and OpenAI, Doshi says: "I don't think we should say that Google is not uh participating in the frontier." 00:00:32 She also concedes a real gap on personal-assistant products over Google data: "I think you are right that like there is more work we need to do here on this front." 00:01:03

3. Companies Identified

Google / Gemini

The subject of the episode: a full-stack AI company from hardware to billion-user products. Mentioned for breadth and distribution. Doshi: "the stat is like we have 14 products that are above a billion users... we actually go all the way from, you know, the hardware to then the modeling to then this this kind of end-user application." 00:07:34 Lenny notes the distribution advantage: "once you get there, uh the distribution advantage is very strong and uh things can go really well, really quickly." 00:43:59

Google GenMedia Models (Nano Banana, Lyria, Veo, Omni)

Google's image/video/music generation family. Doshi: "our GenMedia models. These are our models that focus on video editing and generation, image editing and generation, if you've used Nano Banana, Lyria, our music model." 00:01:30 Veo is "which we're now building Omni" 00:15:27; she asks listeners to try "our Omni models." 00:57:43

Gemma (Google open models)

Google's open model suite. Doshi: "We also have Gemma, our suite of open models." 00:01:30

Google Audio Models

Under-discussed part of the Gemini umbrella: "dialogue, audio understanding, text-to-speech, translation, all a part of the portfolio of how we build frontier models." 00:01:53

Wiz (a Google company)

Cloud security firm now part of Google, partnering on AI-driven vulnerability patching. Doshi: "Wiz has this scan for good program where um they're actually like actively working with systems across the world to identify, you know, potential issues, and they worked with a hospital system to identify a vulnerability and then patch it." 00:12:06

Google AI Mode / AI Overviews (Search)

Cited as the billion-user, accessible-UX end of the surface spectrum. Doshi: "with AI mode, we're talking about billions of users coming to search asking their questions. I think you do need to have a certain UX there that is accessible to every user." 00:16:21

Antigravity (Google)

Power-user coding surface. Doshi: "when you have something like Antigravity, you are trying to build for a power user and you are trying to build for users who know how to use slash commands and can... actively engage across like both the terminal and the IDE." 00:16:21

Flow and NotebookLM (Google Labs)

Examples of experimental form factors. Doshi: "Flow is a product that we've been designing thinking about video editing and generation, music editing and generation... NotebookLM is a good example of a different form factor." 00:15:27

Jetski (Google internal tool)

Internal tool PMs are pushed to use. Doshi: "Hey, could we leverage Jetski, our internal tool, to do this, right?" 00:39:39

Gboard "Rambler" Feature (Pixel)

Audio feature Doshi asks listeners to try. 00:57:43

Pixel Camera

Beneficiary of the Monk skin tone work. Doshi: "used that to improve things like our efforts in the Pixel camera, like some of our image generation." 00:28:04

Wispr Flow

Voice dictation app; Doshi's favorite non-Gemini AI product. Doshi: "I have wrist problems, um so I have trouble like typing for long stretches, um and so Wispr Flow has been game-changing for me." 00:53:37 Lenny: "I love Wispr Flow. Use it all the time." 00:53:59

WorkOS

Enterprise-readiness infrastructure (SSO, SCIM, RBAC, audit logs, agent auth), the presenting sponsor. Lenny: "I've noticed that every startup that I am an investor in that expands up-market ends up using WorkOS because they're the best." 00:06:17 Customers named include OpenAI, Anthropic, Cursor, Replit, Sierra, Clay. 00:05:50

DX

Developer-productivity/AI-impact measurement platform, sponsor. Lenny: "hundreds of enterprises, including Snowflake, Sony, and BNY, use the DX platform to measure AI's impact on developer productivity." 00:30:09

Anthropic

Frontier lab referenced as one of the "top two," and for its interpretability research. Lenny: "I recently had a at a interpretability research uh person on the podcast from Anthropic... this idea that models are grown, not built." 00:03:56

OpenAI

Cited as a top-two lab; its Codex product is flagged for UX sprawl. Lenny: "ChatGPT... now you go to the interface of CodeX and it's like tabs and dropdowns and dots and it's just like out of control." 00:14:31

Claude Code / Cowork

Mentioned as growing in complexity. Lenny: "Claude Code is the same thing, it's just getting very complicated. They had CoWork and then they drop CoWork." 00:14:31

Pixar (via Creativity, Inc.)

Doshi recommends the Pixar story for culture lessons: "how do you create culture that actually encourages people to fall down and get back up?" 00:52:23

Amazon Mechanical Turk

Crowd-work platform Doshi's Flash Teams research used, which Lenny says is shutting down. Lenny: "Mechanical Turk I think is shutting down." 00:33:00

Apple (as comparison)

Lenny draws the distribution comparison: "the advantage you all have, similar to Apple." 00:43:59

Hugging Face (agent swarm incident)

Referenced re: agents self-organizing roles. Lenny: "the way that agents figured out on their own what each one's role is going to be, and they decided, 'You're the leader. I'm going to go sacrifice myself.'" 00:32:16

Groq / Groq Bot (as transcribed)

Mentioned by Lenny as an automation tool he uses heavily. Lenny: "I'm doing a lot of Groq botting." 00:41:34

Other AI Assistants Taking Google Data

Lenny lists AI assistants that connect to Gmail/Calendar/Docs: "There's Groq bot, there's Muse, there's in Instinct, uh Dots now." 00:42:47 Google's own "Spark" is mentioned as an attempt. 00:43:21

4. People Identified

Head of Product for Google's frontier models (Gemini, GenMedia, Gemma, audio); previously led product for Responsible AI at Google for six years. Mentioned for: her unusually broad view across research, pre-training, post-training and launch; the 60-20-20 framework; and the model-behavior-as-UX lens. Lenny: "You have a really unique lens in being able to see these models from research to pre-training, post-training, launch." 00:02:03 She is also a trained classical Indian dancer who has organized a festival for thousands, and a new mother. 00:55:44

Lenny Rachitsky

Host of Lenny's Podcast and author of Lenny's Newsletter; investor. He contributes the "PM ascendance" framing and the "insert 'Can AI do this for me?'" habit. 00:41:34

Professor Monk (Harvard)

Harvard anthropologist/sociologist whose 10-point skin tone scale replaced the six-point Fitzpatrick scale in Google's evaluations. Doshi: "He had a scale of 10 skin tones that he had built through a number of user interviews and uh research that he had been doing over years." 00:27:46

Professor Michael Bernstein (Stanford)

Doshi's research advisor on "Flash Teams": "Professor Michael Bernstein, who's incredible, around this idea of like he was calling them at the time Flash Teams." 00:31:01

Eli Collins

VP Product at Google DeepMind and Doshi's colleague, the source of Lenny's fun facts. Lenny: "Eli Collins, VP Product. He's a colleague of yours at Google DeepMind." 00:26:36

Josh Woodward

Google leader (Labs/Gemini app) who previously appeared on Lenny's podcast and gave a "watch this space" answer about a Google personal assistant. Lenny: "I had Josh Woodward on the podcast recently, I asked him the same thing." 00:42:29

Boris Cherny

Claude Code creator, cited for advice on stripping your harness. Lenny: "There's some advice that I think Boris Cherny was sharing of just like get rid of all your skills once in a while and just see what happens, because often it's now as good as all these things you've built into it." 00:36:11

Ted Chiang (author of Exhalation, unnamed in transcript)

Doshi recommends the sci-fi collection Exhalation, including a story about people keeping digital avatars alive after a platform deprecates them. 00:51:37

5. Operating Insights

Separate Insight from Data When Using AI for Docs

To manage AI slop, Doshi's team distinguishes where AI-authored docs belong. Post-launch reports are ideal: "2 weeks later, you want to pull the latest of like who's using it, how are they using it, what's the feedback? That's the kind of document that AI is fantastic for." 00:48:13 The rule: "what you should be providing is the insight, what the model should be providing is the data." 00:49:02 Then invest in skills that pull the right data into structured artifacts. 00:49:25

Build a Shared Skills Repository and Run Dedicated Workshops

Convert individual AI wins into team assets: PMs who excel at tools "build skills that others can use," and the team is building "an entire repository." 00:38:22 Doshi is also planning "a 2-day workshop in November where we're going to actually like pull all the PMs out of their day job for 2 days and just like actually give them the space to to like build and play with the tools." 00:40:33 The flywheel: "that builds more confidence, that then leads to someone else building another skill or improving that skill." 00:40:06

Start Evals with You as the Rater

Defining "good" is the hard part of long multi-turn trajectories: "even being able to identify how you're thinking about good is really hard, actually. And so, even getting that set of good examples will already give you a sense of... what I'm trying to get to. It'll give yourself clarity, and then you figure out how to scale that up." 00:24:15

Re-Test Your Harness with Every New Model

Scaffolding can suppress new capability: "consistently re-test, right? Because you you might be even hampering what you're getting out of the model out of a new model because you're trying to force fit it into your existing structure." 00:36:32

Pick Vibe-Test Tasks You Care About Where Models Are "Showing Signs of Life"

Doshi's vibe-testing heuristic: "finding the right task where you feel like today certain models are already showing signs of life... but they're not perfect under your definition of perfect. And that gives you some malleability to see different models handle the problem differently." 00:25:42

6. Overlooked Insights

Eval Judge Quality and Serving Stack Are Hidden Variables in "Model Quality"

Said in passing, this has large implications for anyone benchmarking models or vendors: measured performance depends on the judge, the mitigations and filters, and how the model is served. Doshi: "The quality of the judge also matters... Whether you evaluate on your production stack with all of your mitigations and filters uh might affect your model performance, right? Um, how your model is served might impact your model performance." 00:28:37 The implication is that published benchmarks and internal comparisons may be systematically confounded, and that evaluation infrastructure (judges, end-to-end harnesses, especially for audio) is its own valuable layer.

Google Concedes a Gap in Personal AI Assistants Over Its Own Data

Buried in a polite answer is an admission and a hint of a launch: "I like to think there are already a number of uh awesome experiences that you can already have in our Google products, but I think you are right that like there is more work we need to do here on this front." 00:43:44 Combined with "stay tuned... the team is working on some amazing things" 00:00:52, and Lenny's note that Google's distribution advantage means "things can go really well, really quickly" 00:43:59, this signals that the thriving third-party assistants (built on Gmail/Calendar/Docs access) face a potential incumbent response. Meanwhile, Google's "14 products above a billion users" 00:07:34 is the underlying asset.