Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone

1. Key Themes

The Last 10% Is the Whole Business in Essential Industries

The gap between academic reliability and production reliability is the central tension of the episode. A model that looks strong in a paper becomes a flood of failures at real-world volume, and in housing and healthcare the failures land on people's lives. Martone contrasts how academics and operators think about the same number. He says: "When they are talking about reliability they're like, 'Okay, if the LLM can take it up to 90%.' And then we say, '90%? That means for like 60 million of phone calls per month per year means like an enormous amount of failures, that is totally unbearable.'" [00:08:37]. Marina Mogilko adds the industry joke that "every nine that you add is another hill to climb" [00:08:29]. Martone also notes the stakes: "The decisions that LLMs make for Elise for our products they're not just like some random things that appears on the screen, but like drive impact some people's life, and that could be very positive or very negative." [00:04:20]

Match the Tool to the Problem: LLMs Are Not the Answer to Everything

EliseAI runs "a full spectrum of tools." Renewal prediction, for example, uses in-house deep neural networks, not LLMs, because the company has abundant structured data and LLMs lack determinism and good probability handling. Martone says: "We have a mixture of cutting-edge AI and kind of more LLMs frontier based, but we also use a lot of traditional machine learning. Because we have a ton of data, and so for that LLMs are not really the answer." [00:05:40] And: "The predictions of renewals we're using really elaborate kind of deep neural network that we build in the house, but it have nothing to do with LLMs." [00:05:58] He critiques newer hires who default to LLMs: "Many people who have joined VR very recently, they just do have one solution for everything, LLMs, right?... Having as you mentioned determinism is not there. Like trying to run the same behavior given that input, LLMs fail. And probabilities is also something that the LLMs are really hard time to elaborate." [00:07:15]

Evals at Scale Are the Real Product Discipline

Small-sample testing hides divergence between methods. Martone says: "If you just like try at small scale, like five or six conversations, all look very quite quite good, but at scale, when you get to the millions, then you see that different methods diverge significantly." [00:09:51] The team treats the product like a scientific experiment, with evaluations that let roughly ten people understand daily how millions of automated conversations are going: "Building those evaluations that we are able to really gather with few people, with 10 people, like that are able to like get an understanding every day how the conversations are going, that's a really, really hard task, which we have largely cracked, to be honest." [00:16:51] The interviewer frames it as "applying the scientific method to product design" [00:10:10], and Martone agrees fully.

Trust Your Own Benchmarks, Not the Internet's

EliseAI built an internal experimentation loop so that when a new model or technique appears, they test it on proprietary benchmarks instead of believing public claims. Martone describes it: "Somebody on Slack messaged me like, 'Dude, like I saw this post on X.' I don't really have to believe it. I don't even look at the benchmark. The thing we do is like, 'Okay, let's set it up. Let's run an experiment internally.' So, we can actually see a performance of every model on our own internal benchmarks versus the other ones that you kind of don't trust." [00:12:17] They are working toward automating the loop fully: "We do have some humans in the loop, but we're working on automating it actually fully." [00:12:58]

Owning the Full Voice Stack Is a Compounding Advantage

Voice is brutally hard: "You're talking about doing tens, if not hundreds of operations in the span of like a second and a half." [00:05:20] EliseAI was early, "for three years," and "we definitely started when it was way too early... trying to make it work when the technology wasn't there." [00:05:01] Martone built the stack largely himself and argues that full ownership is the advantage: "We built it entirely on our own, so we have full understanding of every single parts of it." [00:11:18] That lets them see bottlenecks precisely and slot in new speech-to-text, text-to-speech, or fine-tuned open-source components quickly.

Fine-Tuning Open-Source Models Has Become a Strategic Investment

Over roughly the last eight months, the team invested heavily in fine-tuning open-source models. Martone says: "One of the thing that we invested on heavily in the last eight months or so is to try open source models... we set a group of researchers and we started heavily tackling that, and we have been able to actually get incredible results with training our own models and delivering it to production." [00:14:02]

Forecast What LLMs Will Solve in Six Months, Then Move Early

A core strategic discipline is predicting capability frontiers and committing resources ahead of them. Martone says: "Six months from now, what problems will be solvable with LLMs that are not solvable now? And if then push it to a year from now... It's a high-stake decisions, because moving early and moving fast allows us to be in a much better position six months from now." [00:15:25]

End-to-End Automation of the Customer Journey (Almost)

EliseAI has automated leasing from website browsing through move-in. Martone says: "They basically can do the entirety of starting from like looking at a website and deciding that they are interested in apartments to moving in, it could be literally fully automated. We have for example you can book a tour, then the AI sent you the code, you can go to the house, then you open the house with the code, you're interacting with the AI, the AI is explaining you the house." [00:17:25] Maintenance is mostly automated, with the physical fix as the last gap: "The AI is capable to understand whether or not it requires a fix immediately, then escalates it, finds a vendor, calls up a person, and then the last piece is a person, actual physical person coming into your house, fixing it. Maybe we'll get to like automated the whole thing in a year or two." [00:18:29]

Scale and Market Size

EliseAI handles "about 5 million phone calls a month. So, about 60 million a year" [00:10:57], plus "1 million phone calls a day, hundreds of thousands of email and SMSs" [00:16:29], with only "a few hundred people in the company" [00:16:06]. Martone frames the market: "Healthcare and housing it's like 40% of the GDP in the US." [00:19:30]

2. Contrarian Perspectives

Production AI May Be Harder Than String Theory

Martone, a physicist who spent 15 years on black holes and string theory, argues that the complexity of real-world production AI exceeds that of theoretical physics. "I did not expect that, but even after coming from string theory, I would say the complexity of the problem we are working now, is probably higher now in Elise than it was before." [00:00:00] Most people assume frontier research is the hardest work, but he argues consequence-bearing deployment at scale is harder.

Choosing Industry Over a Frontier Lab for Impact

Martone passed on the obvious prestige path for a top AI researcher. "One alternative I could've done... is for example try to join one of these like frontier lab... But one thing I really wanted is to have an impact in the real world." [00:03:45] The contrarian read is that the highest-leverage research talent may sit in vertical application companies where decisions carry real consequences, not in the labs.

Don't Believe Public Benchmarks, and Don't Chase Every New Model

Where the industry treats leaderboards as truth, Martone ignores them: "I don't even look at the benchmark." [00:12:38] The edge comes from proprietary evals tied to the company's own real conversations, which suggests a durable moat in evaluation infrastructure, not model selection.

Deep Thinkers Have an Edge Because AI Is "A New Field Every Day"

Martone argues the fast pace of AI rewards slow, deep thinking: "The people who are able to really think deeply about the problems, can be very creative, can constantly kind of come up with new ideas have a huge edge, because AI is like a new field every day." [00:02:51] That cuts against the notion that speed alone wins in AI.

Being Way Too Early on Voice Was a Feature, Not a Bug

Martone frames three years of working on voice before the technology was ready as a strength. "We definitely started when it was way too early. So, we're trying to make it work when the technology wasn't there." [00:05:01] Years of production learning compounded into a full proprietary stack.

3. Companies Identified

EliseAI

Conversational AI company for essential industries, primarily housing (leasing, maintenance, renewals) and now healthcare. Mentioned throughout as the subject. Operating at significant scale: "1 million phone calls a day, hundreds of thousands of email and SMSs" [00:16:29], with workflows automated "from looking at a website... to moving in" [00:17:25]. It is also expanding into affordable housing: "We are finding that affordable housing is a hugely complex yet extremely impactful... The amount of tech depth that is in the currently in the industry is nearly nonexistent." [00:20:10] Martone describes its team composition as a unique advantage: "We have top academic talent, you know, including like myself, professors of top universities, as well as like top developers." [00:09:12]

Frontier AI Labs (unnamed)

Mentioned as the alternative path Martone considered. Marina notes EliseAI was working on voice "way before some of the frontier labs were working on that" [00:10:25], signaling that EliseAI's voice stack is ahead of or in line with the labs' offerings in its niche.

Affordable Housing Operators (an industry opportunity)

Not a named company, but a sector Martone flags as massively under-automated: "It's basically not automated at all. The amount of tech depth that is in the currently in the industry is nearly nonexistent, and it's a massive component of the entirety of real estate." [00:20:32] This is a potential greenfield for vertical AI.

4. People Identified

Mario Martone

Head of Applied Research at EliseAI. Holds two PhDs, spent 15 years in theoretical physics (black holes, string theory), and was a professor at a top London university. He built EliseAI's voice stack largely himself and leads the fine-tuning push. Mentioned as the guest and as an exemplar of the researcher-turned-operator. He says: "I mostly worked for 15 years on trying to understand the universe, black holes, string theory." [00:02:20] On his voice work: "I built it mostly, largely myself. So, I have a really unique vision." [00:11:18]

Minna Song

CEO/co-founder of EliseAI (referred to by the hosts as the prior guest from the New York Stock Exchange conversation). Mentioned for her vision, particularly the early bet on voice. Martone says: "That's one of the things I love about Minna and Tony and their vision. We definitely started when it was way too early." [00:05:01] Marina notes their conversation on essential industries: "We just had Minna Song on last week at the New York Stock Exchange." [00:03:19]

Tony

Co-founder of EliseAI, mentioned alongside Minna for the vision that drove the early bet on voice. Martone says: "One of the things I love about Minna and Tony and their vision." [00:05:01]

5. Operating Insights

Make the Model-Choice Decision Early and Explicitly

Martone's team evaluates every problem on two axes before choosing a technique: impact and speed to near-optimal. "Look at a problem, try to understand two things. Like one, what's the most impactful? And then two, because we also have to ship at a very fast pace, okay, what's the solution that can really get us almost as close as possible to the optimal outcome, but in the fastest way possible? And then we have to double down on that." [00:07:53] Choosing between an LLM, a deep network, and simple ML is a leadership decision made early, not left to engineer preference.

Build an Experimentation Loop That Turns Rumors into Results in Hours

Treat every external claim as a hypothesis, and make testing cheap. The Slack-to-experiment pipeline means anyone in the company can surface a new model or paper and see it run on internal benchmarks. "A very close loop where something new change happens... We can actually see a performance of every model on our own internal benchmarks." [00:12:17] Automating this loop converts market noise into a systematic advantage.

Mix Academic Depth with Engineering Throughput, and Hire for "Fire to Impact"

Martone's hiring filter is researchers who are driven by real-world impact, which he calls "unicorns to find." "People who have exceptional academic background, but yet they are really interested in driving that into real world. And so, researchers with a clear focus on impact people's life and make that better." [00:22:17] He also needs engineers: "There's a ton of engineering that needs to be done, and we are so short on that." [00:21:31] Building teams where researchers and top developers share eyes on the same problem lets them cover both coding and academic questions.

Concentrate Hiring in Two Hubs

They hire technical talent mostly in San Francisco and New York for the highest bar: "The reasons why we are mostly hiring, especially on the technical side, in San Francisco and in New York, is because we want the best." [00:21:31] The second hub in San Francisco is growing quickly, but "we just can't grow hard enough." [00:21:56]

Let Clients Drive the Next Big Bet

Rather than speculating about the next vertical, Martone says: "We want to be driven by what is the next big bet based on what the client our clients say." [00:20:10] Product direction is customer-pulled, which is how affordable housing emerged as an area to attack.

6. Overlooked Insights

The Robotics Gap Is the Last Mile of Housing Automation, and It Is a Named Opportunity

In passing, Martone notes that maintenance automation stops where physical labor begins: "There still is a gap that we have not been able to automate, because we need robotics to fully automate that, because some of the fixes require actual human labor, and so LLMs can't really do that." [00:18:08] He then says "maybe we'll get to automate the whole thing in a year or two." [00:18:29] The significance is that the AI layer now handles triage, escalation, vendor selection, and calls, so the remaining unautomated step is purely physical. That makes robotics for property maintenance a clearly defined adjacent investment theme, with an existing software orchestration layer ready to dispatch it.

Affordable Housing Is an Untouched, Zero-Tech-Debt Market

Martone's offhand description of affordable housing carries a large implication: "It's basically not automated at all. The amount of tech depth that is in the currently in the industry is nearly nonexistent, and it's a massive component of the entirety of real estate. If we could bring down the cost of affordable housing significantly, so that will impact consumers immediately." [00:20:32] The industry combines high complexity, heavy regulation and compliance workload, and no incumbent automation. That is the kind of environment where a vertical AI player with proven scale in conventional multifamily can reduce operating costs materially for the least-resourced renters.