What’s Your AI Agent’s ROI?
1. Key Themes
Theme 1: Standard agent metrics (tokens, tasks, time saved, output) measure consumption, not value
Tokens measure consumption without context
Token counts can't distinguish valuable work from waste, so a big number is ambiguous.
"5 million tokens could either be the month's most valuable work or it could just be an agent running in circles."
Tasks completed hides quality and difficulty
Task counts ignore downstream failure.
"A thousand closed support tickets aren't really worth much if a third of them end up getting reopened."
Time saved is unreliable because people misjudge it
The METR study showed a gap between perceived and actual productivity, though the author hedges on its scope.
"16 experienced programmers took 19% longer on real tasks when they used AI tools. Afterwards, they believed AI had made them 20% faster."
"The actual conclusion is that people are just bad at guessing their own time saved."
Output can rise while quality and stability fall
"Google's DORA research linked a 25% rise in AI use to a 7.2% drop in delivery stability, meaning more releases that broke something."
Theme 2: Measure ROI per goal and workflow, not per session, with full-cost accounting
Session-level views misattribute spend and results
Agents spawn sessions on their own, so the unit of measurement must be the outcome.
"The session the engineer started cost only about a fifth of that. Both fixes came from sessions the agent opened on its own. But if you look at one session at a time, you misread both the spending and the result."
Multi-goal runs are rare but dominate spend
"Only 17% of runs pursued more than one goal, but those runs took 58% of the spending."
True cost goes well beyond the model bill
"The model bill is what you get to see but once you add setup, upkeep, the people who review its work, the fixes when it gets things wrong and the runs that produce nothing, you start gaining real, complete insights."
A five-part framework per workflow
"The best thing to do is to pick one workflow, meaning one repeated job the agent does at each time."
The five steps are Objective, Method (baseline), Cost, Evaluation, and Progress. Objectives need specificity: "'fix billing bugs within 48 hours' is better" than "fix bugs."
Theme 3: Cost-optimized agent metrics can mask quality erosion (the Klarna lesson)
Good dashboards, worse service
"Every number on the dashboard looked good. The CEO still ended up saying quality had suffered because cost drove the decisions."
Cheap unit economics can be illusory
"Say a support agent costs $1.10 per resolved ticket, now that is an opportunity. Then the share of tickets customers reopen climbs from 8% to 14%. Each reopen means a second contact, which can be with a human."
"The unit you measure has to fit the job."
Theme 4: Most enterprises can't demonstrate agent ROI, creating cancellation risk
The measurement gap is widespread
"Gartner expects over 40% of agentic AI projects to be cancelled by the end of 2027, with unclear business value among the reasons."
"IBM surveyed 2,000 CEOs in 2025, they said only a quarter of their AI projects had delivered the return they expected. In the same survey, 68% said they had clear metrics for measuring it. Both can't be fully true."
"Just 16% of companies saw a positive return on more than half of their AI projects."
Measurement, not capability, is often the bottleneck
"The agent often isn't the problem as Ramp's agent writes three quarters of the code the company accepts, yet Ramp still had to build a record before it could say what any of that work was worth."
Theme 5: Agent-native infrastructure (data consolidation) is an emerging category
Agents query differently than humans
"Most CRMs grew up around people clicking through records, while agents query constantly and in parallel."
Scattered data inflates agent cost
"An agent that works across a dozen scattered tools pays for every lookup, and the bill shows up in your ROI."
Note: this section is sponsored content from Attio, so the claims are vendor-reported (e.g., "what Attio says is 1,000x the agent traffic of a legacy CRM").
2. Contrarian Perspectives
"Time saved" is the worst usual metric, not the best
Most vendors and dashboards lead with hours saved. The author argues it is actively misleading because perception diverges from reality.
"Time saved is possibly the worst one out of the usual metrics. In a 2025 study by the research group METR, 16 experienced programmers took 19% longer on real tasks when they used AI tools. Afterwards, they believed AI had made them 20% faster."
The author adds a caveat: "this was a small group of testers, so it doesn't really prove that AI slows everyone down."
More AI adoption can reduce delivery quality
Higher usage is typically treated as progress. The DORA finding suggests more output can mean more breakage.
"Google's DORA research linked a 25% rise in AI use to a 7.2% drop in delivery stability, meaning more releases that broke something."
Even the poster child for AI replacing humans walked it back
Klarna's claim that AI did the work of hundreds of staff is widely cited as proof of replacement economics. The article notes the CEO's retreat, though Klarna framed it as an addition rather than a reversal.
"In 2025, its CEO admitted that focusing too hard on cost had lowered quality. Klarna began piloting a new team of human agents, which they called an 'addition' , instead of a reversal."
"The human pilot that made the headlines started with two people, while several thousand outsourced agents had been on the job the whole time. Since then the staff figure has passed 850."
3. Companies Identified
Ramp
- Description: Corporate card and finance software company
- Why mentioned: Case study in building the measurement infrastructure for agent ROI; its agent "Inspect" writes most accepted code
- Quotes:
"Ramp, the corporate card and finance software company recently set out to answer this question, exactly because they could see every dollar. But they could not see what every dollar bought." "Ramp's agent is called Inspect, and by August 2026, 75% of the code changes Ramp accepted into its products came from it." "Ramp hasn't published an ROI number. What it built is the record you'd need to calculate one."
Klarna
- Description: Payments company
- Why mentioned: Public cautionary tale of optimizing for cost metrics at the expense of quality
- Quotes:
"The payments company said its AI assistant did the work of 700 full-time staff, and later that number went up to 800." "At launch, Klarna reported answers in under 2 minutes instead of 11, with 25% fewer repeat inquiries."
Attio
- Description: CRM vendor (sponsor of this issue)
- Why mentioned: Launched a CRM designed for both people and agents; sponsored content
- Quotes:
"Today Attio launched its answer: A CRM built for people and agents in equal measure" "up to 100 million records per object, and what Attio says is 1,000x the agent traffic of a legacy CRM"
METR
- Description: Research group
- Why mentioned: 2025 study on programmer productivity with AI tools
- Quotes:
"16 experienced programmers took 19% longer on real tasks when they used AI tools."
Google (DORA research)
- Description: Google's DevOps research program
- Why mentioned: Linked higher AI use to lower delivery stability
- Quotes:
"Google's DORA research linked a 25% rise in AI use to a 7.2% drop in delivery stability"
Gartner
- Description: Research and advisory firm
- Why mentioned: Forecast of agentic AI project cancellations
- Quotes:
"Gartner expects over 40% of agentic AI projects to be cancelled by the end of 2027, with unclear business value among the reasons."
IBM
- Description: Technology company
- Why mentioned: 2025 CEO survey showing an ROI/measurement contradiction
- Quotes:
"IBM surveyed 2,000 CEOs in 2025, they said only a quarter of their AI projects had delivered the return they expected."
Battery Ventures
- Description: Investment firm
- Why mentioned: 2026 survey on AI project returns
- Quotes:
"Just 16% of companies saw a positive return on more than half of their AI projects."
4. People Identified
Ruben Dominguez
- Description: Author of The AI Corner newsletter
- Why mentioned: Writes the article and related prior pieces on agent economics (token waste, agent cost, pricing)
- Quotes:
"Your AI bill is mostly wasted tokens" (prior article by Dominguez, linked in the post)
Klarna's CEO (unnamed)
- Description: Chief executive of Klarna
- Why mentioned: Publicly admitted the quality cost of an aggressive cost focus
- Quotes:
"In 2025, its CEO admitted that focusing too hard on cost had lowered quality."
5. Operating Insights
Write the board sentence, then find the blanks you can't fill
Use a single defensible sentence as a forcing function. The blanks you can't complete reveal your measurement gaps, usually the baseline or full cost.
"'We spent $X in total for Y and accepted results at $Z each, against $W the old way, and [metric] moved by N.'"
"Can't fill in a blank? Start there. It's usually the baseline or the full cost, because failed runs and review time never show up on the vendor's bill."
Roll linked sessions into "complete runs," then split into goal-based work items
Ramp's method: group sessions, then use an AI model to break runs into work items (one goal, one result), each tied to a product, owner, and share of cost. The $93 run became three items.
"Ramp first gathers every linked session into one 'complete run,' so the $93 bug becomes 1 record instead of 8."
"A work item is one goal with one result. Investigating, editing and testing the same fix count as one item."
Pair every efficiency metric with a quality counter-metric, and discount reversed work
Track cost per unit alongside reopen rates or reversals, and don't count work that gets undone.
"A code change that gets undone within a week doesn't really count."
"Pick one number (cost per unit, volume handled, uptime or revenue) and track."
6. Overlooked Insights
Agents spawn their own sessions, which breaks human-centric cost attribution
The $93 bug involved 8 sessions across 3 codebases, and both shipped fixes came from sessions the agent opened autonomously. This implies that billing, tooling, and attribution built around a human-initiated "session" will systematically misstate what agents spend and produce.
"Both fixes came from sessions the agent opened on its own."
Value benchmarks for a single agent run depend entirely on counterfactual urgency
The $93 figure is meaningless without knowing what the bug was costing or how soon it would have been found, which suggests ROI measurement requires business-impact data (customer harm, detection time) that engineering tooling doesn't capture.
"If the bug was costing customers money everyday, $93 is cheap. If nobody would have noticed it for months, it's hard to call a bargain."