AI makes it more crucial than ever to measure what truly matters

Sketchnotes from Rosenfeld Media Advancing Research 2025 (by Manuel Herrera)

There’s a pressing need to measure AI adoption success not in terms of output, but outcomes.

A lot has changed since I gave this talk on “Coexisting with AI” at Rosenfeld Media’s Advancing Research in March, but this core principle hasn’t.

Measuring our success by our productivity paves the way for automation. And while I’m not against automation, we should not be using it in places where AI errors cause bad business or human outcomes.

We need to combat this trend by measuring what actually matters in our work, even when it’s harder than collecting productivity metrics.

⛔️ It’s easy to measure how many participants you talked to to inform your product strategy.

✅ It’s a lot harder to measure whether that strategy was any good.

⛔️ It’s easy to measure time saved using AI-moderated interviews.

✅ It’s harder to measure if those interviews identified any meaningful product issues.

Figuring out how to measure the longer-term, higher-value impacts of our work should be a priority for tech professionals in an AI world. This is the most effective strategy to ensure the safety of our businesses, our users–and also our jobs–because it lets us recognize what tasks AI is fit to take over, and where you need a real human.

🧩 Output ≠ value, and AI makes it more crucial than ever to measure what truly matters.

If you’re looking to get this message across in your own organization, consider booking me for a talk!

Thanks, as always, to Rosenfeld Media and Manuel S. Herrera for this wonderful talk artifact.