Advancing Research 2025
Co-existing with AI: A practical guide for researchers
Thanks for joining me at Advancing Research 2025!
References from the talk
-
AI for UX researchers (a workshop I’m conducting through Rosenfeld Media)
-
-
-
More on AI from LLewyN
Agents like OpenClaw aren’t just a design tool; they’re your new user. And I’m on tour teaching UX practitioners how to design for them!
LLMs are getting better at qualitative analysis tasks. But when errors do show up, they're often important. And they’re now almost impossible to catch by just eyeballing the output.
I am thrilled that we can finally share the speakers for Rosenfeld's Designing with AI 2026.
This should be a wake-up call for all of us using AI as a research tool. We can’t trust that our familiarity with the data is enough to protect us from making mistakes.
Many of the best-known AI research influencers were first on the scene, building their AI brand and selling training before any rigorous validation had been conducted. What are the consequences of this?
Research leaders: When you choose AI tools, are you thinking about how it will impact your star performer, or someone who’s never done research before?
Workshop announcement: I'll be giving a strategic workshop on Agent Experience at UXDX in NY in May.
The real differentiator today is not being able to operate AI tools, but understanding how they work. A true research expert should recognize how to evaluate them critically, and develop discernment in where and how they’re applied.
What's the business impact when AI lets everyone generate research deliverables? And how can research leaders get ahead of this?
If you could get an inside look at how AI-moderated interviews work under the hood, what would you want to know?
"AI analysis ≠ qualitative analysis" is one of most misunderstood things I teach about AI for UX research. I'm not saying that AI analysis is *worse quality* than human qual analysis. I'm saying that if you look under the hood, they're fundamentally different processes.
Our workshop tools are designed to give you a transparent view inside the black box of commercial research tools. With our new tool, you get direct access to the prompts that guide the AI moderator, so you can see for yourself how changes impact your results.
The prompts and infrastructure behind commercial AI research tools are a closely guarded secret. But if you can’t know what’s happening with your research data once it goes into the tool, you can’t know whether that process can be trusted to yield reliable results. I asked if Condens could do something different: share a basic account of their specific AI safeguards.
I’ve been updating my AI for UX Researchers workshop with new content for 2026. Here’s a sneak peak of the outline.
Teaching general principles only gets you so far when you’re working with tools that change as rapidly as AI. I see my workshops as less about teaching static best practices, and more about teaching researchers how to think about AI–and that starts with learning how to experiment with the latest models and assess the quality of their output.
As curator of an AI+UX conference, here’s what I’d love for you to submit to our 2026 Call for Proposals (closing Sunday, 1/25!).
Choosing how to design with AI is the most important design decision you can make in 2026.
If you agree, you should submit a case study to Designing with AI 2026!
Verifying qualitative research analysis falls into Andrej Karpathy’s category of “lagging” AI tasks, depending, as it does on “real-world knowledge…context, and common sense.” There’s no definitive “correct answer” to train a computer by. So what can we use AI for in product research?
US courts have just seen their first documented case where a deepfaked video was submitted as evidence. How long before we’re consistently seeing deepfaked participants completing usability tests and interviews?
Since the beginning of LLM hype in November 2022, it’s been clear that AI can’t stay in chat windows forever. To solve real problems, we need to bring AI into the physical world. One way of doing that is with vision language models (VLMs).
Nobody wants to be their team’s AI “speedbump,” even if it’s better for the company and their users. Here’s what World Usability Day speakers recommended in this situation.
Happy World Usability Day! I’m presenting on AI Privacy & Trust at the Austin event through UXPA Austin.
Last year Vitorio Miliano and I built a demo using AI to redact biometric data from user research videos. Vitorio built a new, expanded, agent-based version that: 1) runs entirely locally, 2) “understands” entire scenes using a perceptive-language model (Perceptron's Isaac).
Revisit Designing with AI 2025 with a new Rosenverse playlist. Our focus at DwAI25 was twofold: sharing AI best practices, and looking ahead to the future of design with new AI use cases. In the five months since the conference, we've seen AI and UX evolve rapidly, but the guidance of our DwAI25 speakers remains relevant.
I recently debuted a new talk on AI best practices for research. Here are the slides that got the biggest reaction.
We tested Google’s Gemini 2.5 Flash (used in NotebookLM) on qualitative analysis by running the same prompt multiple times. It improved on GPT-4o and Claude Sonnet 3.5 at quote accuracy but still struggled with relevance and meaning.
Alex Allwood is one of my past AI+research workshop attendees, and she’s written up a great account of her learnings.
NotebookLM is useful, but it’s still just LLMs deep down. There’s no magic technology that makes it immune to the problems of other AI systems.
Seen last week on social media: a market researcher trying to save their job after reporting a nonsense, LLM-generated analysis of survey data. It’s a common mistake, and there’s not much training available for researchers on how to account for differences between generative AI and traditional research tools.
The reality is that many researchers are now expected to use LLM-based tools in their work, despite their shortcomings. And it can be hard to push back on mandates from above, or competition from peers who don’t understand the risks. Are there ways we can integrate LLMs into research while still being responsible professionals about it?