Qualitative analysis with AI: Strengths and limitations

As of 2025, research synthesis is one of the top reported use cases for AI by UX practitioners, but are large language models (LLMs) really up to the task?  In this workshop, you’ll learn a framework for evaluating LLMs in the context of user research analysis, and practice running an LLM-enabled analysis yourself.

The first run of this workshop sold out and generated a lot of attention on social media–read more here.


Target audience

Researchers (and people who do research) looking to understand the strengths and limitations of qualitative analysis using LLMs, for considered incorporation into their research toolkit

What you’ll learn

  • Sources of error in large language models (LLMs)

    • Prompt factors

    • RAG factors

  • Practice qualitative synthesis with a long-context LLM

    • Explore a test dataset

    • Discuss strategies for handling context limits

    • Review qualitative analysis prompting best practices

  • Evaluate output at scale and compare results

    • Compare AI-generated themes to human-generated themes

    • Assess accuracy

Attendee feedback

“I now have more talking points for leadership that using LLMs for research may have some value but may not actually save time to start.”

– Workshop participant

“I was surprised that [Llewyn] chose to do a ‘see for yourself’ experiment approach, instead of just telling us what AI is good and bad at.  That was amazing.”

– Workshop participant


“[Getting] hands-on experience with interview synthesis was great.  Having both the capability and a dataset with permission for use isn’t an everyday occurrence.”

– Workshop participant

 

Interested in hosting this workshop for your team?

 
Llewyn Paine