What does it mean to “verify” qualitative research analysis for AI?
What does it mean to “verify” qualitative research analysis for AI? Is it even possible?
A recent post by Andrej Karpathy illustrates one of the reasons why using LLMs for user research is so difficult:
“Software 1.0 easily automates what you can specify. Software 2.0 easily automates what you can verify.”
“Tasks that are verifiable progress rapidly, including possibly beyond the ability of top experts (e.g. math, code…anything that looks like puzzles with correct answers), while many others lag by comparison (creative, strategic, tasks that combine real-world knowledge, state, context and common sense).”
“If [a task/job] is not verifiable, it has to fall out from neural net magic of generalization fingers crossed, or via weaker means like imitation.”
Verifying qualitative research analysis falls into Karpathy’s category of “lagging” tasks, depending, as it does on “real-world knowledge…context, and common sense.” There’s no definitive “correct answer” to train a computer by.
But that doesn’t mean qualitative research is purely subjective either. When social scientists verify qualitative analysis, they look for five things (per Miles & Huberman, 1994):
⚖️ Objectivity
🛡️ Reliability
❤️ Authenticity
↔️ Transferability
🏃♂️ Actionability
How do you train an AI judge to determine if output is “authentic” or “transferable”? Spoiler alert: You can’t, at least not 100% reliably with the tech we have right now.
And that’s why many researchers are skeptical about using AI as a 1:1 replacement for human qualitative analysis.
So what can we use AI for in product research?
Quite a lot! As long as it’s *verifiable* (e.g., code), or *disposable* (e.g., prototypes).
These are the methods for product teams who want to use AI in their product strategy, without compromising the business value of their insights.
To learn more about these methods and when to use them, book me for a private workshop! (It’s a great end-of-year event for a team.)
📅 Grab time on my calendar to chat about your needs: https://book.llewyn.net