Error-checking AI research synthesis is getting harder

Good news, bad news: AI synthesis tools are improving, but their errors are much harder to spot.

Vitorio Miliano makes transparent, custom AI tools for my AI + UX research workshops. And we recently rebuilt our AI synthesis tool from the ground up.

The reason is both good and bad!

LLMs are getting better at qualitative analysis tasks. We’re seeing far fewer errors than we were a year ago.

But when they do show up, they're often important. And they’re now almost impossible to catch by just eyeballing the output.

Our new workshop tool supports AI evaluation work by making it easier to find these errors:

• 🔬 It visually highlights quotes that appear in the original transcripts (so you can find the ones that don't)

• 🙋‍♀️ It surfaces which participant actually said each quote (so you can find discrepancies with what's reported by the LLM)

• 📚 It does the same for source documents

And it still gives you direct access to the prompt used to analyze your dataset.

No other workshop lets you see the inner workings of AI research tools like this. But we think researchers need to understand AI tools at this level--because that's what gives you the expertise to go beyond simply operating tools, to strategic AI application and leadership.

Our next workshop is in less than a month. Consider joining us.