Revealing the infrastructure behind commercial AI research tools

For UXRs who care about using AI responsibly, Condens shared details about their AI pipeline with me:

If you’ve been following me, you know I’m a vocal critic of the indiscriminate use of AI in product research. So when I agreed to write an article on AI research tools for Condens, it was with the understanding that this would be an honest comparison of where AI tools work well, where they introduce risks, and where you need a real human researcher to do the job.

But comparing commercial AI tools turned out to be harder than expected–not because of LLMs, but because of industry secrecy.

Most AI research tools are built on the same few LLMs (think GPTs and Claudes). This means they share the same fundamental limitations regarding qualitative research, which I’ve written about elsewhere.

The differences in performance come from the prompts and infrastructure used to pipe data in and out of these LLMs. And this remains a closely guarded secret for most AI research tool companies. They rarely describe their AI processing at even the most rudimentary level, let alone details of *how* it’s done.

This is an enormous problem when trying to compare research tools. If you can’t know what’s happening with your research data once it goes into the tool, you can’t know whether that process can be trusted to yield reliable results–nor how that compares to any other competitor.

So, I asked Phillis Chang if Condens could do something different: share a basic account of the specific safeguards involved in their AI research features.

It took multiple meetings to decide what could be shared, but Condens delivered. Several details made it into the article, and more were left on the cutting room floor (maybe for a follow-up??).

There are no prompts; and you won’t be able to reverse engineer anything from the information provided. But given the state of the industry, ANY public discussion of engineering safeguards for research integrity is a bold step forward. Tech companies routinely share entire system prompts and their code specifically to engender trust. I’d like to see more research toolmakers follow suit.

This is a small but vital aspect of our article on the strategic choices involved in selecting AI research tools. You can read it here.

Many thanks to Phillis, a tireless collaborator on what ended up being a much more involved project than either of us expected.