Why credible researchers don't "just ask the AI" how to use LLMs
There’s a widely circulated piece of bad advice when it comes to AI: “If you don’t know how to use AI, just ask the AI!”
Someone in my last AI for UX Researchers workshop had gotten this advice and wanted to know why it’s a bad idea.
(Note: I'm running this workshop again in October!)
The problem with this advice is that it assumes the AI “knows” about its own functioning. Spoiler alert: it doesn’t! (But it is good at making up plausible sounding answers!)
An LLM has no understanding of its own internal state.
It can’t introspect. This is why scientists have to invent techniques like circuit tracing to understand how LLMs create connections–because there’s no innate way for the LLM to tell us how it works under the hood.
So if it doesn’t know its own internal state, what does it know?
The LLM knows:
The patterns it learned from its training corpus (and any subsequent training)
Whatever’s in its system prompt
Neither of these things gives it any special knowledge about its own functioning.
The training corpus for an LLM is usually nothing more than text scraped from the internet or various libraries. If there's not accurate, up-to-date information about how *his particular LLM currently functions in those datasets, the LLM can’t learn it!
In fact, when you look at system prompts published by companies like Anthropic, you’ll see that LLMs are explicitly instructed NOT to answer questions about how they work. Per Simon Willison, this is to “discourage the model from hallucinating incorrect details about itself”–a well-known issue.
So based on what we’ve outlined above, there’s no way for the LLM to reliably answer questions about its own functioning. But that won’t stop it from fabricating an answer that sounds right–but is in fact completely made up.
🧩 Remember: If the scientists who made these models don’t think it’s reasonable to “just ask the LLM” how it works, maybe you shouldn’t either!
One of the great things about a live workshop is it gives you the ability to answer these kinds of questions together in real time. If you have your own pressing questions about using AI in product research, consider joining my next public workshop (or hosting a private one for your team).