Big Design 26 - AX Testing — Llewyn Paine Consulting, LLC

Welcome, Big Design attendees!

Can an AI agent figure out your website?

Why test agent experience (AX)?

AI agents are the new mediators of user experience. When your user sends an agent to your website, and it is unable to complete its task–it’s your user who gets frustrated.

Agent traffic is small but growing. It ballooned by nearly 8,000% YOY in 2025,* and as more users adopt agents like OpenClaw and Claude Code, they’ll become an increasingly important vector of UX. But we’re already feeling their impact today–last year AI and agents were responsible for $67B in sales during Cyber Week alone.*

What’s the best agent to use for testing?

The best agent to test your website is whatever agent your user uses! Different agents use different technological methods to “see” and interact with your website, and different models (LLMs and VLMs) can have drastically different performance. So to cover your basees, it’s safest to test across a variety of different agents and models. You can build that infrastructure yourself, or contact me about using the platform we’ve built.

How many runs do you need for your AX test to be reliable?

It depends how variable the agent is. Some agents perform roughly the same every time, while others make wildly different mistakes across different runs. By default, I run tests 11 times per agent. This gives a naive probability of encountering the most common issues with roughly 95% confidence

Can we use your platform to test our interface for AI agents?

Absolutely! Tertile, LLC, and I are working on a self-service system, but in the meantime we’re happy to partner with you so that you can test your interface in multiple runs across a variety of different agents and models. Contact me to discuss your needs.

Where can I download the slides from your Big Design talk on AX testing?

Right here!