The Agent Can Do It but Should It?

These days when building products, I am thinking a lot about the agentic experience and how a user is going to use their agent to interact with the product. This week we finished an opt-in consent feature around choosing AI models in the AI Gateway. We built this because there are claims the product zero data retention and while the gateway has zero data retention, the model providers don't necessarily retain zero data. Thus a person has click to consent in the UI.

But what happens when an agent is doing the clicking?

It's quite possible the agent could say yes, and I as the user could say I never said yes. It might skip over the explanation or summarize it in a way that leaves out the thing I actually needed to know. Meanwhile, our product would have a record saying I'd agreed.

Well, that's a problem.

If you're already doing your work with an agent, being sent off to another dashboard for every little thing is annoying. I use these tools myself. I get it.

So we are building rules and and tools where the agent can't skip the human. Imagine asking someone to help set up your new phone. They can move your apps over and connect it to the Wi-Fi. Then they get to a screen asking whether you're okay with sharing more of your information. I'd want them to hand the phone back for that part. They can carry on helping once I've decided.

Could we send the person to the screen for this step, have them make the choice, and then let the agent resume? That's basically what we're building and we're not the only one. Products like Cloudflare already do this. My agent reaches something I need to handle in the UI, tells me where to go and what to do, and asks me to let it know when I was done.

Being specific with the handoff is paramount. The agent can explain that the model you're trying to use has different data-retention conditions, that you haven't agreed to them, and where you can review the choice. You shouldn't have to open the product and start hunting around to figure out why your task stopped.

And you might decide you don't agree. That is also a possible outcome. Otherwise we've just made a very elaborate way of asking you to click yes. The task may need to change, or the agent may need to help you find another model. Getting to the end of the original request doesn't get to decide the answer for you.

This is a part of designing for agents that I think product people need to spend more time with. We're used to looking for steps we can remove. Now we also have to look at what those steps were asking a person to decide, and whether that choice survives when someone delegates the work.

There's also a fairly obvious way to make this miserable. Ask someone to confirm every tiny action and they'll spend the whole time babysitting the agent they asked to take over the task. The work is in figuring out which choices need human attention, then making it easy to give them that attention. The answer when working with agents isn't to just do it. Sometimes you need to harness your inner Dr. Ian Malcom and let go of being enamored that the agent could and stop and think if they should.