Tend and Flow

An Agent Oversight Checklist

‘Human-in-the-Loop’ checks have some serious failings. Here is a practical, non-technical checklist to use when managing AI agents.

Members of the Hugging Face team wrote an interesting paper about overseeing AI agents. Titled “AI Agents Push Humans Out of the Loop” (Mitchell, Ghosh and Passi, September 2026), the paper argues that current AI agent design and deployment actively erode human oversight (the well-covered idea of ‘Human-in-the-Loop’). It is written for a technical audience of developers and engineers who create and manage AI agents. As I have found with similar papers, this one contains really useful recommendations for a non-technical audience, especially those who are creating and using agents in the operational side of their businesses. In my opinion, a lot of the same research and lessons still apply.

The Problem

The concept of ‘Human-in-the-loop’ is often written into governance frameworks and guidance, but I’m not aware of much that has been written about making agent oversight actually work well in practice. Here are some of the problems the paper’s research calls out:

  • Agent output is dense, often fleeting, and hard to read;
  • The overseer plays 2 roles at the same time (expert and approver) causing fatigue;
  • The current interfaces people use to interact with agents are not built for oversight.

It’s the “irony of automation”: the more capable the system, the more the overseer’s skill atrophies, and the worse prepared they are for the rare, high-stakes moment when something goes wrong.

A Non-Technical View

I’ve reworked the content of their technical advice into a working checklist I think anyone using agents can use. Feel free to adapt it to your context and organisation. The point of the checklist is to build habits; you don’t need more software or more technical skills. Please note my assumption that you are using agents day-to-day and that most of your agents are not fully autonomous. In other words, your agents regularly need your input and approval throughout the day.

The Checklist

Before and After

  • Before you kick off an agent, quickly sketch out a short overview of the expected plan and outcomes. Then review the agent’s work and compare the two.
  • Decide upfront what an agent may do without asking and be explicit in your instructions to it. Then ask for a review at the end of the task.
  • Instruct the agent to check its own numbers or facts before it comes back to you.

During Execution (for anything consequential)

  • Read the approval prompt properly. Do not click through it.
  • Before you approve something, ask the agent what assumptions have been made.

Good Habits

  • Keep a written, running list of known issues in your process. Do occasional quick scans of the list so that these issues stay front of mind when you are reviewing output.
  • Pace yourself. Do not run long, unbroken sessions of review-heavy work. When you notice you are skimming, take a break.
  • Do the task yourself sometimes, without the agent. Make sure your assumptions about the process and task are still valid.

By Design

And for managers:

  • Name who is responsible for checking the agent’s output on each process. Do not let the person who benefits from quick approvals be the only checker.
  • Reward people for catching mistakes, not only for speed.

If you end up experimenting with some of these ideas, or customising your own checklist, let me know how it goes.

(This is my original thinking, developed and produced with AI support from Claude Sonnet 5 and Opus 5)

← Back to Journal