AI agents pose significant risks as they are granted increasing autonomy. A commonly proposed solution is human oversight and keeping a “human in the loop”, but this is not a simple solution: Not only do current approaches to AI agent design impede effective human oversight, but the cognitive capacities required for it are also themselves degraded by extended use of AI systems.
This position paper argues that current approaches to the development and deployment of AI agent systems do not support effective human oversight — they contribute to its degradation. A top priority in advancing AI agents should be supporting the situated goals and cognitive requirements of effective human oversight, treating the human needs of overseers at the same level of importance as AI agent capability.
The authors connect work on automation and HCI to AI agent processes, outlining design-level affordances and organizational protocols that (1) support overseers in exercising critical judgement and (2) counteract the skill atrophy that arises from extended use of automation. Without explicit support for the cognitive demands of effective human-agent interaction, AI agent systems will continue to passively incentivize the degradation of the very human skills they rely on.
The role of human oversight
Effective oversight is critical because agents can take wrong, costly, or irreversible actions, and errors often cannot be detected from output review alone. Agentic systems expand beyond single-turn GenAI: multi-step plans, tool use (including tool hallucination), multi-agent misalignment, and behavioral unpredictability make real-time assessment necessary — while comprehensive pre-deployment evaluation of multi-step behavior is often economically infeasible.
Human-in-the-loop lineages (aviation harm prevention, autonomous-weapons authorization, ML feedback) converge on users who must prevent harm, authorize consequential actions, and supply training signals — yet presence of an overseer does not entail reliable oversight.
What current AI agent oversight misses
- The amount of content relevant to an overseer is massive (CoT, tools, plans, inter-agent messages).
- Users must play multiple roles simultaneously and suffer approval fatigue.
- Cognitive demand for System-2 scrutiny is unmet by interfaces optimized for System-1 flow.
- Vendor HITL guidance often centers the agent’s processing rather than the user’s cognitive needs.
The irony of automation
Sustained AI use is linked to deskilling, reduced vigilance, overreliance, “cognitive debt,” and measurable drops in brain connectivity in some study settings. Biases (automation, anchoring, complacency) further impede oversight. Interaction patterns that feel helpful (fluency, sycophancy, single authoritative answers) can weaken skepticism. Users report “babysitting” outputs and cognitive distance from agent-written code. Degraded approvals can then poison preference-learning feedback loops — reward hacking of human oversight.
The result is Bainbridge’s irony of automation: the more capable the system, the less prepared humans are for the rare moments when intervention matters most. Oversight degrades the overseer.
Solutions (inventory)
Development (design affordances): strategic friction (pre-commitment, delay-and-choice, reasoning probes, action gating); decision design (bounded autonomy, batch review, automated pre-checks, supportive interfaces); behavioral monitoring (time/override/evidence-seeking signatures, canaries, audits).
Deployment (organizational protocols): trainings and exercises (domain skill maintenance, critical evaluation, self-monitoring); workload and scheduling (breaks, rotations); role design (expertise, separation of incentives, rewarding scrutiny).
Alternative viewpoints addressed
Adaptation will not automatically solve structural differences between search tools and action-taking agents; better transparency tooling is necessary but insufficient without organizational protocols; preference-based alignment alone can worsen oversight if degraded raters are the reward channel.
Conclusion
Human oversight is written into governance and vendor docs, yet current agent trajectories actively degrade the capacities oversight requires. Cognitive scaffolding for overseers must be a first-class concern on par with capability. The authors urge developers and deployers to adopt these or similar approaches, and invite empirical work on cognitive degradation and system-level audits of whether agents support the oversight they presume.
Full paper: arxiv.org/abs/2608.23642