The Long-Term User Experience of Agentic Systems

Delegation, Authority, and the Path to Ambient Computing

Working Paper, June 2026

A theory of how human-computer interaction changes when the user stops manipulating objects and starts supervising delegated machine work. The paper also gives a 5–10 year outlook grounded in agentic embodiment. The companion paper, "Constraint, Capacity, and Self-Maintaining Agency," addresses the machine-side account; this paper foregrounds the person.

---

Abstract

For forty years, direct manipulation organized computing [U1]: the user selects an object and acts on it; the machine renders the result. Agentic systems change the design problem from making computation visible enough to manipulate to making delegation visible enough to supervise.

The central claim is that the most important on-screen state is no longer selection (which object is active) but authority (what the system is permitted to do next). Authority therefore needs to become a first-class interface object. The paper develops an interaction grammar for delegated agency and places it along an agency gradient drawn from levels-of-automation theory [U4]. It then connects that gradient to the system's memory model.

Agentic embodiment [U2][U3] introduces problems that direct manipulation did not have to address. A device with persistent sensing affects everyone who shares the space, including people who did not activate it. Embodied agents also encourage people to infer a degree of selfhood the evidence does not support. These are shipping constraints, not ethics notes added after the interaction model is complete.

The organizing principle is that friction is a feature. In a delegated system, the controls that preserve judgment are part of the experience.

---

I. From Manipulation to Supervision

Direct manipulation won because it made the machine's state continuously visible and continuously actable [U1]: a file you can see is a file you can move. Agentic systems break the premise. The work no longer happens in front of you, object by object; it is delegated, runs over time, and spans tools and surfaces you are not watching. The user's role shifts from operator to supervisor.

The change reaches beyond widgets because it changes what the interface is for. A manipulation interface answers what is selected and what can be done to it. A supervision interface has to show what was delegated, what the agent is doing now, what it has already done, what it may do next, and how a person can stop or revoke it. The discipline's primitive shifts from the button to the boundary, from the action a person takes to the scope, permission, commitment, provenance, and interruption that frame what the agent may do.

A useful agentic interface makes machine authority legible. It shows what the agent can access, what it has already done, and how a person can intervene. When legibility conflicts with speed, legibility should win because the alternative is loss of control.

---

II. Authority as a First-Class Interface Object

In a traditional application, the most important state is selection: which object is active. In an agentic system, the most important state is authority: what the system is permitted to do next. Authority must become an interface object that represents the constraints around the agent. A person should be able to inspect the current authority and change it directly.

Authority is not monolithic; it decomposes into dimensions that each demand their own surface:

• Perception: what the agent may sense (screen, microphone, camera, location).
• Context access: what stored or personal context it may read.
• Tool access: what external tools and integrations it may invoke.
• Commit authority: what it may do irreversibly without confirmation.
• Memory: what it retains, and what it may forget; memory must be inspectable, scoped, editable, and forgettable.
• Accountability: what record it keeps of what it did and why.
• Recovery: how its actions can be undone, halted, or rolled back.

Distinct controls let a user grant narrow authority, such as broad perception with no commit permission. The companion paper treats authority as internal state the machine must respect. This paper treats it as an external control the person operates. Those states must never diverge. The authority the user believes they granted must equal the authority the system believes it holds.

---

III. The Interaction Grammar

Operating delegated agency requires a vocabulary richer than open/save/select. Twelve primitives compose the grammar:

Summon, Scope, Delegate, Watch, Interrupt, Commit, Explain, Undo, Escalate, Forget, Prove, Transfer.

They are not a feature list; they are how a user moves an agent through the agency gradient (§IV). Scope and Delegate set an initial position; Watch and Interrupt hold it open for oversight; Commit crosses the irreversibility line and is where confirmation belongs; Undo and Escalate are the recovery rails; Explain and Prove make the agent's reasoning and provenance auditable; Forget operates pruning; and Transfer carries a delegation contract across surfaces, so the same authority object persists from phone to desk to room display.

Underneath the grammar sit eight design models that the primitives express, Intent, Permission, Agency, Interruption, Memory, Provenance, Fallback, and Bystander. Two of these bind this paper to its companion: Memory (the inside axis, §V) and Bystander (the embodiment problem, §VI). The grammar's job is to make every one of these operable by an ordinary person under time pressure, not just specifiable by an administrator.

A note on friction. Confirmation and recovery controls may slow direct manipulation, but they protect judgment in a supervision interface. The design question is where to place that friction. It belongs at the point where an action becomes difficult to undo.

---

IV. The Agency Gradient as a UI Scale

Autonomy is not binary, and the interface should not pretend it is. Drawing on levels-of-automation theory [U4], agency is a tunable scale the user sets and adjusts:

Rung	Mode	What the agent does	Required oversight
0	Show	Renders information; takes no action	Not applicable
1	Suggest	Proposes; the human acts	Read
2	Draft	Produces a draft the human edits	Review
3	Prepare	Stages a complete action and waits for approval	Approve
4	Act-within-policy	Acts inside set bounds; human monitors	Monitor / intervene
5	Act-autonomously	Acts and commits; supervised autonomy	Audit / revoke

The rungs map onto the supervision taxonomy of autonomy doctrine [U5]: rungs 0–3 are human-in-the-loop (the human is in the path of each action), rung 4 is human-on-the-loop (the human monitors and can intervene), and rung 5 approaches human-out-of-the-loop. The doctrine's standard, designing for "appropriate levels of human judgment" [U5], is, in interface terms, a requirement that the right rung be selectable per task and per consequence, and that the UI make the current rung and its required oversight unmistakable. A small screen must never hide large authority: the surface a delegation is operated from must escalate its prominence with the autonomy it carries.

---

V. The Memory Gradient as Device Experience

The companion paper develops memory as the machine's internal architecture. For the user, memory determines whether work can continue across time and devices. The gradient has five observable rungs: M0 Stateless (no memory beyond the turn); M1 Archive (it can look things up); M2 Context/Harness (the environment restores prior state); M3 Governed (it retains selected material and lets the rest decay); M4 Self-maintaining (it preserves commitments and calibration across time).

The UX consequence is that continuity is carried by state, not by hardware. Governed memory and authority can persist as work moves between devices through the Transfer primitive. At M3 and M4, memory must remain inspectable and editable. A continuity the user cannot correct is not supervisable.

---

VI. Embodiment, Bystanders, and Consent

The hardest UX problems appear when agency becomes embodied and occupies a space other people share [U2][U3]. Direct manipulation did not have to account for people who never activated the system.

• Shared-space consent. The operator's consent does not extend to everyone in the room. Sensing must be visible to non-operators, and perception should remain off until someone with authority enables it.
• Anthropomorphism risk. A persistent embodied agent encourages people to infer selfhood from presentation. The interface must distinguish self-maintenance from evidence of a self.

Embodiment should ship only when the platform can make sensing legible to everyone affected by it. A system that cannot disclose its perception state to the room should not operate there.

---

VII. The 5–10 Year Outlook

2026–2031: agents over existing software. Agents first arrive as a coordination and memory layer above current applications, not as a replacement for laptops or phones. Interfaces mix capture, review, commitment, and audit according to the task. The operating system takes responsibility for policy and revocation across those applications. Most products remain at draft-and-prepare autonomy while enterprises cap actions inside explicit policy.

2031–2036: role-based and ambient. Work may move across personal and shared devices while governed memory and authority persist through Transfer. This shift is not guaranteed. Ambient systems can ship only if shared-space consent and anthropomorphism are addressed at the platform level.

Four alternative futures (not mutually exclusive):

• A, Governed delegation substrate. Agent identity, observability, and policy become normal infrastructure; supervised autonomy over governed memory becomes routine. The thesis confirmed.
• B, Fragmented agent sprawl. Everyone ships agents; nobody controls continuity. Systems scatter into high-autonomy/low-memory-governance, the danger zone where action outruns the memory needed to make it predictable. The thesis defeated by execution, not argument.
• C, Screen-bound backlash. Privacy, labor, and regulatory pressure keep agents inside PCs and phones; bystander consent is never solved, so embodiment never ships.
• D, Consumer bleed-through. Home, elder-care, and family agents borrow enterprise patterns, raising the bystander/consent stakes precisely where bystanders, children, elders, guests, can least give informed consent.

Falsification of the outlook (distinct from the scientific claims of the companion paper): the thesis is disproven if agent-first devices lack ROI, if enterprises reject ambient sensing, if regulation makes cross-context memory impractical, or if sprawl outpaces governance benefit.

---

VIII. The Capture Risk

The deepest systemic risk is the capture of human attention and authority by organizational or platform priorities. An attention filter decides what the user sees and what remains hidden. That makes interruption design a power problem. If platform incentives control the governance layer, the system may remain technically coherent while serving the wrong interest.

This is why the design commitments of this paper are not neutral. Legibility, scoped authority, inspectable memory, placed friction, bystander standing, and revocability are the features that keep a supervision interface supervisory rather than custodial. They are the difference between an agent you direct and an agent that, very politely, directs you. The interface is where that difference is either protected or quietly surrendered.

---

Coda: Supervision Is the Experience

Direct manipulation made machine state visible enough to act on. Agentic computing must make delegation visible enough to supervise. A person needs to see what they handed over and how to take it back. Authority, memory, and embodiment all need controls that preserve that ability.

The useful systems will make machine authority legible and keep the person able to intervene. The companion paper argues that the human supervisor is the only observer outside the system's own error propagation. This paper argues that the interface must preserve that position. Otherwise the supervision exists only in policy.

---

External References

Foundational HCI and autonomy works are cited at standard detail; DoD 3000.09 was verified during this revision. Claims about memory, self-maintenance, and the Oracle Problem are developed and cited in the companion paper.

[U1] Shneiderman, B. (1983). Direct Manipulation: A Step Beyond Programming Languages. IEEE Computer, 16(8), 57–69. The canonical statement of the manipulation paradigm this paper argues is changing. (See also Hutchins, Hollan & Norman, 1985, Direct Manipulation Interfaces.)

[U2] Weiser, M. (1991). The Computer for the 21st Century. Scientific American, 265(3), 94–104. Ubiquitous and ambient computing; the source of the situated, beyond-the-window commitment.

[U3] Dourish, P. (2001). Where the Action Is: The Foundations of Embodied Interaction. MIT Press. Embodied interaction; meaning constituted through situated action.

[U4] Sheridan, T. B., & Verplank, W. L. (1978). Human and Computer Control of Undersea Teleoperators. MIT Man-Machine Systems Laboratory; Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A Model for Types and Levels of Human Interaction with Automation. IEEE Transactions on Systems, Man, and Cybernetics, Part A, 30(3), 286–297. The levels-of-automation framework underlying the agency gradient (§IV).

[U5] U.S. Department of Defense. DoD Directive 3000.09: Autonomy in Weapon Systems. Issued November 2012; updated January 25, 2023. The "appropriate levels of human judgment" standard and the in-/on-/out-of-the-loop taxonomy. Verified during this revision.

---

Corpus Sources (Primary Provenance)

The design material derives from the Solara delegable-computing corpus and associated notes:

• The delegable-computing keystone (authority-as-state; supervise-not-manipulate; the twelve primitives and eight design models; priority-capture).
• The agentic-systems design notes (authority as a manipulable HCI object).
• The HCI outlook (pruning as precondition of delegation quality; the 5–10 year trajectory and alternative futures).
• The 2026-04-27 camera hard-block (operator authority over the Perception dimension; default-off, removed from self-heal), the bystander/consent principle enacted in miniature.

Provenance caveat: this paper synthesizes a private design corpus; its claims are design theory and forecast, not experimental results. Its falsifiers (§VII) are market and adoption predictions, and are kept separate from the empirical falsification battery of the companion paper so that being wrong about the outlook costs nothing on the science.

---

Companion paper: "Constraint, Capacity, and Self-Maintaining Agency" treats the machine-side account of the same principle. This paper foregrounds the person and their supervision; the companion foregrounds the machine and its governance.