M. StokesProduct systems

Case study

Duplex on the Web

Led the design from concept through launch. Duplex used ML to complete routine web tasks while keeping each action visible and interruptible.

Year
2018–2019
Role
Design Lead
Platform
ML Automation
Domain
Assistant, ML Automation

I led the design from concept through launch. Duplex used ML to complete routine web tasks while keeping each action visible and interruptible.

Routine web tasks still required too much manual work

Buying a movie ticket or renting a car often meant navigating several pages and repeating the same form entry on a phone. APIs covered only part of the web. Most sites still exposed a changing, unstructured DOM that automation had to interpret.

Duplex operated through the same web interface as the user

UNSTRUCTUREDDOM TREESTRUCTUREDACTIONS<input><button><div>DUPLEX MLPARSINGCHROME AUTOFILLFILL FORMCHOOSE SEATSPAY & BOOKASSISTANT-MEDIATED WEB ORCHESTRATION

DOM parsing // Browser automation

Duplex reads the page structure, identifies the controls required for the task, and uses Chrome Autofill data only where the user has already approved it.

I designed an Assistant-mediated flow that worked through the page instead of depending on a custom backend integration. Duplex parsed the DOM, found the relevant controls, and used approved account data to complete the task with less repeated input.

The interaction model keeps execution visible. A persistent overlay reports the current action and keeps a Stop control available. The user can intervene before the system crosses an unrecoverable boundary.

The system launched as Assistant in Chrome

Google announced the work at I/O 2019 and released it on Android as Assistant in Chrome. The capability expanded beyond movie tickets to compromised-password changes, flight check-in, and commerce tasks. The launch showed that multi-step browser automation could remain observable and interruptible.

Recovery behavior mattered as much as the successful path

AUTOMATEDEXECUTIONSUPERVISEDHANDOFFUSER VERIFICATIONTAP TO SOLVEHUMAN-ON-THE-LOOP RECOVERY GATEWAY

Recovery gate // User handoff

When Duplex reaches an ambiguous control or security gate, execution pauses and the user takes over at the same point in the flow.
Duplex on the Web Demo
Fig. 01

I focused the prototyping on unhappy paths. We mapped what should happen when the model selected the wrong element, met a CAPTCHA, or reached a payment confirmation. Those recovery patterns let the ML system improve without changing the user's basic control model.