Duplex on the Web

Led from conception to successful launch. Automating web browsing using ML to reduce friction for everyday tasks—buying movie tickets, renting a car, changing compromised passwords, and flight check-ins.

The Challenge of the Mobile Web

Simple tasks remain cumbersome on the mobile web. Buying movie tickets or renting a car should be trivial but require navigating multiple pages, pinch-and-zoom form filling, and manual data entry across highly fragmented checkout flows. While APIs and structured data exist, the web is overwhelmingly unstructured DOM elements.

Approach & ML Orchestration

Designed Assistant-mediated fulfillment through automated actions on the web. Rather than relying on backend integrations, Duplex uses ML to visually parse the DOM, identifying buttons, text fields, and semantic structure. The system pulls information from Gmail (upcoming trips, reservations) and Chrome autofill to complete tasks with minimal user input.

Crucially, the interaction model is designed around observability. The user watches as Duplex navigates the site, fills out forms, and clicks buttons. A persistent overlay provides status updates ("Selecting your seats", "Entering payment info") and a prominent "Stop" button. Users see exactly what the AI is doing and can intervene at any point, ensuring the system never takes an unrecoverable action without supervision.

Impact

Announced at Google I/O 2019 and rolled out as "Assistant in Chrome" on Android. Expanded from movie tickets to include automated password changes for compromised accounts, flight check-in, and discount finding for e-commerce. It proved that complex, multi-step UI automation could be safe and trustworthy if designed with the human firmly on the loop.

Scaling Ambiguity

When designing for ML automation, the primary challenge is scaling ambiguity. My process involved heavily prototyping "unhappy paths." Instead of just designing the ideal checkout flow, I led the team in mapping out systemic recovery patterns for when the model misinterpreted a DOM element or encountered an unexpected CAPTCHA. By establishing these robust fallback paradigms early, we empowered the engineering organization to iterate aggressively on the ML architecture without breaking the core UX.