M. StokesProduct systems

Case study

Pixel & Android Intelligence

Leading the interaction model for system intelligence across Android while keeping sensitive context on the device.

Year
2019–Present
Role
Staff Designer, Lead
Platform
System Intelligence
Domain
Context Understanding, App Functions, Visual Intelligence

System intelligence has to behave as part of Android, not as a separate assistant layered over it. My work defines how local models use screen context, present available actions, and hand control back to the user.

Sensitive context stays inside a protected compute boundary

LOCALSENSINGCLOUDISOLATIONSECURE SANDBOX COMPUTE CORE

On-device processing // Private Compute Core

Sensitive input enters the protected on-device environment for processing. The source data stays inside that boundary while the system returns only the result needed by the interface.

Private Compute Core is an isolated environment inside Android. It lets system features use screen context and device signals without sending the source data to a cloud service. That boundary informs every interaction pattern: the interface must show what the model can use, exclude protected content, and keep consequential actions under user control.

App Functions turns a model response into an observable action

PROMPTINPUTIMMEDIATEREFLOWSUB-10ms DIRECT MORPH

01 // Local execution

On-device execution removes a server round trip. The interface can move directly from a request to the resulting app state without hiding the transition behind a loading sequence.
OBSERVABLEPREVIEWEXPLICITCONSENTSKIPRUNHUMAN-ON-THE-LOOP CONTROL

02 // Supervision gate

The execution preview shows the pending action before it runs. The user can revise or stop the action before approval.
Gemini Teamwork Animation
Fig. 01App Functions enables real interaction through observable execution flows.

App Functions lets Gemini Nano call capabilities exposed by Android apps. I defined the interaction primitives that make those actions previewable, interruptible, and explicit about the point of commitment.

View Interactive AppFunctions Playground & Demo →

Context only helps when the exclusion rules are clear

OBSERVEDCONTEXTAUTOMATICREDACTIONREDACTEDEXTRACTING CONTEXTINTENT-BASED SECURITY FILTERING

Context policy // Protected regions excluded

Secure fields and system privacy flags keep protected regions out of the observation buffer before the model receives screen context.

The Content Capture API lets the system surface actions from the active screen without a separate prompt. Protected contexts are excluded before analysis. Banking and password-management surfaces remain outside the observation buffer by default.

Shared primitives keep intelligence consistent across Android

DIVERSESURFACESUNIFIEDORCHESTRATIONACTION CHIPKEYBOARD SUGGESTSHARED SYSTEM PRIMITIVE ENGINEMULTI-SURFACE INTENT RESOLUTION ENGINE

Shared primitives // Consistent intent handling

One intent-resolution model drives contextual actions across Android surfaces. The presentation changes with the surface, but the control behavior stays consistent.
Android Context Recognition Demo
Fig. 02Screen context informing available actions while respecting strict privacy boundaries.

My role is to define the interaction rules that feature teams build on, not a separate screen for each model capability. The same intent and supervision patterns can appear across Android surfaces without changing what the user expects.