Mr. StokesProduct Designer
← Back to Selected Work
RESEARCH & INTERACTIVE DEMO

On-Device Android Intelligence

Exposing application features as secure, local tools using the newly released Jetpack androidx.appfunctions API. An exploration of latency-free, privacy-preserving client-side intelligence on Android 16.

AppFunctions Playground

Simulate how on-device AI models discover, parameterize, and run native app code via Binder IPC.

AI_AGENT_PROMPT >
01. PARSING INTENT
NL Parsing
Waiting for input...
02. SCHEMA MATCHING
@AppFunction
Searching AppSearch...
03. BINDER CALL
On-Device IPC
Awaiting invocation...
04. VIEW UPDATE
Grid Reflow
Ready...
LOCAL VISUAL CANVAS // RENDER_OUTPUT
ELEMENT_01[100]
Layout Unit A
MATRIX_CELL_0
Swiss
ELEMENT_02[101]
Layout Unit B
MATRIX_CELL_1
Reflow
ELEMENT_03[102]
Layout Unit C
MATRIX_CELL_2
ELEMENT_04[103]
Layout Unit D
MATRIX_CELL_3
Swiss
ELEMENT_05[104]
Layout Unit E
MATRIX_CELL_4
ELEMENT_06[105]
Layout Unit F
MATRIX_CELL_5
Reflow
ELEMENT_07[106]
Layout Unit G
MATRIX_CELL_6
Swiss
ELEMENT_08[107]
Layout Unit H
MATRIX_CELL_7
ELEMENT_09[108]
Layout Unit I
MATRIX_CELL_8
ELEMENT_010[109]
Layout Unit J
MATRIX_CELL_9
Swiss
Reflow
ELEMENT_011[110]
Layout Unit K
MATRIX_CELL_10
ELEMENT_012[111]
Layout Unit L
MATRIX_CELL_11
RESOLVED_SCHEMAS_JSON
androidx.appfunctions Schema Log:
{
  "library": "androidx.appfunctions:appfunctions:1.0.0-alpha09",
  "class": "com.stokes.portfolio.EditorialDesignFunctions",
  "resolvedAppFunction": {
    "name": "createLayout",
    "description": "Triggers a new layout generation inside the visual editor using a custom Swiss grid style.",
    "parameters": {
      "columns": {
        "type": "kotlin.Int",
        "value": 3,
        "description": "Number of structural columns in the grid"
      },
      "hasStickers": {
        "type": "kotlin.Boolean",
        "value": true,
        "description": "Toggles absolute design badging inside the canvas cells"
      }
    },
    "kDocExtracted": true,
    "binderTransaction": {
      "status": "IPC_TRANSACTION_COMPLETED",
      "latency": "4.8ms",
      "deviceContext": "Google Pixel Fold (Android 16 API 36)"
    }
  }
}
DEEP DIVE

Standardizing On-Device AI Actions

For years, integrating applications with virtual assistants required cloud-side schemas, custom webhook servers, or rigid, intent-matching frameworks like Android Intents/Slices.

The newly announced **Jetpack AppFunctions API** completely reinvents this flow. By combining Kotlin Symbol Processing (KSP) with a secure, OS-level indexers, apps can annotate their native methods, automatically converting local database, asset-loading, or UI reflow code into standardized schemas that on-device models like Gemini Nano can invoke.

How It Works Under the Hood

When you compile your application with the KSP dependency, the compiler aggregates all methods marked with the @AppFunction annotation. It parses the Kotlin code, parameters, return types, and most importantly, **your KDoc comments**.

The compiled annotations generate static structural schemas which the operating system indexes via **AppSearch** during application installation. When a user tells Gemini Nano to perform a complex request, the model uses this search index locally to discover which app functions match the user's intent.

Developer Setup & Dependency Configuration

To use AppFunctions, ensure your environment runs **Android 16+ (API Level 36)** and includes the KSP plugin inside your Gradle project configurations:

// build.gradle.kts (App Module)
plugins {
    id("com.google.devtools.ksp") version "2.1.0-1.0.29"
}

android {
    compileSdk = 36 // Required for AppFunctions API
}

ksp {
    arg("appfunctions:aggregateAppFunctions", "true")
}

dependencies {
    implementation("androidx.appfunctions:appfunctions:1.0.0-alpha09")
    implementation("androidx.appfunctions:appfunctions-service:1.0.0-alpha09")
    ksp("androidx.appfunctions:appfunctions-compiler:1.0.0-alpha09")
}

Designing Premium Visual Playgrounds

By integrating AppFunctions interactive visualizers directly into design portfolios, developers can clearly demonstrate to design teams and stakeholders how abstract AI intent calls manifest in deterministic UX results.

This demo showcases that by exposing strict UI parameters (like structural column counts or asset indicators) directly to Binder-level IPC, we create fully unified interfaces where local intelligence drives the visual grid layout without ever speaking to a centralized server.