Engineering AI for Creativity and Curiosity on Mobile

Bhavuk Jain, a Tech Lead at Google working on applied AI, uses two products he helped build with his teams, AI Wallpapers and Circle to Search, to explain what lies between a foundation model and a dependable mobile feature. The examples differ in their objective: AI Wallpapers generates creative content, while Circle to Search interprets screen content and retrieves information. Both show that model capability is only one part of the product.

These notes summarize Jain's 49-minute presentation at QCon San Francisco 2025 and its Q&A; InfoQ published the recording and transcript on July 21, 2026. Descriptions of Google's systems and results below are Jain's claims, not independently verified findings. Sections labeled Synthesis and the review checklist are my interpretation.

What You Will Learn

• Why Jain separates alignment, specialization, grounding, guardrails, and production serving.
• How a constrained prompt-building UI improved the tractability of wallpaper quality and safety.
• How human review and more than one million generated images informed a small set of user-facing prompt choices.
• How Circle to Search combines screen context, multimodal understanding, parallel retrieval, synthesis, and safety filtering.
• Why OS integration can improve both interaction cost and the context supplied to a model.
• Which cost techniques Jain reports using: a more efficient model, batching, prefix caching, and a shared internal endpoint.
• What the talk does not disclose about models, evaluation, latency, cost, and privacy.

From Foundation Model to Product

Jain describes a progression with four broad engineering stages:

1. Post-training aligns behavior. Jain presents RLHF as a flexible but operationally complex path involving a reward model and reinforcement-learning loop. Direct Preference Optimization (DPO) works directly from preferred and rejected outputs with a supervised objective; he presents it as simpler and easier to scale. Either approach still requires explicit evaluation criteria because qualities such as a "good" summary or image are subjective.
2. Fine-tuning specializes behavior. Supervised task or instruction tuning teaches input-output behavior. Parameter-efficient methods such as LoRA, QLoRA, and adapters leave most base weights frozen and train smaller adjacent modules, reducing training time and cost.
3. Retrieval and grounding supply knowledge. If the problem is missing current, factual, private, or contextual information, Jain says grounding is a better answer than further fine-tuning. Sources can include structured knowledge, fresh web results, search or RAG systems, tools, and on-device context.
4. Inference, guardrails, and deployment make operation viable. The product must serve multimodal requests reliably while controlling latency and accelerator cost, observing failures, and inspecting what the system can see, say, or do.

This is not a claim that every product uses every named training method. It is a decision framework: change behavior through training, supply changing facts at inference time, and protect the complete system rather than assuming the model alone is safe.

Guardrails Are Layered

Jain's general guardrail model covers four layers:

• Data: filter or redact sensitive and toxic material in training and evaluation data.
• Model: start with safety-tuned models and add system instructions or dedicated classifiers or safety heads.
• Runtime: inspect inputs and outputs for risks such as personally identifiable information, toxicity, and jailbreak attempts; restrict callable tools and APIs.
• Governance and UX: retain logs and audits, route high-risk operations to human review, and give users reporting and feedback controls.

As general production guidance, he recommends a unified multimodal API or pipeline rather than one stack per input type. He names batching, caching, and autoscaling as ways to address latency, compute cost, and scale, and logs and metrics as ways to observe latency, errors, and usage. This part of the talk is not a disclosure that every technique was used in both case studies. The talk does not provide service-level objectives or per-request costs.

Case Study: AI Wallpapers

AI Wallpapers lets a user generate a phone wallpaper from guided choices. Jain says wallpapers were a useful first canvas because personalization fits Android's Material You system, which derives interface colors from the wallpaper, and because changing the wallpaper was already a popular form of phone customization.

The team faced four connected problems:

• Ordinary users should not need vocabulary such as "cinematic lighting" or "ethereal watercolor" to obtain a useful result.
• A technically valid generated image may still be aesthetically poor.
• Harmful, inappropriate, or copyrighted output had to be addressed before launch rather than after it.
• Server-side image generation was expensive enough that growing usage threatened the feature's long-term viability.

Constrain the Product, Not Only the Model

The team considered a spectrum from pre-made concepts, through guided prompt building, to an unrestricted text box. It chose a partially restricted, guided experience. Jain argues that a blank prompt moves prompt-engineering work to the consumer and makes the product's output space harder to evaluate and guard.

Restriction provided two engineering benefits. It concentrated requests in areas where the model performed well, and it made quality criteria and safety testing more tractable. User-experience researchers tested the concepts with real users before the team committed to the approach.

The trade-off was less freedom. The team tried to compensate by investing in delight: distinct motion for prompt chips and playful error states, such as a T-Rex trying to paint, made the bounded workflow feel intentional rather than merely limited. Jain also says restriction reduced both the likelihood and the cost of severe safety failures and made risk assessment tractable; the talk does not quantify either reduction.

Architecture and Safety Checks

Jain presents this high-level request path:

guided mobile UI
  -> prompt-engineering service
  -> backend input checks
  -> fine-tuned media-generation service
  -> output-image classifiers
  -> image upscaling
  -> wallpaper returned to the device

The prompt service converts simple selections into a structured prompt with artistic modifiers and negative keywords. Fast server-side classifiers and block lists check that prompt before inference. Visual classifiers inspect the generated image before upscaling and delivery. Jain says these practices supported making the initial feature available to users aged 13 and older; the talk does not enumerate the policies, classifier thresholds, or residual failure rates behind that decision.

Turn Taste into an Evaluation Process

Rather than asking one model to perform well across every visual category, the team mapped categories and styles in which it already generated strong results. It then constrained choices to those regions.

The testing pipeline generated prompt variations at scale and ended in manual review by designers and researchers. Human raters considered format, quality, completeness, overall appeal, and what Jain calls the "wow factor." The team generated and reviewed more than one million images across prompt combinations to decide which small set of words should become suggestion chips in the UI. That figure describes evaluation volume, not users, training examples, or distinct production images.

In Q&A, Jain describes this as a continuing loop. Product managers, UX designers, and human reviewers inspected results, adjusted prompts and tests, and worked with research teams as models evolved. He says the initial project took roughly a year, rather than three or four months, partly because the models available at the time made more mistakes.

Make Structure Pay for Itself

Repeated prompt templates made prefix caching possible because shared prompt segments could be reused. In Q&A, Jain confirms that the team used prefix caching and names three other cost responses:

• Move to a more efficient model.
• Batch generation instead of running each prompt independently.
• Reuse one scalable inference endpoint across multiple Google products, which he says reduced cost, including feature-development cost, and improved reliability.

He does not quantify the savings or state which model, hardware, batch size, or cache policy was used. The talk reports that the feature expanded across form factors and was adopted by major device manufacturers, including Samsung, but does not provide usage or retention figures for AI Wallpapers.

Case Study: Circle to Search

Circle to Search replaces a screenshot, app switch, upload, and search sequence with a system-level gesture over content already visible on an Android screen. Jain frames it as a real-time multimodal retrieval problem: infer what the user selected and intended, retrieve relevant information, and return an overlay without making the user leave the current app.

The product builds on Google Lens but changes the interaction surface. In Jain's comparison, Lens is world-facing through a camera, while Circle to Search is screen-facing.

Architecture

The on-device gesture captures more than a flat screenshot. Jain says the multimodal prompt contains pixels and the screen's structural layout. The presented server-side flow is:

gesture + screen pixels + structural layout
  -> fine-tuned multimodal intent model
  -> parallel knowledge-graph, web, and image retrieval
  -> AI Overview synthesis model
  -> safety and responsibility filters
  -> on-screen result overlay

For a circled pair of shoes plus a textual refinement such as "blue," Jain says Google's Multitask Unified Model (MUM) combines visual attributes with semantic intent. Gemini supports richer reasoning and AI Overviews. The retrieval stage can query a knowledge graph for an entity while web and image search obtain products, articles, reviews, or similar images. A synthesis model forms the natural-language response, which passes through a final responsible-AI filter.

Integration Changes the Input

Deep Android integration removes the need to find and open another app, but its technical value is not limited to invocation. Access to semantic screen layout helps the system isolate the selected object among surrounding pixels. Jain therefore treats integration as both a UX improvement and an improvement to the context available for inference.

The system also uses three trust mechanisms described in the talk:

• Cross-reference recognized entities with the knowledge graph rather than trusting visual recognition alone.
• Filter synthesized answers through a dedicated safety layer.
• When confidence is low, return broader suggestions instead of inventing a precise answer.

Jain reports launch on more than 300 million devices and a 70% year-over-year increase in visual searches. He says visual search is a fast-growing query type, especially among younger users, and that shopping and translation are performing well. He attributes the increase to easier access, but the presentation does not define the measurement window, comparison population, causal method, or exact scope of the 70% figure. It should therefore be read as a speaker-reported product metric, not independently established evidence that Circle to Search alone caused all of the growth.

Synthesis: Lessons Across Both Products

The following conclusions are my synthesis of the two case studies, not direct quotations or additional Google disclosures.

UX Constraints Can Be System Constraints

In AI Wallpapers, guided choices narrow the inference distribution, reduce the number of states that need evaluation, enable reusable prompt prefixes, and make input policy easier to enforce. A UI decision therefore changes safety, evaluation, latency, and cost together. Product boundaries can be part of the technical architecture.

Integration Can Matter More Than Another Model Upgrade

Circle to Search gains value by removing context switching and by collecting better structured context at the operating-system boundary. This suggests a useful product question before changing models: can the feature improve how it is invoked or what evidence reaches the model?

Evaluation Must Match the Product's Failure Modes

Wallpaper evaluation centers on aesthetics, consistency, and visual safety; search evaluation must also consider intent, retrieval relevance, factual grounding, confidence, and response safety. "Model quality" is not one portable metric. The product's action and consequences determine what must be measured.

Shared Infrastructure Helps Only After the Workflow Is Defined

The talk favors unified serving and shared endpoints, but its strongest results come from product-specific work: curated wallpaper choices and semantic screen context. A platform can share serving, caching, observability, and safety mechanisms; it cannot define each feature's quality bar or user workflow.

Practical Review Checklist

Derived from Jain's presentation and the synthesis above, a team reviewing a mobile AI feature can ask:

1. Is the main gap behavior, missing knowledge, or missing runtime context?
2. Which user freedoms create value, and which merely enlarge an untested output space?
3. What observable criteria turn subjective quality into repeatable human evaluation?
4. Which checks belong at data preparation, model, input, output, tool, and UX layers?
5. What should the product do when confidence is low?
6. Can system or application integration eliminate steps and provide richer context?
7. Can structured prompts support batching or prefix caching without degrading relevance?
8. Which serving components can be shared without erasing product-specific evaluation and policy?
9. Are claims about scale, quality, latency, and cost tied to defined metrics and comparison windows?

Jain's explicit closing advice is audience-specific: AI engineers should productize rather than merely deploy a model; technical leaders should define and defend product principles; and product managers and designers should focus on removing friction from the user's workflow.

Source Limitations

The InfoQ page provides a 49-minute video, a transcript including three Q&A exchanges, speaker biography, publication metadata, and a synchronized presentation view. InfoQ marks the downloadable slide deck as access-controlled. The page exposed only one public slide image, so the architectural diagrams and visual examples were checked against Jain's transcript descriptions rather than a separately inspectable PDF.

The talk is an experience report, not a reproducible technical paper. It does not identify every model version, dataset, prompt template, classifier, threshold, benchmark, latency target, infrastructure configuration, privacy control, or cost. It also does not publish raw evaluation or product analytics. Reported scale and growth numbers remain Jain's claims, and the transcript does not provide enough detail to independently verify causality or generalize them to another product.

Canonical source: Bhavuk Jain, "Engineering AI for Creativity and Curiosity on Mobile", QCon San Francisco 2025 presentation published by InfoQ on July 21, 2026.