Designing the context and control layer for an AI brand agent

Overview

Users brought brand signals in radically different forms—from a sentence and a few images to full guideline libraries—while every downstream generator still needed one current, usable direction.
I redesigned the experience for both users and AI agents: agents interpret and prioritize mixed inputs; users review the visual direction, correct uncertain details, and see how those decisions affect content produced by other agents.

My role

I led the end-to-end design for both end users and AI agents, synthesizing user and model research, mapping user and agent workflows, designing the interaction model, and planning validation. I partnered with engineers on LLM capabilities, constraints, and agent orchestration, and worked with the GTM team to define product scope and design goals. I also defined agent behavior and orchestration, built code-based prototypes, ran prompt and model tests, and evaluated generated outputs.

Design challenge

Turn brand materials into reliable context for AI agents

Some users bring detailed guidelines, while others have scattered references or only a sentence. Agents need to select and analyze these inputs, build shared brand context, and coordinate content creation. I needed to design how agents understand users’ intent across different inputs and materials, with review layers that let users inspect their work as agents update the brand.

  • Ideation (Day 1): explore directions through discussion with ChatGPT.
  • Planning: review features, architecture, tech stack, database, and APIs.
  • Break the plan into scoped tasks and dependencies.

Research

I began with user research and model research to understand users’ brand context and what models could help them do. These findings informed both the agents’ workflow and the user experience.

01 — Users bring different inputs and levels of brand clarity

User interviews revealed two starting points: some people were still learning to describe their brand, while other teams had extensive material to sort through. I interviewed end users to understand their needs and what they hoped to achieve with Brand Kit.

User Interview
User interview spreadsheet
Brand Resource Reference
Brand resource reference

02 — AI agents need clear direction; users need help conveying their intent

Model and prompt testing exposed two risks: limited inputs could lead to unsupported brand facts, while too much material could dilute the direction. Image references guided generation better than text alone.

Partial Testing Results

Models under test

Read

Turn a website or images into a brand spec

GPT-4o mini vs GPT-4o

Website to brand profile

GPT-4o mini vision

Reference images to style spec

Generate

The outputs under test

GPT-4o mini

Captions, tone, review replies

gpt-image-1

Brand images, seed-paired

GPT Image 2 · Nano Banana 2 · Seedream 5 Pro

Production models the prompts map to; not scored here

Judge

Did the output follow the rule?

GPT-4o

Text judge, temperature 0

GPT-4o vision

Image judge

Code checks

Counts, word lists, color distance

Brand extraction

GPT-4o mini and GPT-4o · 4 prompt versions

Original promptv1
Role and worked examplev2
One rule per fieldv3
Low-information modev4

Captions, tone and replies

GPT-4o mini, judged by GPT-4o · 4,179 outputs

Control labels144
Tone: labels, bundles, persona729
Signal repair1,080
Sentence count and energy405
Relative tone grading135
Five tones in four business types720
Warm tone, paired rerun480
Review replies243
Replies with a word cap243

Images

gpt-image-1, judged by GPT-4o vision · 597 images

Eight style rules162
Judge-first rerun90
Thirteen art-direction specs150
Wording ladder135
One-shot vs two-step60

Amount (random on-brand)

1 image
0.25
3 images
0.45
9 images
0.59
14 images
0.56

Mix (nine images)

Curated 9
0.66
6 + 3 same category
0.44
6 + 3 other category
0.35
Everything, 23
0.35

Reading visuals is easier

System direction68%
Image prompt51%
Generic prompt37%

Control contribution

Composition+18
Image treatment+14
Palette+09
Subject+07

03 — Users need to keep brand guidance current and changes visible

Collaborating with the marketing team and founders, I mapped how brand decisions evolve through users’ choices in AI-assisted content creation. Agents across the platform need the latest brand context at each handoff, while users need to edit their preferences and see how updates affect later content.

Feature Map
User Journey Mapping
User Comments on Maintenance and Updates
User comments on maintaining and updating the brand

Agentic design ideation

The research did not go straight into wireframes. I translated each failure mode into an agent rule, a guardrail, a memory behavior, or a test case first. The interface then became the visible control layer for those rules, helping users understand, review, and steer what the agents were doing.

01 — Define how agents read, guide, and update

To define how agents analyze inputs, select references, share brand context, and coordinate content creation, I wrote brand_kit_agent.md. The specification translates research findings and testing results into explicit rules. I organized it into ten areas to guide agent behavior.

Role of the agent
The Brand Kit agent writes profile versions and supplies context. Owners edit and accept changes; reviewers approve or request changes. Generators read context without updating the profile.
Scope
Read brand materials, posts, reviews and photo choices to maintain the profile and prepare creation guidance. Content generation and publishing stay with creation tools.
Constraints
Turn caption and reply styles into explicit writing rules, and photo readings into visual plans. Preserve product facts and supplied logos, and respect HQ restrictions.
Entry points
Route onboarding, material updates, Taste regeneration and accepted feedback to the right rules. Supply context on request; learn after ten newly published posts, replies or photo choices.
Core rules
Prioritize HQ-locked values and owner corrections, select brand photos for the board, and derive the palette, visual direction, and caption and reply styles. Save changes as new profile versions.
Memory
Keep profile versions, the working board, corrections, source evidence, reviewer requests and event history so brand learning remains traceable.
Output format
Return a reviewable brand profile or task-specific context for captions, replies, images, video and campaigns, with explicit gaps, skipped sources, owner decisions and error states.
Definition of done
Check evidence-backed brand facts, board and Taste rules, active corrections and sample quality before saving a version. Validate each creation tool’s context before returning it.
Failure modes
Retry failed tools, skip unreadable brand files or links, and flag missing products or logos. Stop without saving when processing or validation fails.
Human escalation
Surface conflicting brand guidelines and reviewer requests that cannot map to a brand or Taste field. Give owners the alternatives and flag low-confidence values.

02 — Refine agent rules through prototype testing as the interface evolves

After several rounds of testing, we settled on a visual-first, nine-tile brand board. Visual references made the direction easier to discuss than text prompts alone and gave agents a focused basis for interpreting user intent.

To evaluate output quality, I connected the frontend prototype to model APIs and tested how agents interpreted brand materials and followed brand guidelines. Based on the findings, I further refined the agent rules to cover additional edge cases.

03 — Build multi-agent orchestration to connect brand guidance with cross-feature feedback

I extended the entry points, core rules, memory, and output format in the specification to cover brand updates after initial setup. Selecting new materials or requesting regeneration triggers a fresh reading by the agents; changed results become part of the handoff for brand agents to review and evaluate. The supporting flow shows how user choices feed into agent work and shape the evolving brand system.

Users and agents co-create
User, Brand Kit agent and cross-feature agents share guidance, review outputs and return feedback to the brand.
User input — Brand agent
User materials are read and filtered into visual and voice guidance, reviewed, saved and shared as task-specific brand context.
Multi-agent orchestration
Feature agents generate and check content; people review it, and scoped feedback returns to Brand Kit for owner-approved changes.

Key design decisions

Across the agentic design steps, user research and model testing helped me refine agent behavior and clarify what users could review, edit, and control. I translated those findings into the UX flows and UI design shown below.

Ideation canvas, part 1 of 3
Ideation canvas, part 2 of 3
Ideation canvas, part 3 of 3

01 — Start with what users already have

Users can begin with a sentence, a few references, or a full library. Agents fill the interpretation gap by organizing and prioritizing the evidence; users decide what is representative and correct anything uncertain before it becomes shared context.

02 — Make brand direction visible and easy to steer

Large reference sets could lead the model to hallucinate, while text prompts made visual intent difficult to judge. I designed a focused, visual-first board of up to nine references so the model received a bounded direction and users could control the same evidence the agent was using. At the same time, the Taste Panel translates that interpretation into colors, visual cues, and voice samples that users can easily read.

03 — Let brand context evolve without losing user control

A brand evolves through everyday decisions, but those decisions should not silently rewrite the source of truth. Agents can identify signals in the photos, captions, and replies users choose. They propose an update; users decide whether it belongs in the shared profile. Every accepted change creates a reviewable version, so users can see what changed, regenerate intentionally, and recover the previous direction.

Evaluation

Evaluate user experience, AI-generated content quality, and agent workflow

I evaluated the experience against three questions: Did a user action change the output in the intended direction? Did Brand Kit improve brand alignment? Was the method efficient enough for repeated use? For control tests, I held other variables constant and compared paired variants. AI provided a first-pass blind evaluation, which I audited against failure patterns rather than treating it as ground truth. The findings informed revisions to agent rules, prompt wording, and the claims the interface could make about its controls.

Evaluation Metrics
Controlled prompt variants for image tone, video direction, caption voice, and review replies
Signal specification table comparing metrics, level granularity, concrete prompts, and measured effectiveness

Impact

Impact across user experience, content quality, and system reuse

Brand Kit launched in July. The evidence has three levels: user-reported experience, controlled model evaluation, and system reuse. These metrics show that the interaction and guidance layer improved; they do not by themselves prove revenue impact.

8/10
Users satisfied with AI-generated content
User-reported satisfaction with generated content
50%
Fewer prompt iterations per post
Before: 6 iterations → After: 3
93%
Refined-tone compliance
33% → 93% in controlled evaluation
80%
Satisfaction with brand setup
Users satisfied with setting up their brand system
Reusable
Brand decisions reused across workflows
Shared context for campaigns, captions, and AI review replies

Takeaways

Design the collaboration experience between agents and users

This project changed how I define AI product design. The interface is one control layer; the agent specification, context, memory, evaluation, and recovery rules are the others. The product becomes trustworthy when AI handles uncertain interpretation while consequential decisions remain visible, reviewable, and reversible for people.

Field Research
Field research: merchant interview
Field research: observation with a merchant
Field research: reviewing the interface in context