dave v

Researching contextual AI frameworks to evaluate user-AI interactions and support better creative outcomes

My Role

Product Designer
Product Researcher

Team

1 Research Lead
4 Designers + Researchers

Skills

Interaction Design
Prototyping
UX Research

Timeline

10 months
Dec. 2023 - Sep. 2024

Table of Contents

01 - Solution

Design smarter: An adaptive plugin that guides, challenges, and strengthens your creative problem solving

POV: Knee deep in brainstorming novel solutions? Imagine how our AI-powered plugin can leave your idea with a better value prop, with more resilience against edge cases, and more!

The work I did led to:

✅

Shipped redesigned research plugin for CHI’25 research study.

✅

Drove 61% more topic diversity + 20% faster iteration during user sprints.

Scoping the Problem

Engage with different perspectives by generating reflective questions.

Consolidate your line of reasoning by generating a root cause to the problem.

Researching the Space

Get a focused snapshot of how competitors in the market stack up on each key factor.

Uncover more market players and factors by expanding your table with AI-suggested competitors and dimensions.

02 - Background

AI tools are major disrupters in creative work

Cursor AI
Adobe Generative Fill

It's true. They're not just accelerating productivity, but redefining how ideas are produced and refined altogether. Embedding these features directly into the workflow can reduce friction and spark new directions for creative problem-solving.

Problem

Speed and seamlessness aren’t enough to secure users' trust

Criticisms about creative AI tools

If AI products in this space want to secure a foothold, their products have to help people become more creative instead of providing shortcuts or a magic fix to it. For this reason, it's risky to only be concerned with just AI performance or user satisfaction.

While so many evaluations focus on outputs, fewer study how the structure of user-AI interaction shapes the experience in terms of cognitive engagement and creative depth.

Research Question

How does the positioning of AI within a creative workflow influence creative outcomes, cognitive effort, and how users perceive their own agency and the value of the AI?

Our Goals

To understand how the framing and placement of AI support shapes users’ creative thinking.

To evaluate if different forms of AI positioning can help users produce more creative results.

03 - Design

Redesigning the original plugin prototype

In reviewing the PhD’s earlier version of the plugin, we identified two major limitations with integrating AI into the problem-solving workflow:

❌

Lacks user guidance

Users got stuck on how to reflect on AI-generated responses and extract key insights.

❌

Not immediately usable

Users found AI-generated responses to be long-winded, inaccurate, repetitive, and hard to leverage.

In the interest of our research question, we needed the interactions to keep users thinking so we could focus on how they’re affected by the tool. Here are the major adjustments we made to the interface:

Interactions with AI should be easy to comprehend and actionable.

Before picture of old plugin, showing AI generation features for the Competitive Analysis exercise. Before and after picture of generated insights.
❌

Original outputs were long and repetitive, which left users skimming instead of thinking.

✅

We restructured responses into shorter insights embedded directly in the template so we could see how users actually applied them, not just skim through them.

The plugin should guide users to keep thinking and exploring across sections without dictating exactly what to do next.

A snippet of additional guidance screens we added. A snippet of additional guidance screens we added.
❌

Early participants often lost momentum because they weren’t sure what to do next.

✅

We added lightweight guidance screens to provide context to each interactive element in the template, keeping flow without dictating what to do next.

Keep users focused on their main goals by ensuring every interaction directly supports their progress.

Before picture of old feature 'Actions to Take'. After picture of copywriting.
❌

There was a feature for users to mark AI responses as “valid” or “invalid.” In practice, it was ignored because it felt like busywork.

✅

For AI conditions, we replaced it with subtle language nudges (e.g. “Review and edit if needed”) so reflection happened naturally within the task itself.

Some design features we didn't go with.

Rejected Explorations

Possible actions to keep the flow going

We explored adding an idle state for the plugin for possible actions.

Something we considered was creating an idle state that suggested next steps for exploration. Due to how non-linear the exercise is, we wanted to prevent decision paralysis by offering possible actions to keep them actively engaged.

❌

We realized later this risked narrowing or biasing their decision-making—especially in an open-ended task where we want to observe how AI organically influences their thinking.

Rejected Explorations

Populate competitors as well

Explored allowing competitor population as well.

In addition to populating dimensions, we explored letting users auto-generate entire rows for new competitors. The idea was to see whether giving users a full view of a single competitor might shift how they approach comparisons or structure their analysis.

❌

We realized this risked over-structuring the competitor at once, turning it into a one-by-one review of each (just like the old plugin) instead of encouraging comparison and synthesis.

04 - Methods

How we constructed our research methodology

User Study Setup

Defining our conditions

We ran a between-subjects study with N=47 university participants, most with little to no design thinking experience, randomly assigned to one of three conditions:

No-AI

Users approach a problem/solution without LLM assistance, manually filling out the templates based only on their current knowledge.

Co-Led

Users gain access to LLM generation features in specific parts of the templates, assisting with reflection or proposing alternative ideas.

AI-Led

Templates are already filled out by AI. Users don't have to initiate any writing, only reading and processing what was generated.

Challenge

Aligning design, development, and research


With multiple AI conditions and overlapping features, we needed to isolate changes in user behavior to avoid confounding our study results.

We created a single source of truth that mapped each screen to a condition and feature. This kept the prototype aligned with research goals and ensured all interactions could be accurately compared and implemented.

05 - Results

Results & Takeaways

The Golden Question

Can AI empower users to be more creative?

AI conditions had more idea units.

To a certain extent, yes. As expected, AI integration exposed participants to more topics and perspectives. Hence, Co-Led and AI-Led users tended to cover more categories in their understanding of the problem and their written solutions.

But quantity wasn’t the whole story.

Behavioral Insights

Differences emerged in how participants engaged with the template.

No-AI

They spent the most time revisiting and editing earlier responses, refining half-formed ideas as their understanding evolved.

No-AI participants' engagement behavior
Co-Led

Re: Users got stuck on how to reflect on AI-generated responses...

→ In contrast, these participants treated the AI as a dialogue partner. Their focus was on shaping responses in the moment, responding rather than revising.

Co-Led participants' engagement behavior
AI-Led

Re: Users found AI-generated responses…hard to leverage

→ They spent time digesting generated content. Less on imagination and more on remixing already provided information rather than a new line of reasoning.

AI-Led participants' engagement behavior

As a result,

While No-AI participants expressed more confidence and ownership,

No-AI participants' feeling about the process

They were burdened with keeping track of everything as their understanding evolved or while gathering more context, which left less time for actually synthesizing ideas together.

While AI-led participants had exposure to more perspectives and topics early on,

AI-Led participants' feeling about the process

Users found it hard to explore beyond the AI’s suggestions because they seemed complete and convincing. And so, they spent more time deciding on their idea’s direction, leaving less room for depth and creativity.

And due to the AI’s perceived comprehensiveness, some users even accepted surface-level ideas without fully questioning them.

Co-Led participants struck more of a happy medium, but...

Co-Led participants' feeling about the process

Users expanded their thinking with AI while staying active in shaping/challenging ideas without being overwhelmed by information.

However, having a more complex understanding made them more self-critical toward their idea, as they grappled with unanswered “what-ifs” they felt weren't easy to resolve.

Product Strategy

What might creative AI support look like moving forward?

From our findings and a survey of the AI design landscape, we distilled three principles for designing AI experiences that product teams can apply for better user outcomes (whether you’re designing for user intent or agentic integrations).

Re: “hard to explore beyond the AI’s suggestions”

Proactively prime users with reflective prompts to guide exploration

In order to better align with users’ real needs, AI could proactively prime users with reflective questions that guide exploration in specific, fruitful directions.

A pop-up modal that appeared when a user fixated on a solution before understanding the problem and audience.

Re: “burdened with keeping track of everything”

Say bye to context switching, hello to contextually-aware support

Adapt wherever the user is—gather inspiration during ideation, help users decide during comparison, or step back when users need to focus.

AI serving in mulitple contexts seamlessly (in this case, gathering sources for the user & comparing two designs).

Re: “more self-critical towards their idea, finding difficulty in resolving the bigger threats”

Ensure users feel confident and in control while navigating complexity

If users are stuck with big ideas and bigger unknowns, AI could offer gentle scaffolding by helping users test their ideas against constraints.

User asks a question using cursor chat to help troubleshoot viability of their idea.
07 - Reflection

Informing design guidelines for future AI-assisted creativity tools

Impact

Goal: Understand how the framing and placement of AI support shapes users’ creative thinking

→

From engaging with AI reflective questions and suggestions, Co-led users were able to develop their ideas 20% faster without getting stuck or prematurely settling on half-baked ideas.

Goal: Evaluate if different forms of AI positioning can help users produce more creative results

→

AI support expanded users’ perspectives and helped users synthesize insights they wouldn’t have thought of originally. Resulted in 61% higher topic diversity in final problem statements and solutions.

Learning

Humanizing our key findings

I thought research was only about pushing the envelope and finding novelty. Over these 10 months, I realized that results only matter if they reflect something about us, the people who use and get influenced by this type of technology.

Challenge

Doing academic research for the first time

It’s no secret that academic research demands rigor and many hats to be worn. While it took time to get up to speed with the literature and methods, the biggest lesson was staying present, being a self-starter, and staying hungry for more.

What I would have done differently

Starting with a storytelling perspective

In hindsight, I see how easy it was to get lost in patterns and numbers without a clear anchor. We had coding schemes, but next time I’d lean less on proving absolutes and more on using data to tell stories about how people think/do.

Have more questions about our paper?

I’m happy to walk through the process in more depth or talk about creative support tools, feel free to reach out to me at [email protected] or LinkedIn. Click the image below to read our paper!

Link to our paper