My Role
UX Researcher
Team
1 PhD Lead
4 UX Researchers
Skills
Interaction Design
Prototyping
UX Research
Timeline
10 months
Dec. 2023 - Sep. 2024
Table of Contents
POV: Knee deep in brainstorming novel solutions? Imagine how our AI-powered plugin can leave your idea with a better value prop, with more resilience against edge cases, and more!
Scoping the Problem
Engage with different perspectives by generating reflective questions.
Consolidate your line of reasoning by generating a root cause to the problem.
Researching the Space
Get a focused snapshot of how competitors in the market stack up on each key factor.
Uncover more market players and factors by expanding your table with AI-suggested competitors and dimensions.
Impact!
61% more topic diversity
Users with AI support tended to gain exposure to more perspectives, which reflected in more well-rounded understandings & solutions.
20% faster iteration
From engaging with AI reflective questions and suggestions, AI-assisted users were able to develop their ideas faster without getting stuck on updating half-baked ideas.
It's true. They're not just accelerating productivity, but redefining how ideas are produced and refined altogether. Embedding these features directly into the workflow can reduce friction and spark new directions for creative problem-solving.
The Current Space
They also raise more questions about their usefulness long-term.
Today’s first pick might not be the same a year from now. Let's look at image generation for example: first it was DALL-E, then Midjourney took over, and now Adobe Firefly is built right into the Adobe Suite.
Tools come and go fast, and creatives have learned to be more selective with their toolkit. Adopting a new tool can be a significant decision, especially when it could mean changing habits, workflows, or sometimes even their own creative voice.
Identifying a Gap
If AI products in this space want to secure a foothold, their products have to help people become more creative instead of providing a shortcut to it. For this reason, it's risky to only be concerned with just AI performance or user satisfaction.
While so many evaluations focus on outputs, fewer study how the structure of user-AI interaction shapes the experience in terms of cognitive engagement and creative depth.
Research Question
Our Goals
To understand how the framing and placement of AI support shapes users’ creative thinking.
To evaluate if different forms of AI positioning can help users produce more creative results.
In reviewing the PhD’s earlier version of the plugin, we identified two major limitations with integrating AI into the problem-solving workflow:
Lacks user guidance
Users got stuck on how to reflect on AI-generated responses and extract key insights.
Not immediately usable
Users found AI-generated responses to be long-winded, inaccurate, repetitive, and hard to leverage.
In the interest of our research question, we needed the interactions to keep users thinking so we could focus on how they’re affected by the tool. Here are the major adjustments we made to the interface:
Original outputs were long and repetitive, which left users skimming instead of thinking.
We restructured responses into shorter insights embedded directly in the template so we could see how users actually applied them, not just skim through them.
Early participants often lost momentum because they weren’t sure what to do next.
We added lightweight guidance screens to provide context to each interactive element in the template, keeping flow without dictating what to do next.
There was a feature for users to mark AI responses as “valid” or “invalid.” In practice, it was ignored because it felt like busywork.
For AI conditions, we replaced it with subtle language nudges (e.g. “Review and edit if needed”) so reflection happened naturally within the task itself.
Rejected Explorations
Something we considered was creating an idle state that suggested next steps for exploration. Due to how non-linear the exercise is, we wanted to prevent decision paralysis by offering possible actions to keep them actively engaged.
We realized later this risked narrowing or biasing their decision-making—especially in an open-ended task where we want to observe how AI organically influences their thinking.
Rejected Explorations
In addition to populating dimensions, we explored letting users auto-generate entire rows for new competitors. The idea was to see whether giving users a full view of a single competitor might shift how they approach comparisons or structure their analysis.
We realized this risked over-structuring the competitor at once, turning it into a one-by-one review of each (just like the old plugin) instead of encouraging comparison and synthesis.
User Study Setup
We ran a between-subjects study with N=47 university participants, most with little to no design thinking experience, randomly assigned to one of three conditions:
Users approach a problem/solution without LLM assistance, manually filling out the templates based only on their current knowledge.
Users gain access to LLM generation features in specific parts of the templates, assisting with reflection or proposing alternative ideas.
Templates are already filled out by AI. Users don't have to initiate any writing, only reading and processing what was generated.
User Study Setup
To explore how different conditions think and engage in creative thinking, we looked at integrating an LLM into design templates. The key advantage comes in being both a familiar and task-oriented format for guiding users to dive into problems and develop solutions.
This allowed us to visibly observe their thinking at each stage and which content/factors inspired their end deliverables.
Data Analysis
We were curious how our plugin impacted our users’
After conducting all the sessions, the research team and I began coding timestamps for key task activities and running thematic analyses on user interviews/survey responses to pinpoint patterns.
The Golden Question
To a certain extent, yes. As expected, AI integration exposed participants to more topics and perspectives. Hence, Co-Led and AI-Led users tended to cover more categories in their understanding of the problem and their written solutions.
But quantity wasn’t the whole story.
Behavioral Insights
They spent the most time revisiting and editing earlier responses, refining half-formed ideas as their understanding evolved.
In contrast, these participants treated the AI as a dialogue partner. Their focus was on shaping responses in the moment, responding rather than revising.
On the other hand, they spent their time digesting generated content. Their process leaned less on imagination and more on remixing already provided information rather than constructing a new line of reasoning.
As a result,
They were burdened with keeping track of everything as their understanding evolved or while gathering more context, which left less time for actually synthesizing ideas together.
Users found it hard to explore beyond the AI’s suggestions because they seemed complete and convincing. And so, they spent more time deciding on their idea’s direction, leaving less room for depth and creativity.
And due to the AI’s perceived comprehensiveness, some users even accepted surface-level ideas without fully questioning them.
Users expanded their thinking with AI while staying active in shaping/challenging ideas without being overwhelmed by information.
However, having a more complex understanding made them more self-critical toward their idea, as they grappled with unanswered “what-ifs” they felt weren't easy to resolve.
Product Strategy
The patterns we saw in our study echoed a broader trend in today’s AI tools.
Much of the market still leans into one of two extremes:
Generate fast, connect later
AI can flood users with insights instantly, but this “instant gratification” risks leaving them prematurely satisfied with surface-level ideas or stunt deeper exploration.
Exclusively serve a supporting role
Respecting users’ agency is crucial for aligning with their real needs. But AI that only reacts to user input can be limiting at times — especially if the user get stuck or leaves their own assumptions unspoken.
My initial thoughts going forward was “Oh, we just need to structure AI to invite users into deeper engagement, expose them to more ideas, all while empowering their creative control”. But then, I felt a tinge of hesitation. Is that advice really enough to fuel a better experience with creative AI tools?
In many ways, the AI bubble is this generation’s dot com bubble. Early internet tools weren’t just a way to help people search, create, and have fun. They fundamentally changed the way we approach those things.
So what?
If we want to both improve creative outcomes and imagine creative thinking in a new light, we need to build past the cliché problems that come with the territory.
Justine Du discovered this quite well in her journey designing Microsoft’s Co-Pilot:
“Well, why aren’t we doing more? ... Aren’t we trying to make the Outlook experience easier so users don’t have to go into Settings? We have the freedom to put whatever we want in there.”
With this amount of potential, why stop at just doing?
AI Design Principles
From our findings and a closer look at the AI industry’s trajectory, we distilled three design principles. Across any domain, they point to an opportunity to guide users into new territory step by step, helping them bring out the best in their ideas:
Feed-Forward Prompting
In order to better align with users’ real needs, AI could proactively prime users with reflective questions that guide exploration in specific, fruitful directions.
Catering to Evolving Needs
Adapt wherever the user is—gather inspiration during ideation, help users decide during comparison, or step back when users need to focus.
Gentle Troubleshooting
If users are stuck with big ideas and bigger unknowns, AI could offer gentle scaffolding by helping users test their ideas against constraints.
This project taught me about discovering the invisible experiences that shape how people create. It also reinforced how design and research are inseparable in crafting effective user studies. Thank you UCSD Design Lab for bringing me on!
I thought research was only about pushing the envelope and finding novelty. Over these 10 months, I realized that results only matter if they reflect something about us, the people who use and get influenced by this type of technology.
It’s no secret that academic research demands rigor and many hats to be worn. While it took time to get up to speed with the literature and methods, the biggest lesson was staying present, being a self-starter, and staying hungry for more.
In hindsight, I see how easy it was to get lost in patterns and numbers without a clear anchor. We had coding schemes, but next time I’d lean less on proving absolutes and more on using data to tell stories about how people think/do.
I’m happy to walk through the process in more depth or talk about creative support tools, feel free to reach out to me at [email protected] or LinkedIn. Click the image below to read our paper!
