My Role
UX Researcher
Team
1 PhD Lead
4 UX Researchers
Skills
Interaction Design
Prototyping
UX Research
Timeline
10 months
Dec. 2023 - Sep. 2024
Table of Contents
POV: Knee deep in brainstorming novel solutions? Imagine how our AI-powered plugin can leave your idea with a better value prop, with more resilience against edge cases, and more!
Scoping the Problem
Engage with different perspectives by generating reflective questions.
Consolidate your line of reasoning by generating a root cause to the problem.
Researching the Space
Get a focused snapshot of how competitors in the market stack up on each key factor.
Uncover more market players and factors by expanding your table with AI-suggested competitors and dimensions.
It's true. They're not just accelerating productivity, but redefining how ideas are produced and refined altogether. Embedding these features directly into the workflow can reduce friction and spark new directions for creative problem-solving.
The Current Space
They also raise more questions about their usefulness long-term.
Today’s first pick might not be the same a year from now. Let's look at image generation for example: first it was DALL-E, then Midjourney took over, and now Adobe Firefly is built right into the Adobe Suite.
Tools come and go fast, and creatives have learned to be more selective with their toolkit. Adopting a new tool can be a significant decision, especially when it could mean changing habits, workflows, or sometimes even their own creative voice.
Identifying a Gap
If AI products in this space want to secure a foothold, their products have to help people become more creative instead of providing a shortcut to it. For this reason, it's risky to only be concerned with just AI performance or user satisfaction.
While so many evaluations focus on outputs, fewer study how the structure of user-AI interaction shapes the experience in terms of cognitive engagement and creative depth.
Research Question
Our Goals
To understand how the framing and placement of AI support shapes users’ creative thinking.
To evaluate if different forms of AI positioning can help users produce more creative results.
In reviewing the PhD’s earlier version of the plugin, we identified two major limitations with integrating AI into the problem-solving workflow:
Lacks user guidance
Users got stuck on how to reflect on AI-generated responses and extract key insights.
Not immediately usable
Users found AI-generated responses to be long-winded, inaccurate, repetitive, and hard to leverage.
In the interest of our research question, we needed the interactions to keep users thinking so we could focus on how they’re affected by the tool. Here are the major adjustments we made to the interface:
Original outputs were long and repetitive, which left users skimming instead of thinking.
We restructured responses into shorter insights embedded directly in the template so we could see how users actually applied them, not just skim through them.
Early participants often lost momentum because they weren’t sure what to do next.
We added lightweight guidance screens to provide context to each interactive element in the template, keeping flow without dictating what to do next.
There was a feature for users to mark AI responses as “valid” or “invalid.” In practice, it was ignored because it felt like busywork.
For AI conditions, we replaced it with subtle language nudges (e.g. “Review and edit if needed”) so reflection happened naturally within the task itself.
Design Decisions
The initial plugin had these AI-generated Q&As to prompt reflection. However, previous participants felt like they were repetitive summaries without offering specific angles or concrete ways to explore a competitor further.
We found that users struggled to act on AI insights because they had to sift through long, repetitive text before rewriting their own takeaways in the template.
To close that gap, we shifted to concise, targeted insights embedded directly in the template—letting users cut straight to the action.
For us as researchers, this also made it easier to observe how they synthesized and applied AI input, rather than just skimming it.
Design Decisions
As mentioned previously, there was no smooth handoff between reading AI insights and applying them. Users were often left unsure of what to do next, breaking their momentum instead of advancing their analysis.
So we decided to expand the interface to include guidance screens for every interactive element in the table, not just competitor headers or add-buttons.
These screens provide light context on what each element is, how users can interact with it, and what role it plays in the overall exercise, keeping exploration fluid without defining a fixed path.
Design Decisions
The previous researchers included a feature for users to mark AI-generated content as “valid” or “invalid.” In practice, most skipped it—seeing it as an extra task that broke their flow and pulled focus away from research and synthesis.
We decided to not move forward with this feature and instead embedded simple, contextual language like “Review and edit if needed” or “This may or may not reflect your own analysis.”
This subtly encourages reflection without pulling users out of the task.
Rejected Explorations
Something we considered was creating an idle state that suggested next steps for exploration. Due to how non-linear the exercise is, we wanted to prevent decision paralysis by offering possible actions to keep them actively engaged.
We realized later this risked narrowing or biasing their decision-making—especially in an open-ended task where we want to observe how AI organically influences their thinking.
Rejected Explorations
In addition to populating dimensions, we explored letting users auto-generate entire rows for new competitors. The idea was to see whether giving users a full view of a single competitor might shift how they approach comparisons or structure their analysis.
We realized this risked over-structuring the competitor at once, turning it into a one-by-one review of each (just like the old plugin) instead of encouraging comparison and synthesis.
User Study Setup
We ran a between-subjects study with N=47 university participants, most with little to no design thinking experience, randomly assigned to one of three conditions:
Users approach a problem/solution without LLM assistance, manually filling out the templates based only on their current knowledge.
Users gain access to LLM generation features in specific parts of the templates, assisting with reflection or proposing alternative ideas.
Templates are already filled out by AI. Users don't have to initiate any writing, only reading and processing what was generated.
User Study Setup
To explore how different conditions think and engage in creative thinking, we looked at integrating an LLM into design templates. The key advantage comes in being both a familiar and task-oriented format for guiding users to dive into problems and develop solutions.
This allowed us to visibly observe their thinking at each stage and which content/factors inspired their end deliverables.
Data Analysis
We were curious how our plugin impacted our users’
After conducting all the sessions, the research team and I began coding timestamps for key task activities and running thematic analyses on user interviews/survey responses to pinpoint patterns.
The Golden Question
To a certain extent, yes. As expected, AI integration exposed participants to more topics and perspectives. Hence, Co-Led and AI-Led users tended to cover more categories in their understanding of the problem and their written solutions.
But quantity wasn’t the whole story.
Behavioral Insights
They spent the most time revisiting and editing earlier responses, refining half-formed ideas as their understanding evolved.
In contrast, these participants treated the AI as a dialogue partner. Their focus was on shaping responses in the moment, responding rather than revising.
On the other hand, they spent their time digesting generated content. Their process leaned less on imagination and more on remixing already provided information rather than constructing a new line of reasoning.
As a result,
They were burdened with keeping track of everything as their understanding evolved or while gathering more context, which left less time for actually synthesizing ideas together.
Users found it hard to explore beyond the AI’s suggestions because they seemed complete and convincing. And so, they spent more time deciding on their idea’s direction, leaving less room for depth and creativity.
And due to the AI’s perceived comprehensiveness, some users even accepted surface-level ideas without fully questioning them.
Users expanded their thinking with AI while staying active in shaping/challenging ideas without being overwhelmed by information.
However, having a more complex understanding made them more self-critical toward their idea, as they grappled with unanswered “what-ifs” they felt weren't easy to resolve.
Product Strategy
The patterns we saw in our study echoed a broader trend in today’s AI tools.
Much of the market still leans into one of two extremes:
Generate fast, connect later
AI can flood users with insights instantly, but this “instant gratification” risks leaving them prematurely satisfied with surface-level ideas or stunt deeper exploration.
Exclusively serve a supporting role
Respecting users’ agency is crucial for aligning with their real needs. But AI that only reacts to user input can be limiting at times — especially if the user get stuck or leaves their own assumptions unspoken.
My initial thoughts going forward was “Oh, we just need to structure AI to invite users into deeper engagement, expose them to more ideas, all while empowering their creative control”. But then, I felt a tinge of hesitation. Is that advice really enough to fuel a better experience with creative AI tools?
In many ways, the AI bubble is this generation’s dot com bubble. Early internet tools weren’t just a way to help people search, create, and have fun. They fundamentally changed the way we approach those things.
So what?
If we want to both improve creative outcomes and imagine creative thinking in a new light, we need to build past the cliché problems that come with the territory.
Justine Du discovered this quite well in her journey designing Microsoft’s Co-Pilot:
“Well, why aren’t we doing more? ... Aren’t we trying to make the Outlook experience easier so users don’t have to go into Settings? We have the freedom to put whatever we want in there.”
With this amount of potential, why stop at just doing?
AI Design Principles
From our findings and a closer look at the AI industry’s trajectory, we distilled three design principles. Across any domain, they point to an opportunity to guide users into new territory step by step, helping them bring out the best in their ideas:
Feed-Forward Prompting
In order to better align with users’ real needs, AI could proactively prime users with reflective questions that guide exploration in specific, fruitful directions.
Catering to Evolving Needs
Adapt wherever the user is—gather inspiration during ideation, help users decide during comparison, or step back when users need to focus.
Gentle Troubleshooting
If users are stuck with big ideas and bigger unknowns, AI could offer gentle scaffolding by helping users test their ideas against constraints.
Iteration
In the early stages of data analysis, we were primarily concerned with which conditions expanded more, who had the more defined angle, who went deeper, and so on.
This lens captured differences in behavior—yet it didn’t explain why they happened, or how they connected to the experience of being creative.
Each pass gave us a new angle on what really shaped participants’ behavior. AI wasn’t just a well of extra ideas—it actively shifted how people approached the task (for better or for worse). Same data, different interpretations.
An Unexpected Turn
In a pre-survey, we asked users if they had any design thinking background. Then, experts blindly rated user responses across Likert-scale benchmarks, which we graphed against experience levels.
We initially anticipated a clear correlation—but the data suggested something more complex.
Most expert ratings aligned with the quality of the participants’ solutions—but a few outliers reminded us that strong ideas don’t always look polished.
One participant, for instance, proposed a loosely structured solution around adaptive cultural change in education.
Experts blindly rated their solution low in several areas due to the lack of “how” this solution would be implemented and “how” this affects its audience.
Yet in their interview, they demonstrated clear signs of deep reflection: they questioned the AI’s logic, merged multiple ideas, and grounded their decisions in a human-centered way.
The Lightbulb Moment
Cases like those highlighted a key nuance in evaluating creativity: strong thinking isn’t always neatly packaged.
This further drove our final thesis, saying that AI tools should go beyond generating outputs. Instead, they should anchor their value propositions in providing structures for nuanced thinking and fluid ways to elevate their creative workflows.
That’s the real value.
This project taught me about discovering the invisible experiences that shape how people create. It also reinforced how design and research are inseparable in crafting effective user studies. Thank you UCSD Design Lab for bringing me on!
I thought research was only about pushing the envelope and finding novelty. Over these 10 months, I realized that results only matter if they reflect something about us, the people who use and get influenced by this type of technology.
It’s no secret that academic research demands rigor and many hats to be worn. While it took time to get up to speed with the literature and methods, the biggest lesson was staying present, being a self-starter, and staying hungry for more.
In hindsight, I see how easy it was to get lost in patterns and numbers without a clear anchor. We had coding schemes, but next time I’d lean less on proving absolutes and more on using data to tell stories about how people think/do.
I’m happy to walk through the process in more depth or talk about creative support tools, feel free to reach out to me at [email protected] or LinkedIn. Click the image below to read our paper!
