Summary
LLMs are everywhere, but when it comes to real research, they often fall short. Generic LLMs weren’t built for continuous research workflows, and product researchers quickly see the problem: the outputs are generic, lack full context, and struggle to connect multiple data sources. Instead of surfacing meaningful insights, they can amplify noise. In this session, Daniel will break down why AI often fails research teams and what’s missing. He’ll show how to make AI actually useful for continuous product research. Accelerating analysis, connecting insights across sources, and keeping researchers at the center, equipped with a powerful tool rather than replaced by one.
Key Insights
-
•
AI in research struggles with large datasets, often averaging results and missing subtle but important signals.
-
•
Curating and filtering datasets by removing irrelevant data improves AI research output quality.
-
•
Scoping research into focused projects or topics helps AI deliver more precise responses.
-
•
Asking one question at a time significantly enhances the quality of AI-generated answers.
-
•
Providing detailed contextual information (personas, company background, product details) to AI boosts specificity and nuance in responses.
-
•
AI hallucinations and trust issues necessitate human-in-the-loop processes to verify output quality and citations.
-
•
Iterative refinement of AI outputs, similar to app development, is critical for achieving polished research results.
-
•
Spot checking AI-generated citations can be an effective and efficient way to validate research quality.
-
•
Context passed as embedded knowledge rather than repeated in prompts yields better AI results.
-
•
Using multiple specialized AI agents to critique each other’s outputs can mitigate bias and improve research accuracy.
Notable Quotes
"AI has this strange weakness that when working with a large dataset, they often miss crucial, subtle findings."
"The larger the dataset you work with, the more costly it is to run a single operation on AI models."
"Whenever possible, you should be breaking down your work into specific research projects or topics."
"When you ask a question, try to ask one at a time so the model doesn't get lost."
"Context is everything — providing AI with a folder of your company’s knowledge makes responses more detailed and useful."
"Research with AI requires as much iteration and verification as building an app or prototype."
"AI-generated research reports should always be tied to real feedback that you can verify behind every sentence."
"There's no way to deny it: every industry needs to adapt to AI, but nobody really knows how yet."
"Human in the loop means constantly interacting with AI, documenting your thoughts and assuring quality."
"Some engineers build a council of agents that debate and generate responses, which can help with bias and accuracy."
Or choose a question:
More Videos
"When we showed the Iceland Ministry of Foreign Affairs their portfolio, the Foreign Minister said, we have to pivot our innovation investments to early-stage catalytic innovations."
Milan Guenther Benjamin KumpfThe $212 billion ‘so what?’: unlocking impact in development cooperation
November 20, 2025
"Build a program management community of practice so you can formalize partnerships and solve process delivery issues together."
Alnie FigueroaTeamwork: Strategies for Effective Collaboration with Other Program Management Teams
September 8, 2022
"Adoption is the only thing that matters in innovation. If you’re not changing behavior, you haven’t innovated."
Saara Kamppari-Miller Nicole Bergstrom Shashi JainKey Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”
November 13, 2024
"This is a topic near and dear to my heart and I feel really long overdue conversation."
Christian CrumlishIntroduction by our Conference Chair
December 6, 2022
"Working with AI is a lot less like working with another human and more like working with some weird force of nature."
Helen ArmstrongAugment the Human. Interrogate the System.
June 7, 2023
"The Silicon Valley dream is a problematic ideology that normalizes what the correct research looks like and excludes others."
Verónica Urzúa Jorge MontielThe B-side of the Research Impact
March 12, 2021
"Designers adhere to standards not scientific ones, but design rigor tailored to dealing with complexity."
Yoel SumitroActions and Reflections: Bridging the Skills Gap among Researchers
March 9, 2022
"Each first take was the only take, which got pressed on the album."
Jim KalbachJazz Improvisation as a Model for Team Collaboration
June 4, 2019
"I see design ops as deeply linked to how internal operations influence what our customers experience."
Bud Caddell Kristin Skinner Alana WashingtonDesignOps Community Sensing Session
May 13, 2021
Latest Books All books
Dig deeper with the Rosenbot
How do UX skills apply in customer operations roles within a clean energy startup like Vayu?
In what ways can AI speed up qualitative data analysis without compromising insight quality?
What does a successful healthcare UX career look like in terms of accumulating influence and aligning with clinical/business goals?