Rosenverse

Why AI Is Bad at Research (and how to make it actually useful)

Gold
Tuesday, March 10, 2026 • Advancing Research 2026
Share the love for this talk
Why AI Is Bad at Research (and how to make it actually useful)
Speakers: Daniel Korczynski
Link:

Summary

LLMs are everywhere, but when it comes to real research, they often fall short. Generic LLMs weren’t built for continuous research workflows, and product researchers quickly see the problem: the outputs are generic, lack full context, and struggle to connect multiple data sources. Instead of surfacing meaningful insights, they can amplify noise. In this session, Daniel will break down why AI often fails research teams and what’s missing. He’ll show how to make AI actually useful for continuous product research. Accelerating analysis, connecting insights across sources, and keeping researchers at the center, equipped with a powerful tool rather than replaced by one.

Key Insights

  • AI in research struggles with large datasets, often averaging results and missing subtle but important signals.

  • Curating and filtering datasets by removing irrelevant data improves AI research output quality.

  • Scoping research into focused projects or topics helps AI deliver more precise responses.

  • Asking one question at a time significantly enhances the quality of AI-generated answers.

  • Providing detailed contextual information (personas, company background, product details) to AI boosts specificity and nuance in responses.

  • AI hallucinations and trust issues necessitate human-in-the-loop processes to verify output quality and citations.

  • Iterative refinement of AI outputs, similar to app development, is critical for achieving polished research results.

  • Spot checking AI-generated citations can be an effective and efficient way to validate research quality.

  • Context passed as embedded knowledge rather than repeated in prompts yields better AI results.

  • Using multiple specialized AI agents to critique each other’s outputs can mitigate bias and improve research accuracy.

Notable Quotes

"AI has this strange weakness that when working with a large dataset, they often miss crucial, subtle findings."

"The larger the dataset you work with, the more costly it is to run a single operation on AI models."

"Whenever possible, you should be breaking down your work into specific research projects or topics."

"When you ask a question, try to ask one at a time so the model doesn't get lost."

"Context is everything — providing AI with a folder of your company’s knowledge makes responses more detailed and useful."

"Research with AI requires as much iteration and verification as building an app or prototype."

"AI-generated research reports should always be tied to real feedback that you can verify behind every sentence."

"There's no way to deny it: every industry needs to adapt to AI, but nobody really knows how yet."

"Human in the loop means constantly interacting with AI, documenting your thoughts and assuring quality."

"Some engineers build a council of agents that debate and generate responses, which can help with bias and accuracy."

Ask the Rosenbot
Lena Shenkarenko
Collaborative Wireframing for Creating Team Alignment and Shipping Better Products
2020 • DesignOps Summit 2020
Gold
Uday Gajendar
The Rise of Meta-Design: A Starter Playbook
2022 • Enterprise Community
Anna Avrekh
Diversity In and For Design: Building Conscious Diversity in Design and Research
2021 • Design at Scale 2021
Gold
Liam Thurston
Why Your Design Team Is Quitting, And How To Fix It
2022 • Design at Scale 2022
Gold
Bob Baxley
Theme 4: Intro
2024 • Enterprise Experience 2020
Gold
Clara Kliman-Silver
UX Futures: The Role of Artificial Intelligence in Design
2023 • Enterprise UX 2023
Gold
Deanna Zandt
The Unspoken Complexity of “Self-Care” with Deanna Zandt
2022 • Civic Design Community
Sam Proulx
Accessibility: An Opportunity to Innovate
2022 • Design at Scale 2022
Gold
Dean Broadley
Not Black Enough to be White
2024 • DesignOps Summit 2020
Gold
Husani Oakley
Theme Three Intro
2023 • Enterprise UX 2023
Gold
Llewyn Paine
Day 1 Using AI in UX with Impact
2025 • Designing with AI 2025
Gold
James Wieselman Schulman
Research is a team sport: advancing the work when everyone does the research
2026 • Advancing Research 2026
Gold
Dave Malouf
Closing Keynote: Amplify. Not Optimize.
2019 • DesignOps Summit 2019
Gold
Deirdre Hirschtritt
Research is Only as Good as the Relationships You Build
2022 • Civic Design 2022
Gold
Nick Cochran
Growing in Enterprise Design through Making Connections
2019 • Enterprise Community
Dana Bishop
2022: The Year UX Demonstrates its Business Impact
2022 • Advancing Research 2022
Gold

More Videos

Daniel Gloyd

"The Shakers patented clever affordances like chair buttons that let you rock without damaging wooden floors."

Daniel Gloyd

Warming the User Experience: Lessons from America's first and most radical human-centered designers

May 9, 2024

Aletheia Delivre

"Data is power; find your eigen metric, the one or two things that prove your hypothesis."

Aletheia Delivre

New Shapes and Emerging Identities for Design Ops

September 11, 2025

Dr. Jamika D. Burge

"Women are more likely to die in car crashes because crash test dummies are not designed with women in mind."

Dr. Jamika D. Burge Mansi Gupta

Advancing the Inclusion of Womxn in Research Practices

September 15, 2022

Peter Van Dijck

"Everything’s changing really quickly—if you don’t revise your plan in a few months, you’re building for a world that’s already outdated."

Peter Van Dijck

Designing AI-first products on top of a rapidly evolving technology

June 10, 2025

Ricardo Martins

"Cluster analysis is very useful if you want to avoid a one-size-fits-all approach to product or service design."

Ricardo Martins

Unlocking the power of advanced quantitative methods

March 12, 2025

Emily Danielson

"We turned all of these files in boxes into an Excel file, and then into a shared Google Doc accessible to multiple people."

Emily Danielson

“I mean, I can lift a shovel”: Design Skills in Disaster Response

June 9, 2022

Bria Alexander

"We are on our third and final day of Advancing Research 2023."

Bria Alexander

Opening Remarks

March 29, 2023

Jerome “Axle” Brown

"Sometimes the loudest voices dominate, but asynchronous collaboration helps give quieter voices a chance to speak."

Jerome “Axle” Brown

How to Use Self-Directed Learning to Ensure Your Research Insights are Heard and Acted Upon

March 11, 2021

Sarah Williams

"Principles at this level act like a company mission that product teams can translate into their own more specific design principles."

Sarah Williams

A Framework for CX Transformation

June 11, 2021