Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Qualitative synthesis with ChatGPT: Better or worse than human intelligence?
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
Qualitative synthesis with ChatGPT: Better or worse than human intelligence?
Speakers: Weidan Li
Link:

Summary

Following the emergence of Generative AI as a potential revolution in the UX field, a great deal of AI-driven tools arose to enhance the efficiency of UX research, including data analysis. Qualitative data analysis is a process that conventionally relies on human intelligence to discern patterns, establish connections, and derive actionable insights and frameworks. Many studies have involved comparing the quality of qualitative analyses generated by humans with those produced by AI language models like ChatGPT (Hamilton et al., 2023). Despite the undeniable appeal of automation and speed, there is ongoing debate about AI’s ability to replace human intelligence in qualitative analysis, which may be unlikely at this moment. Then the question is: To what extent can AI contribute to qualitative data analysis? In this case study, I delved into the thematic analysis and post-analysis stage, i.e. synthesizing insights into a framework. Framework, in this context, refers to a conceptual structure that illustrates the components of a human experience and how the components interconnect and operate within the structure. It is a concise model that encapsulates the entirety of research insights. The topic of my case study is "trust relationships between job seekers and hirers in the marketplace,, aligning with the business focus of my company. From my secondary research, I found that, ChatGPT needed multiple rounds of training using diverse prompts to conduct precise and comprehensive thematic analysis. ChatGPT can execute fine-quality thematic analysis under the help of right prompts, yet it falls short in replacing human intelligence for synthesizing insights and crafting frameworks for engaging narratives. Its limitation lies in lacking the depth of contextual understanding within a company, such as understanding what’s missing from the company’s mainstream discourse to create a human-centered story based on data analysis. To craft a framework that conveys good storytelling and organizational impact, it requires the researcher's introspection into knowledge gaps in the specific organizational context. Thus, the best practice is to combine human interpretation and AI production. In my talk, I will demonstrate several principles to guide this practice. Takeaways We’ll cover principles of how to employ ChatGPT in qualitative analysis, specifically focusing on its application in synthesizing and crafting frameworks that convey compelling and insightful narratives: Effectiveness of ChatGPT in thematic analysis: Learn about my process of training ChatGPT to conduct precise thematic analysis. You’ll gain insights into the capabilities and limitations of ChatGPT in providing accurate and comprehensive analysis for framework construction Value of human potential: We’ll address the value of human self-reflection and the ability of interpreting organizational context for crafting engaging frameworks Comparison between human and ChatGPT: By comparing the human-driven outcomes against ChatGPT for qualitative analysis, you’ll see how effective the synthesized frameworks are generated by the researcher and ChatGPT separately. Collaboration between human and ChatGPT: You’ll also learn when and how to incorporate human interpretation with ChatGPT to achieve the best practice in qualitative analysis and synthesis

Key Insights

  • ChatGPT is more thorough and comprehensive than a fatigued human analyst, capturing details humans may miss.

  • ChatGPT struggles to create proper hierarchical organizing themes, often equating them in granularity with basic themes.

  • Human-generated frameworks excel in illustrating connections and flows between concepts, which ChatGPT tends to present as siloed, overlapping components.

  • AI-generated analogies, while existent, lack the coherence and nuance of human metaphors that effectively communicate complex relationships.

  • Qualitative synthesis requires subjective first-person experience and curiosity, qualities AI inherently lacks.

  • Users can use ChatGPT effectively as a tool to validate, triangulate, and summarize qualitative data, supplementing human analysis.

  • Thematic networks benefit from a visual and iterative human process, facilitating mental models that AI's bullet-point summaries do not provide.

  • Prompt engineering and role specification in ChatGPT do not currently resolve fundamental limitations in framework generation.

  • AI synthesis results should be viewed as stimuli for human curiosity and interpretation, not as definitive conclusions.

  • The most advanced qualitative synthesis goals, such as creating engaging analogies, remain a largely human prerogative due to AI's lack of experiential learning.

Notable Quotes

"ChatGPT has a big advantage in being thorough and comprehensive, especially compared to myself."

"Many of ChatGPT's organizing themes are at the same level as the basic themes, which is confusing and inappropriate."

"ChatGPT's frameworks generate siloed insights but fail to show the connections and flows essential for storytelling."

"Using analogy is very difficult for AI because it requires a unique human ability to connect and engage through storytelling."

"Qualitative synthesis is like watching a movie in a cinema; AI synthesis feels like watching a recap on YouTube—fast but missing the immersive experience."

"AI can’t experience data subjectively or fulfill curiosity, which drives human insight and discovery in qualitative research."

"We should not replace our curiosity and experience with AI but should instead use AI to speed up summarization and then slow down for human sensemaking."

"I tried different prompts and roles, but they didn’t fundamentally improve ChatGPT’s ability to generate meaningful frameworks."

"ChatGPT’s approach in analogy generation segmented the narrative into unrelated metaphors, which confused the overall story."

"AI synthesis should be considered a human-machine mutual learning process, with AI stimulating new perspectives rather than providing conclusions."

Ask the Rosenbot
Anna Poznyakov
Get The Most Out Of Stakeholder Collaboration—and Maximize Your Research Impact
2021 • Advancing Research 2021
Gold
Mike Davidson
Fireside Chat
2022 • Advancing Research 2022
Gold
John Calhoun
Bring your DesignOps Story to Life! The Definitive DesignOps Book Jam
2023 • DesignOps Summit 2023
Gold
Abbey Smalley
Today’s Design Ops and Programs Landscape & Career Paths
2023 • DesignOps Summit 2023
Gold
Kate Towsey
ResearchOps AMA with Kate Towsey & Jake Burghardt
2025 • Advancing Research Community
Bassel Deeb
Do More With Less: Equip and Lead Design Orgs Through Adversity
2023 • DesignOps Summit 2023
Gold
Sharbani Dhar
Breathing Room for Delight
2024 • Enterprise Experience 2020
Gold
James Rampton
The Basics of Automotive UX & Why Phones Are a Part of That Future
2024 • Rosenfeld Community
Shanti Mathew
Civic Design at Scale: Introducing the Public Policy Layer Cake
2021 • Civic Design 2021
Gold
Ted Booth
Discussion
2016 • Enterprise UX 2016
Gold
Anna Avrekh
Expert Panel: Leading in and with Research
2022 • Advancing Research 2022
Gold
Joe Meersman
Use AI to Drive Outcomes that Go Beyond the Design Sprint
2024 • DesignOps Summit 2024
Gold
Kit Unger
Theme 2: Introduction
2021 • Design at Scale 2021
Gold
Dave Hora
Advice for Establishing Research
2022 • Advancing Research Community
Courtney Kaplan
Taking it to the next level: Career paths in DesignOps
2018 • DesignOps Summit 2018
Gold
Gabrielle Verderber
Documentation Your Team Will Actually Use
2023 • DesignOps Summit 2023
Gold

More Videos

Doug Powell

"We are truly in a golden age for our profession."

Doug Powell

Closing Keynote: Design at Scale

November 8, 2018

Mila Kuznetsova

"If a group session with middle schoolers goes off the rails, one-on-one sessions can save the research."

Mila Kuznetsova Lucy Denton

How Lessons Learned from Our Youngest Users Can Help Us Evolve our Practices

March 9, 2022

Sarah Gallimore

"Signals are something that’s happening at the horizon, percolating at the fringes of society or in a narrow subset of the population."

Sarah Gallimore

Inspire Progress with Artifacts from the Future

November 18, 2022

Lada Gorlenko

"Field usability testing showed stakeholders immediately how much value UX can add by catching issues before deployment."

Lada Gorlenko Sharbani Dhar Sébastien Malo Rob Mitzel Ivana Ng Michal Anne Rogondino

Theme 1: Discussion

January 8, 2024

Alnie Figueroa

"It is our job to lead and transform with empathy and heart because there’s fear in everyone."

Alnie Figueroa

The Future of Design Operations: Transforming Our Craft

September 10, 2025

Landon Barnes

"Customers benchmark your company to their last best experience, often outside of your industry."

Landon Barnes

Are My Research Findings Actually Meaningful?

March 10, 2022

Emily Eagle

"I thrive on learning about how people overcome obstacles."

Emily Eagle

Can't Rewind: Radio and Retail

June 3, 2019

Malini Rao

"Keep a pulse on your team morale because re-platforming journeys can either enrich or burn out your people."

Malini Rao

Lessons Learned from a 4-year Product Re-platforming Journey

June 9, 2021

Asia Hoe

"Building rapport and rituals helped our remote team foster camaraderie and trust."

Asia Hoe

Partnering with Product: A Journey from Junior to Senior Design

November 29, 2023