Qualitative synthesis with ChatGPT: Better or worse than human intelligence?
Summary
Following the emergence of Generative AI as a potential revolution in the UX field, a great deal of AI-driven tools arose to enhance the efficiency of UX research, including data analysis. Qualitative data analysis is a process that conventionally relies on human intelligence to discern patterns, establish connections, and derive actionable insights and frameworks. Many studies have involved comparing the quality of qualitative analyses generated by humans with those produced by AI language models like ChatGPT (Hamilton et al., 2023). Despite the undeniable appeal of automation and speed, there is ongoing debate about AI’s ability to replace human intelligence in qualitative analysis, which may be unlikely at this moment. Then the question is: To what extent can AI contribute to qualitative data analysis? In this case study, I delved into the thematic analysis and post-analysis stage, i.e. synthesizing insights into a framework. Framework, in this context, refers to a conceptual structure that illustrates the components of a human experience and how the components interconnect and operate within the structure. It is a concise model that encapsulates the entirety of research insights. The topic of my case study is "trust relationships between job seekers and hirers in the marketplace,, aligning with the business focus of my company. From my secondary research, I found that, ChatGPT needed multiple rounds of training using diverse prompts to conduct precise and comprehensive thematic analysis. ChatGPT can execute fine-quality thematic analysis under the help of right prompts, yet it falls short in replacing human intelligence for synthesizing insights and crafting frameworks for engaging narratives. Its limitation lies in lacking the depth of contextual understanding within a company, such as understanding what’s missing from the company’s mainstream discourse to create a human-centered story based on data analysis. To craft a framework that conveys good storytelling and organizational impact, it requires the researcher's introspection into knowledge gaps in the specific organizational context. Thus, the best practice is to combine human interpretation and AI production. In my talk, I will demonstrate several principles to guide this practice. Takeaways We’ll cover principles of how to employ ChatGPT in qualitative analysis, specifically focusing on its application in synthesizing and crafting frameworks that convey compelling and insightful narratives: Effectiveness of ChatGPT in thematic analysis: Learn about my process of training ChatGPT to conduct precise thematic analysis. You’ll gain insights into the capabilities and limitations of ChatGPT in providing accurate and comprehensive analysis for framework construction Value of human potential: We’ll address the value of human self-reflection and the ability of interpreting organizational context for crafting engaging frameworks Comparison between human and ChatGPT: By comparing the human-driven outcomes against ChatGPT for qualitative analysis, you’ll see how effective the synthesized frameworks are generated by the researcher and ChatGPT separately. Collaboration between human and ChatGPT: You’ll also learn when and how to incorporate human interpretation with ChatGPT to achieve the best practice in qualitative analysis and synthesis
Key Insights
-
•
ChatGPT is more thorough and comprehensive than a fatigued human analyst, capturing details humans may miss.
-
•
ChatGPT struggles to create proper hierarchical organizing themes, often equating them in granularity with basic themes.
-
•
Human-generated frameworks excel in illustrating connections and flows between concepts, which ChatGPT tends to present as siloed, overlapping components.
-
•
AI-generated analogies, while existent, lack the coherence and nuance of human metaphors that effectively communicate complex relationships.
-
•
Qualitative synthesis requires subjective first-person experience and curiosity, qualities AI inherently lacks.
-
•
Users can use ChatGPT effectively as a tool to validate, triangulate, and summarize qualitative data, supplementing human analysis.
-
•
Thematic networks benefit from a visual and iterative human process, facilitating mental models that AI's bullet-point summaries do not provide.
-
•
Prompt engineering and role specification in ChatGPT do not currently resolve fundamental limitations in framework generation.
-
•
AI synthesis results should be viewed as stimuli for human curiosity and interpretation, not as definitive conclusions.
-
•
The most advanced qualitative synthesis goals, such as creating engaging analogies, remain a largely human prerogative due to AI's lack of experiential learning.
Notable Quotes
"ChatGPT has a big advantage in being thorough and comprehensive, especially compared to myself."
"Many of ChatGPT's organizing themes are at the same level as the basic themes, which is confusing and inappropriate."
"ChatGPT's frameworks generate siloed insights but fail to show the connections and flows essential for storytelling."
"Using analogy is very difficult for AI because it requires a unique human ability to connect and engage through storytelling."
"Qualitative synthesis is like watching a movie in a cinema; AI synthesis feels like watching a recap on YouTube—fast but missing the immersive experience."
"AI can’t experience data subjectively or fulfill curiosity, which drives human insight and discovery in qualitative research."
"We should not replace our curiosity and experience with AI but should instead use AI to speed up summarization and then slow down for human sensemaking."
"I tried different prompts and roles, but they didn’t fundamentally improve ChatGPT’s ability to generate meaningful frameworks."
"ChatGPT’s approach in analogy generation segmented the narrative into unrelated metaphors, which confused the overall story."
"AI synthesis should be considered a human-machine mutual learning process, with AI stimulating new perspectives rather than providing conclusions."
Or choose a question:
More Videos
"Our best selves hold and hear stories and hold power to account."
Jemma AhmedResearch at an inflection point: Adapting to a new era of collaboration, equity, and innovation
March 11, 2025
"Silent leaders are people who prefer action over words and advocate in more intimate settings."
Briana ThomasThe Quiet Force: Uncovering Hidden Leadership in High-Impact Design Teams
September 24, 2024
"The assistant deputy minister described this to me as a design problem, even if his peers couldn’t articulate it that way."
Gordon Ross12 Months of COVID-19 Design and Digital Response with the British Columbia Government
December 8, 2021
"Nobody is going to be satisfied by insight-shaped answers or high-level summaries masquerading as breakthroughs."
Laura KleinHuman vs. machine: Testing AI’s ability to synthesize and analyze research
March 11, 2026
"NPS scores can rise overall, yet new and existing customers might have completely different views that get obscured."
Alberto FerreiraMaking it Count: Developing a custom digital metric framework that works
October 15, 2021
"The Rosenbot’s helping you craft a learning program, maybe even a syllabus, complete with weekly goals, lessons, readings, exercises."
Louis RosenfeldHow to use the Rosenbot
March 16, 2026
"If you target the right audience but can’t recruit them, you fail to capture the community’s true problems."
Noah BondRedefining truth and inclusivity: Navigating data ownership and ethical research in the age of disinformation
March 11, 2025
"You don’t want to wait for permission. If you see something valuable, start doing it and demonstrate its worth."
James LangHopeful Futures for UX Research
January 30, 2026
"If we focus on energy management data points instead of just saying sustainable, that wins over more people."
Amy ParnessScaling Sustainability: Complementary strategies that drive long-term success
May 29, 2025
