Summary
Following the emergence of Generative AI as a potential revolution in the UX field, a great deal of AI-driven tools arose to enhance the efficiency of UX research, including data analysis. Qualitative data analysis is a process that conventionally relies on human intelligence to discern patterns, establish connections, and derive actionable insights and frameworks. Many studies have involved comparing the quality of qualitative analyses generated by humans with those produced by AI language models like ChatGPT (Hamilton et al., 2023). Despite the undeniable appeal of automation and speed, there is ongoing debate about AI’s ability to replace human intelligence in qualitative analysis, which may be unlikely at this moment. Then the question is: To what extent can AI contribute to qualitative data analysis? In this case study, I delved into the thematic analysis and post-analysis stage, i.e. synthesizing insights into a framework. Framework, in this context, refers to a conceptual structure that illustrates the components of a human experience and how the components interconnect and operate within the structure. It is a concise model that encapsulates the entirety of research insights. The topic of my case study is "trust relationships between job seekers and hirers in the marketplace,, aligning with the business focus of my company. From my secondary research, I found that, ChatGPT needed multiple rounds of training using diverse prompts to conduct precise and comprehensive thematic analysis. ChatGPT can execute fine-quality thematic analysis under the help of right prompts, yet it falls short in replacing human intelligence for synthesizing insights and crafting frameworks for engaging narratives. Its limitation lies in lacking the depth of contextual understanding within a company, such as understanding what’s missing from the company’s mainstream discourse to create a human-centered story based on data analysis. To craft a framework that conveys good storytelling and organizational impact, it requires the researcher's introspection into knowledge gaps in the specific organizational context. Thus, the best practice is to combine human interpretation and AI production. In my talk, I will demonstrate several principles to guide this practice. Takeaways We’ll cover principles of how to employ ChatGPT in qualitative analysis, specifically focusing on its application in synthesizing and crafting frameworks that convey compelling and insightful narratives: Effectiveness of ChatGPT in thematic analysis: Learn about my process of training ChatGPT to conduct precise thematic analysis. You’ll gain insights into the capabilities and limitations of ChatGPT in providing accurate and comprehensive analysis for framework construction Value of human potential: We’ll address the value of human self-reflection and the ability of interpreting organizational context for crafting engaging frameworks Comparison between human and ChatGPT: By comparing the human-driven outcomes against ChatGPT for qualitative analysis, you’ll see how effective the synthesized frameworks are generated by the researcher and ChatGPT separately. Collaboration between human and ChatGPT: You’ll also learn when and how to incorporate human interpretation with ChatGPT to achieve the best practice in qualitative analysis and synthesis
Key Insights
-
•
ChatGPT can synthesize data faster and more thoroughly than humans.
-
•
Human insights are richer due to personal experience and engagement with data.
-
•
AI struggles with hierarchical structure in thematic analysis, leading to similar levels in basic and organizing themes.
-
•
Humans excel in creating abstract concepts from data, driven by curiosity.
-
•
Using AI can enhance qualitative research but should complement, not replace human intuition.
-
•
AI frameworks often lack interconnectedness and nuance compared to human-generated insights.
-
•
Qualitative synthesis is inherently experiential for humans, unlike AI's rapid summarization.
-
•
AI can validate findings but lacks the depth of human emotional engagement in analysis.
-
•
AI presents results in a clean format, but this may obscure connections between insights.
-
•
Curiosity and exploration drive human synthesis, providing depth and understanding that AI lacks.
Notable Quotes
"It's interesting to follow Dan's presentation because he brought us onto a bird's view, while I'll bring a worm's view."
"The main question is, how good is ChatGPT at synthesizing qualitative data?"
"AI can generate insights faster, but are they better than what we generate ourselves?"
"In some aspects, ChatGPT has a big advantage, especially in thoroughness."
"AI doesn't have an eye for experience; it presents results without allowing us to experience the data."
"Qualitative synthesis is not just straightforward; it's an intricate process."
"The difference between qualitative synthesis and AI summarization is like watching a movie recap versus experiencing the movie itself."
"If we rely solely on generative AI for thematic analysis, we'll struggle with its results."
"Curiosity is a critical driver in fulfilling our cognitive needs during qualitative synthesis."
"AI can help review and triangulate, but should not replace the insights derived from human experience."
More Videos
"Leadership is best when people barely know she exists."
John MaedaMaking Sense of Enterprise UX
June 9, 2016
"There are a lot of variability in how people approach their intent or purpose."
Indi YoungThinking styles: Mend hidden cracks in your market
January 8, 2025
"Data should be presented in ways that empower users to make immediate decisions."
Chris ChapoData Science and Design: A Tale of Two Tribes
May 13, 2015
"We observed how cultural practices impact technology uptake."
Elizabeth ChurchillExploring Cadence: You, Your Team, and Your Enterprise
June 8, 2017
"We faced challenges as individuals like human connections and networking."
Rusha SopariwalaRemote, Together: Craft and Collaboration Across Disciplines, Borders, Time Zones, and a Design Org of 170+
June 9, 2022
"Power is the ability to change another person's reality."
George AyeThat Quiet Little Voice: When Design and Ethics Collide
November 16, 2022
"I care about my team's resilience, and I'm here to share some strategies and tactics that work well for us."
Ariba JahanTeam Resiliency Through a Pandemic
January 8, 2024
"Burnout is real; recognizing when people fade is key to managing it."
Julie Gitlin Esther RaiceDesign as an Agent of Digital Transformation at JPMC
June 9, 2021
"We are not allowing anyone else to be in a position of power based on the data we collect and share."
Matteo GrattonCan Data and Ethics Live Together?
October 1, 2021
