Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

AI in Real Life: Using LLMs to Turbocharge Microsoft Learn
Thursday, February 13, 2025 • Rosenfeld Community
Share the love for this talk
AI in Real Life: Using LLMs to Turbocharge Microsoft Learn
Speakers: Sarah Barrett
Link:

Summary

Enthusiasm for AI tools, especially large language models like ChatGPT, is everywhere, but what does it actually look like to deliver large-scale user-facing experiences using these tools in a production environment? Clearly they're powerful, but what do they need to make them work reliably and at scale? In this session, Sarah provides a perspective on some of the information architecture and user experience infrastructure organizations need to effectively leverage AI. She also shares three AI experiences currently live on Microsoft Learn: An interactive assistant that helps users post high-quality questions to a community forum A tool that dynamically creates learning plans based on goals the user shares A training assistant that clarifies, defines, and guides learners while they study Through lessons learned from shipping these experiences over the last two years, UXers, IAs, and PMs will come away with a better sense of what they might need to make these hyped-up technologies work in real life.

Key Insights

  • Most AI applications no longer require building foundation models from scratch; the focus is now on application development and integration.

  • Single, all-purpose chatbots (everything chatbots) are insufficient because they handle high ambiguity and diverse, often complex tasks poorly.

  • Sarah introduces the ambiguity footprint as a framework to measure AI application complexity and risks across several axes such as task complexity, context, interface, prompt openness, and sensitivity.

  • AI features that support simple, complimentary user tasks, rather than critical or complex ones, are easier and safer to build and scale.

  • Visible AI interfaces, like chatbots, set clearer user expectations but introduce more ambiguity and management overhead compared to invisible AI (e.g., keyboard optimizations).

  • Prompt engineering plays a crucial role in defining the boundaries of AI output, from very open-ended to highly restricted scopes.

  • Retrieval Augmented Generation (RAG) helps manage up-to-date context by dynamically querying relevant data chunks rather than using static corpus.

  • Evaluating AI outputs rigorously is essential but often underprioritized; without clear quality metrics, teams end up relying on subjective or anecdotal assessments.

  • Data ethics and distributed AI implementations can create blind spots, limiting feedback loops necessary for continuous AI model improvement.

  • Incrementally building AI applications with smaller ambiguity footprints helps organizations develop expertise and controls before tackling more complex, open-ended AI products.

Notable Quotes

"You’re not doing IA, but you’re always doing it."

"An everything chat bot is almost certainly not how you’re going to build it; realistically you’re building three apps in a trench coat."

"AI is ambiguous at best because we’re fully in the realm of probabilistic rather than deterministic programming."

"The more complex the task, the less likely it is to be successful with current AI."

"A task where AI adds a little something is honestly easier to get right than one where it’s absolutely critical."

"Visible AI interfaces introduce another place where you can add ambiguity."

"Retrieval Augmented Generation lets you supply specific relevant information to the model dynamically rather than everything at once."

"Evaluation might be the most important part of your entire development effort and is often the hardest to do well."

"You can’t just eyeball results and call it good; AI applications are expensive and complex and require systematic evaluation."

"Never build or buy an everything chat bot again; start with less ambiguous, targeted AI experiences."

Ask the Rosenbot
Kristin Skinner
Theme 2: Introduction and Provocation
2024 • DesignOps Summit 2020
Gold
Laura Klein
Human vs. machine: Testing AI’s ability to synthesize and analyze research
2026 • Advancing Research 2026
Conference
Sarah Alvarado
How to make UX research leadership more effective [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
Christian Crumlish
Morning Insights Panel
2022 • Design in Product 2022
Gold
Andrew Custage
The Digital Journey: Research on Consumer Frustration and Loyalty
2023 • Advancing Research 2023
Gold
Sean McKay
Whole Product Thinking: Expanding beyond problem and solution space thinking
2024 • Rosenfeld Community
Ali Jeffery
How DesignOps Helped Enable Wall Street to Work Remotely
2020 • DesignOps Summit 2020
Gold
Mackenzie Cockram
Integrating Qualitative and Quantitative Research from Discovery to Live
2022 • QuantQual Interest Group
Louis Rosenfeld
Welcome / Housekeeping
2023 • Enterprise UX 2023
Gold
Christian Crumlish
The Pygmalion Effect: In Which a Vibe Coding Experiment Becomes a Million Lines…
2025 • Rosenfeld Community
Rittika Basu
Age and Interfaces: Equipping Older Adults with Technological Tools
2023 • Advancing Research Community
Victor Udoewa
Beyond Methods and Diversity: The Roots of Inclusion
2024 • Advancing Research 2024
Gold
Sharbani Dhar
Breathing Room for Delight
2024 • Enterprise Experience 2020
Gold
Sheryl Cababa
Expanding Your Design Lens with Systems Thinking
2023 • Enterprise Community
Catherine Blizzard
Using Integrated Insight to Drive Growth
2022 • Advancing Research 2022
Gold
Lija Hogan
Three Years Out: Perspectives on the Near-Term Future of User Research
2024 • Advancing Research Community

More Videos

Richard Buchanan

"Sustainability is dead; what we really need is flourishing – flourishing communities and organizations."

Richard Buchanan

Creativity and Principles in the Flourishing Enterprise

June 15, 2018

Crystal Philcox

"Sometimes government rules grow from a single mistake that leads to congressional hearings and new regulations."

Crystal Philcox

The Many Faces of Operations

November 6, 2017

Jessamyn Edwards

"You are a contractor for your team. They want to know when the thing is done and that you delivered thoroughly."

Jessamyn Edwards

Surviving Your UX Career in Enterprise Design

December 2, 2021

Erika Kincaid

"A great designer doesn’t have all the right answers, they have the right questions."

Erika Kincaid Brenna Heaps Jessica Tsukimura

Connecting the Dots: How to Foster Collaboration and Build a Strong Design Review Culture

June 8, 2022

Meaghan Waters

"Allocate about 30% of your analysis and design effort to change management for successful adoption."

Meaghan Waters Fotina Koutropoulous

Lack of Product Thinking will Doom Your Legacy Modernization

June 9, 2021

Dan Donald

"If you don’t continually check whether people are benefiting, the system’s reputation can quickly suffer."

Dan Donald

Design Systems as a Vehicle for Systemic Change

June 1, 2023

Michael Land

"There are lots of little mushroom patches of design sprouting up, but they're disconnected and not sharing."

Michael Land

Establishing Design Operations in Government

February 18, 2021

Ron Bronson

"Public services in the US are harder to design because everything has to have a cost and be paid for."

Ron Bronson

Design, Consequences & Everyday Life

November 18, 2022

Louis Rosenfeld

"There is a code of conduct in place and it’s not just a document but a process to help the community."

Louis Rosenfeld

Founder’s Welcome

December 6, 2022