Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

AI in Real Life: Using LLMs to Turbocharge Microsoft Learn
Thursday, February 13, 2025 • Rosenfeld Community
Share the love for this talk
AI in Real Life: Using LLMs to Turbocharge Microsoft Learn
Speakers: Sarah Barrett
Link:

Summary

Enthusiasm for AI tools, especially large language models like ChatGPT, is everywhere, but what does it actually look like to deliver large-scale user-facing experiences using these tools in a production environment? Clearly they're powerful, but what do they need to make them work reliably and at scale? In this session, Sarah provides a perspective on some of the information architecture and user experience infrastructure organizations need to effectively leverage AI. She also shares three AI experiences currently live on Microsoft Learn: An interactive assistant that helps users post high-quality questions to a community forum A tool that dynamically creates learning plans based on goals the user shares A training assistant that clarifies, defines, and guides learners while they study Through lessons learned from shipping these experiences over the last two years, UXers, IAs, and PMs will come away with a better sense of what they might need to make these hyped-up technologies work in real life.

Key Insights

  • Most AI applications no longer require building foundation models from scratch; the focus is now on application development and integration.

  • Single, all-purpose chatbots (everything chatbots) are insufficient because they handle high ambiguity and diverse, often complex tasks poorly.

  • Sarah introduces the ambiguity footprint as a framework to measure AI application complexity and risks across several axes such as task complexity, context, interface, prompt openness, and sensitivity.

  • AI features that support simple, complimentary user tasks, rather than critical or complex ones, are easier and safer to build and scale.

  • Visible AI interfaces, like chatbots, set clearer user expectations but introduce more ambiguity and management overhead compared to invisible AI (e.g., keyboard optimizations).

  • Prompt engineering plays a crucial role in defining the boundaries of AI output, from very open-ended to highly restricted scopes.

  • Retrieval Augmented Generation (RAG) helps manage up-to-date context by dynamically querying relevant data chunks rather than using static corpus.

  • Evaluating AI outputs rigorously is essential but often underprioritized; without clear quality metrics, teams end up relying on subjective or anecdotal assessments.

  • Data ethics and distributed AI implementations can create blind spots, limiting feedback loops necessary for continuous AI model improvement.

  • Incrementally building AI applications with smaller ambiguity footprints helps organizations develop expertise and controls before tackling more complex, open-ended AI products.

Notable Quotes

"You’re not doing IA, but you’re always doing it."

"An everything chat bot is almost certainly not how you’re going to build it; realistically you’re building three apps in a trench coat."

"AI is ambiguous at best because we’re fully in the realm of probabilistic rather than deterministic programming."

"The more complex the task, the less likely it is to be successful with current AI."

"A task where AI adds a little something is honestly easier to get right than one where it’s absolutely critical."

"Visible AI interfaces introduce another place where you can add ambiguity."

"Retrieval Augmented Generation lets you supply specific relevant information to the model dynamically rather than everything at once."

"Evaluation might be the most important part of your entire development effort and is often the hardest to do well."

"You can’t just eyeball results and call it good; AI applications are expensive and complex and require systematic evaluation."

"Never build or buy an everything chat bot again; start with less ambiguous, targeted AI experiences."

Nick Cochran
Growing in Enterprise Design through Making Connections
2019 • Enterprise Community
JP Allen
Navigating the UX Tools Landscape
2021 • DesignOps Summit 2021
Gold
Dr. Jamika D. Burge
Advancing the Inclusion of Womxn in Research Practices
2022 • Advancing Research Community
Alla Weinberg
Workers Are Sick of Change: The Cure is Psychological Safety
2023 • Enterprise UX 2023
Gold
Miles Orkin
Creativity and Culture
2018 • DesignOps Summit 2018
Gold
Sam Proulx
Prototype Reviews, People With Disabilities, and You
2021 • DesignOps Summit 2021
Gold
Samuel Proulx
Designing beyond caricatures: Embracing real, diverse user needs
2024 • Advancing Service Design 2024
Gold
Lukas Moro
“Feels Like Paper!”: Interfacing AI through Paper
2025 • Designing with AI 2025
Gold
Laura Klein
Unique challenges of innovation in enterprises
2020 • Enterprise Community
Brad Peters
Short Take #1: UX/Product Lessons from Your Industry Peers
2022 • Design in Product 2022
Gold
Bram Wessel
Enterprise Information Architecture
2020 • Enterprise Community
Kritika Yadav
Optimizing AI Conversations: A Case Study on Personalized Shopping Assistance Frameworks
2025 • Designing with AI 2025
Gold
Kavana Ramesh
Meaningful inclusion: Practicing accessibility research with confidence
2024 • DesignOps Summit 2024
Gold
Sam Proulx
Mobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World
2022 • Civic Design 2022
Gold
Christian Crumlish
AMA with Christian Crumlish, author of Product Management for UX People
2022 • Enterprise Community
Kaitlin Tasker
Fast and Fearless Inclusive Research
2023 • Advancing Research 2023
Gold

More Videos

John Mortimer

"The design is how you show up in response to probing the system day in, day out."

John Mortimer Milan Guenther Lucy Ellis Patrick Quattlebaum

Panel Discussion

December 3, 2024

Sean Dolan

"Most B2B research is done remotely and online, so we needed a tool that was easy to use without in-person facilitation."

Sean Dolan

A Practical Look at Creating More Usable Enterprise Customer Journeys

October 31, 2019

Catt Small

"Levels can mean really different things at different companies, so ask a lot of questions about expectations for each level."

Catt Small Micah Bennett Brian Carr Jessica Harllee

What's Next for ICs: Exploring Staff and Principal Designer Roles

February 22, 2024

Greg Petroff

"Mobile-first is critical because real estate is the largest mobile profession without a full app for business."

Greg Petroff

Design is the Differentiator: Bringing New Design Innovations to a Very Antiquated and Very Large Industry

June 9, 2021

Andy Warr

"We have all these different people throwing insights over the wall and it’s great, but it’s too much."

Andy Warr

Under My (Research) Umbrella: The Benefits and Challenges of Building a Unified Insights Function

March 25, 2024

Alexia Cohen

"Federal funding does not provide food for participants, so we had to plan accordingly to make participation irresistible."

Alexia Cohen Adriane Ackerman

Increasing Health Equity and Improving the Service Experience for Under-Served Latine Communities in Arizona

December 4, 2024

Kevin Bethune

"Leadership sometimes requires going against the status quo to create bandwidth for cross-functional experimentation."

Kevin Bethune

Gatekeepers and Servant Leadership

January 30, 2020

Laura Weiss

"The presence of conflict is often an indication that something new is trying to emerge."

Laura Weiss

Turn Down the Heat: 3 Ways to Handle Conflict in the Moment

November 20, 2024

Sheryl Cababa

"If you’re working at scale, many problems that seem like 'clock' problems actually have 'cloud' elements."

Sheryl Cababa

Living in the Clouds: Adopting a Systems Thinking Mindset

June 6, 2023