Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

AI in Real Life: Using LLMs to Turbocharge Microsoft Learn
Thursday, February 13, 2025 • Rosenfeld Community
Share the love for this talk
AI in Real Life: Using LLMs to Turbocharge Microsoft Learn
Speakers: Sarah Barrett
Link:

Summary

Enthusiasm for AI tools, especially large language models like ChatGPT, is everywhere, but what does it actually look like to deliver large-scale user-facing experiences using these tools in a production environment? Clearly they're powerful, but what do they need to make them work reliably and at scale? In this session, Sarah provides a perspective on some of the information architecture and user experience infrastructure organizations need to effectively leverage AI. She also shares three AI experiences currently live on Microsoft Learn: An interactive assistant that helps users post high-quality questions to a community forum A tool that dynamically creates learning plans based on goals the user shares A training assistant that clarifies, defines, and guides learners while they study Through lessons learned from shipping these experiences over the last two years, UXers, IAs, and PMs will come away with a better sense of what they might need to make these hyped-up technologies work in real life.

Key Insights

  • Most AI applications no longer require building foundation models from scratch; the focus is now on application development and integration.

  • Single, all-purpose chatbots (everything chatbots) are insufficient because they handle high ambiguity and diverse, often complex tasks poorly.

  • Sarah introduces the ambiguity footprint as a framework to measure AI application complexity and risks across several axes such as task complexity, context, interface, prompt openness, and sensitivity.

  • AI features that support simple, complimentary user tasks, rather than critical or complex ones, are easier and safer to build and scale.

  • Visible AI interfaces, like chatbots, set clearer user expectations but introduce more ambiguity and management overhead compared to invisible AI (e.g., keyboard optimizations).

  • Prompt engineering plays a crucial role in defining the boundaries of AI output, from very open-ended to highly restricted scopes.

  • Retrieval Augmented Generation (RAG) helps manage up-to-date context by dynamically querying relevant data chunks rather than using static corpus.

  • Evaluating AI outputs rigorously is essential but often underprioritized; without clear quality metrics, teams end up relying on subjective or anecdotal assessments.

  • Data ethics and distributed AI implementations can create blind spots, limiting feedback loops necessary for continuous AI model improvement.

  • Incrementally building AI applications with smaller ambiguity footprints helps organizations develop expertise and controls before tackling more complex, open-ended AI products.

Notable Quotes

"You’re not doing IA, but you’re always doing it."

"An everything chat bot is almost certainly not how you’re going to build it; realistically you’re building three apps in a trench coat."

"AI is ambiguous at best because we’re fully in the realm of probabilistic rather than deterministic programming."

"The more complex the task, the less likely it is to be successful with current AI."

"A task where AI adds a little something is honestly easier to get right than one where it’s absolutely critical."

"Visible AI interfaces introduce another place where you can add ambiguity."

"Retrieval Augmented Generation lets you supply specific relevant information to the model dynamically rather than everything at once."

"Evaluation might be the most important part of your entire development effort and is often the hardest to do well."

"You can’t just eyeball results and call it good; AI applications are expensive and complex and require systematic evaluation."

"Never build or buy an everything chat bot again; start with less ambiguous, targeted AI experiences."

Ask the Rosenbot
Chris Hodowanec
Agile + User Experience: How to navigate the Agile landscape as an UX Practitioner
2022 • Civic Design 2022
Gold
Karen Pascoe
Developing Experience Teams and Talent in the Enterprise
2016 • Enterprise UX 2016
Gold
Renee Reid
Becoming a ResearchH.E.R (Highly Enterprise Ready)
2019 • Enterprise Experience 2019
Gold
Ned Dwyer
The Future of DesignOps is Tool Consolidation
2024 • DesignOps Summit 2024
Gold
Peter Merholz
The Trials and Tribulations of Directors of UX
2023 • Enterprise Community
Bria Alexander
Opening Remarks Day 2
2024 • Advancing Research 2024
Gold
Katy Mogal
But Do Your Insights Scale?
2021 • Advancing Research 2021
Gold
Louis Rosenfeld
How to use the Rosenbot
2026 • Rosenfeld Community
Steve Chaparro
Bringing Into Alignment Brand, Culture and Space
2020 • DesignOps Community
Gabriela Barneva
Operationalizing Inclusive Design in Service Design
2025 • Advancing Service Design 2025
Gold
Richard Buchanan
Creativity and Principles in the Flourishing Enterprise
2018 • Enterprise Experience 2018
Gold
Andy Barraclough
From Costly Complexity to Efficient Insights: Why UX Teams Are Switching To Voxpopme
2024 • DesignOps Summit 2024
Gold
Peter Van Dijck
Building impactful AI products for design and product leaders, Part 1: The new product journey
2025 • Rosenfeld Community
Robin Beers
Navigating organizational systems: Rethinking researcher’s role in driving change
2025 • Advancing Research 2025
Gold
Jen van der Meer
Service design performs value
2025 • Advancing Service Design 2025
Gold
Sarah Fathallah
Beyond insights: Rethinking the role of researchers as stewards of organizational wisdom
2025 • Advancing Research 2025
Gold

More Videos

Charles Lee

"We kept feeling the same consistent pain around some truths for our design system."

Charles Lee Jennie Yip

Building a New Home for the Atlassian Design System

October 22, 2020

Gina Mendolia

"Increasing coordinated collaboration increased our service capacity by 200%."

Gina Mendolia Jasmine Toy

Coordinated collaboration: a Service Design & DesignOps love story

November 19, 2025

Robin Beers

"You can do everything right, but timing and organizational readiness are beyond your control."

Robin Beers Nalini Kotamraju Andy Warr

Panel: Excellence in Communicating Insights

March 26, 2024

Cheryl Platz

"Instead of rewarding owners of ideas, reward collaboration."

Cheryl Platz

Collaborative Creativity through Improv

November 7, 2018

Laura Klein

"The incentive structure at a big company is set up to reward people for entirely different stuff than innovation requires."

Laura Klein

Unique challenges of innovation in enterprises

April 23, 2020

Kevin Bethune

"We invited BCG’s clients to live with us in multidisciplinary innovation studios for 12 to 18 months."

Kevin Bethune

Reimagining Design: Unlocking Strategic Innovation

June 8, 2022

Daniel Gloyd

"Patricia Moore’s aging research revealed how product design often excludes elderly users by living their constraints."

Daniel Gloyd

Designing From the Inside Out: How Method Acting Can Inspire Design Research

February 12, 2026

Dr Chloe Sharp

"Only 33% of businesses conduct research throughout the whole product lifecycle to inform product and business decisions."

Dr Chloe Sharp

Using Evidence and Collaboration for Setting and Defending Priorities

November 29, 2023

Charlotte Vorbeck

"Without equity, you can only have design theater because it’s not really design if it’s not including everybody."

Charlotte Vorbeck Shahzma Esmail Edward Alton Sarah McArthur Ariel Kennan

Pipeline to Civic Design

December 9, 2021