Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Thursday, April 4, 2024 • Rosenfeld Community
Share the love for this talk
Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Speakers: Cheryl Platz
Link:

Summary

Did you know that you're probably designing multimodal experiences? Most designers today are working in a multimodal environment, but haven't been trained to make the most of the many capabilities the latest generations of devices provide. Your customers have a small universe of devices, and they now come with the ability to handle far more than traditional haptic input like keyboards and mice. Gestural input, from swipes to hand gestures in video calls to stylus input, is becoming more common. Audio input and output are becoming more important in a world where digital assistants are poised to make a second surge powered by large language model AI. Visual output has become much more nuanced, and sometimes spans multiple devices. How do you wrangle all of this, optimize for great experiences, and still keep the human at the center? By becoming more consciously aware of the different inputs and outputs we're working with - and the many ways these inputs and outputs include and exclude our customers - we can build more resilient, more inclusive, and more powerful next-generation experiences.

Key Insights

  • Most designers already practice multimodal design unintentionally by supporting multiple input and output modalities like voice, touch, and haptics.

  • Multimodal design requires orchestrating how different modalities and devices communicate and transition smoothly, rather than designing in isolation.

  • Users switch devices when modality limits push them to—for example, moving from phone to desktop due to input convenience.

  • Cheryl’s CROW framework (Character, Relationship, Objective, Where) provides a comprehensive lens to capture user context relevant for multimodal design.

  • Transitions between input and output modes, devices, and network connectivity represent critical failure points and require intentional design.

  • Proactive, context-aware communication and interruption management improve user experience during multimodal and cross-device interactions.

  • Multimodal design and service design are deeply intertwined, especially across multiple devices and touchpoints.

  • Adaptive experiences that allow users to choose modalities based on context, such as hands-free vs. touch, lead to more inclusive designs.

  • Physical remotes remain relevant because phones are not always accessible, have battery constraints, and because remotes offer lower distraction.

  • Empathy for stakeholders’ goals and framing multimodal design benefits in terms of business priorities helps gain buy-in.

Notable Quotes

"Multimodal design is the design practice you probably don’t realize you’re doing already."

"The future is multimodal because humans are multimodal."

"Customers move from one device to another often because the information density or proximity to the device changes."

"Character, Relationship, Objective, and Where gives us a storytelling shorthand for fully understanding user context."

"Transitions will make or break your cross-device multimodal experiences."

"We don’t want to list every single customer activity; we need to abstract behavior patterns to manage interruptions."

"If you are altering any form of self-expression, you need to be really careful."

"Multimodal design adds a layer of orchestration on top of modality-specific designs like voice or graphical UI."

"Physical remotes have a lower power draw and are often more purpose-driven than phones, which are sources of stress."

"We treat our stakeholders as customers to understand their scenarios and align on a shared north star principle."

Ask the Rosenbot
Dem Gerolemou
Climate technology fundamentals
2024 • Climate UX Interest Group
Ian Johnson
Latent Scope: Finding structure in unstructured data
2025 • Designing with AI 2025
Gold
Peter Morville
The Architecture of Understanding
2015 • Enterprise UX 2015
Gold
Raven Veal
Dark Metrics: Illuminating the Negative Impact of Digital Health Design
2021 • Advancing Research 2021
Gold
Christian Bason
Expand—Rethinking Design for Public Challenges
2022 • Civic Design Community
Erin Hoffman-John
This Game is Never Done: Design Leadership Techniques from the Video Game World
2017 • DesignOps Summit 2017
Gold
Dan Hill
Strategic design, slowdown, and the infrastructures of everyday life
2022 • Enterprise Community
Jon Fukuda
Theme One Intro
2022 • DesignOps Summit 2022
Gold
Katie Hansen
Experimental research: techniques for deep, psychology-driven insights
2025 • Advancing Research 2025
Gold
Séamus Byrne
Aligning Teams with Choreography
2024 • Enterprise Experience 2020
Gold
Jake Burghardt
Stop wasting research: Create new value with insight summaries
2025 • Rosenfeld Community
Peter Van Dijck
Hands-on AI #2: Understanding evals: LLM as a Judge
2025 • Rosenfeld Community
Jaime Creixems
Best Practices when Creating and Maintaining a Design System
2023 • Enterprise UX 2023
Gold
Rachel Posman
"Ask Me Anything" with Rachel Posman and John Calhoun, Authors of the Upcoming Rosenfeld Book, The Design Conductors
2024 • DesignOps Summit 2024
Gold
Noah Bond
Redefining truth and inclusivity: Navigating data ownership and ethical research in the age of disinformation
2025 • Advancing Research 2025
Gold
Aletheia Delivre
New Shapes and Emerging Identities for Design Ops
2025 • DesignOps Summit 2025
Conference

More Videos

Miles Orkin

"You can’t separate emotion from work; telling people not to get feelings involved is like saying don’t use your left leg."

Miles Orkin

Creativity and Culture

November 8, 2018

Mike Brzozowski

"Most people care about climate and are willing to make lifestyle changes, but they don't know how to get involved."

Mike Brzozowski Laura Palotie Steve Isley Nancy Tsang

UX in everyday products: Empowering climate conscious choices

July 17, 2024

Iain McMaster

"Leadership needs to give specific directions to open up space for collaboration and inclusion."

Iain McMaster IHan Cheng

Design and Product: from Frenemy to Harmony

November 29, 2023

JP Allen

"Tools are getting much more complex and have more crossover, which made redesigning the map a challenge."

JP Allen Holly Holden

Navigating the UX Tools Landscape

October 1, 2021

Alla Weinberg

"Stress drops our operating IQ by half, making it impossible to collaborate or problem-solve effectively."

Alla Weinberg

Workers Are Sick of Change: The Cure is Psychological Safety

June 6, 2023

Dan Willis

"Shamus Bern will talk about how to drop into a client site and grok what’s going on quickly without sacrificing why they hired you."

Dan Willis

Theme 3: Intro

January 8, 2024

Dean Broadley

"Crabs grow by shedding their shells, which is uncomfortable but necessary."

Dean Broadley

Not Black Enough to be White

January 8, 2024

Nalini P. Kotamraju

"We built living components as conversation starters, showing instead of telling."

Nalini P. Kotamraju

An Organizational Story: Salesforce Lightning Design System

June 9, 2016

Janelle Estes

"Centralized teams can become 'no machines' when overwhelmed with requests, needing to prioritize carefully."

Janelle Estes

UX Research Trends

January 28, 2021