Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Thursday, April 4, 2024 • Rosenfeld Community
Share the love for this talk
Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Speakers: Cheryl Platz
Link:

Summary

Did you know that you're probably designing multimodal experiences? Most designers today are working in a multimodal environment, but haven't been trained to make the most of the many capabilities the latest generations of devices provide. Your customers have a small universe of devices, and they now come with the ability to handle far more than traditional haptic input like keyboards and mice. Gestural input, from swipes to hand gestures in video calls to stylus input, is becoming more common. Audio input and output are becoming more important in a world where digital assistants are poised to make a second surge powered by large language model AI. Visual output has become much more nuanced, and sometimes spans multiple devices. How do you wrangle all of this, optimize for great experiences, and still keep the human at the center? By becoming more consciously aware of the different inputs and outputs we're working with - and the many ways these inputs and outputs include and exclude our customers - we can build more resilient, more inclusive, and more powerful next-generation experiences.

Key Insights

  • Most designers already practice multimodal design unintentionally by supporting multiple input and output modalities like voice, touch, and haptics.

  • Multimodal design requires orchestrating how different modalities and devices communicate and transition smoothly, rather than designing in isolation.

  • Users switch devices when modality limits push them to—for example, moving from phone to desktop due to input convenience.

  • Cheryl’s CROW framework (Character, Relationship, Objective, Where) provides a comprehensive lens to capture user context relevant for multimodal design.

  • Transitions between input and output modes, devices, and network connectivity represent critical failure points and require intentional design.

  • Proactive, context-aware communication and interruption management improve user experience during multimodal and cross-device interactions.

  • Multimodal design and service design are deeply intertwined, especially across multiple devices and touchpoints.

  • Adaptive experiences that allow users to choose modalities based on context, such as hands-free vs. touch, lead to more inclusive designs.

  • Physical remotes remain relevant because phones are not always accessible, have battery constraints, and because remotes offer lower distraction.

  • Empathy for stakeholders’ goals and framing multimodal design benefits in terms of business priorities helps gain buy-in.

Notable Quotes

"Multimodal design is the design practice you probably don’t realize you’re doing already."

"The future is multimodal because humans are multimodal."

"Customers move from one device to another often because the information density or proximity to the device changes."

"Character, Relationship, Objective, and Where gives us a storytelling shorthand for fully understanding user context."

"Transitions will make or break your cross-device multimodal experiences."

"We don’t want to list every single customer activity; we need to abstract behavior patterns to manage interruptions."

"If you are altering any form of self-expression, you need to be really careful."

"Multimodal design adds a layer of orchestration on top of modality-specific designs like voice or graphical UI."

"Physical remotes have a lower power draw and are often more purpose-driven than phones, which are sources of stress."

"We treat our stakeholders as customers to understand their scenarios and align on a shared north star principle."

Josh Clark
Sentient Scenes and Radically Adaptive Experiences
2025 • Designing with AI 2025
Gold
Erin Hauber
Design is Not the Frosting on the Scaled Agile Layer Cake
2019 • DesignOps Summit 2019
Gold
Melinda Belcher
Bridging the Gap: Making the Most of the Differences Between Agency and Enterprise
2024 • Enterprise Experience 2020
Gold
Peter Van Dijck
Building the Rosenbot
2024 • Designing with AI 2024
Gold
Jilanna Wilson
Distributed Design Operations Management
2019 • DesignOps Summit 2019
Gold
Jen Crim
Culture, DIBS & Recruiting
2021 • Design at Scale 2021
Gold
Josh Clark
Sentient Design, AI, and the Radically Adaptive Experience (1st of 3 seminars)
2025 • Rosenfeld Community
Dave Hoffer
UX Job Search AMA #2 with Joanne Weaver and Dave Hoffer
2025 • Rosenfeld Community
Jemma Ahmed
Theme 2 Intro
2024 • Advancing Research 2024
Gold
Elizabeth Churchill
Exploring Cadence: You, Your Team, and Your Enterprise
2017 • Enterprise Experience 2017
Gold
Dr. Jamika D. Burge
The Future of Research: Bridging the Gaps
2021 • Advancing Research Community
Mila Kuznetsova
How Lessons Learned from Our Youngest Users Can Help Us Evolve our Practices
2022 • Advancing Research 2022
Gold
Lisa Welchman
Cleaning Up Our Mess: Digital Governance for Designers
2018 • Enterprise Experience 2018
Gold
Alexia Cohen
Increasing Health Equity and Improving the Service Experience for Under-Served Latine Communities in Arizona
2024 • Advancing Service Design 2024
Gold
Ovetta Sampson
Turning UX Passion into Real Product Influence
2023 • Enterprise UX 2023
Gold
Rusha Sopariwala
Remote, Together: Craft and Collaboration Across Disciplines, Borders, Time Zones, and a Design Org of 170+
2022 • Design at Scale 2022
Gold

More Videos

Randolph Duke II

"Carol was really grateful for our conversation."

Randolph Duke II

War Stories LIVE! Randy Duke II

March 30, 2020

Corey Nelson

"If you were blindsided recently, I’ve been there. I just really appreciated a kind voice because I didn’t hear a single word they said."

Corey Nelson Amy Santee

Layoffs

November 15, 2022

Landon Barnes

"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."

Landon Barnes

Are My Research Findings Actually Meaningful?

March 10, 2022

Amy Bucher

"Most of my career has been in health, and it’s really important that we don’t just get people to do something once, but sustain a new behavior."

Amy Bucher

Harnessing behavioral science to uncover deeper truths

March 12, 2025

David Sternberg

"Every piece of product can be described using components like intent, viscosity, and flow—making UX a physics problem."

David Sternberg

Uncovering the hidden forces shaping user behavior

July 17, 2025

Deanna Smith

"Anecdotal feedback from team members helps leaders understand the real impact of change."

Deanna Smith

Leading Change with Confidence: Strategies for Optimizing Your Process

September 23, 2024

Jennifer Strickland

"Designers have power. We get to decide who gets heard, who gets included, who gets excluded."

Jennifer Strickland

Adopting a "Design By" Method

December 9, 2021

Rachel Posman

"Our team Ops Charter is about amplifying and celebrating the work and impact of design and design Ops."

Rachel Posman John Calhoun

A Closer Look at Team Ops and Product Ops (Two Sides of the DesignOps Coin)

November 19, 2020

Gina Mendolia

"Creating and holding space is like putting bumpers up in bowling so people know they won’t fail if they engage."

Gina Mendolia

Therapists, Coaches, and Grandmas: Techniques for Service Design in Complex Systems

December 3, 2024