Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Thursday, April 4, 2024 • Rosenfeld Community
Share the love for this talk
Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Speakers: Cheryl Platz
Link:

Summary

Did you know that you're probably designing multimodal experiences? Most designers today are working in a multimodal environment, but haven't been trained to make the most of the many capabilities the latest generations of devices provide. Your customers have a small universe of devices, and they now come with the ability to handle far more than traditional haptic input like keyboards and mice. Gestural input, from swipes to hand gestures in video calls to stylus input, is becoming more common. Audio input and output are becoming more important in a world where digital assistants are poised to make a second surge powered by large language model AI. Visual output has become much more nuanced, and sometimes spans multiple devices. How do you wrangle all of this, optimize for great experiences, and still keep the human at the center? By becoming more consciously aware of the different inputs and outputs we're working with - and the many ways these inputs and outputs include and exclude our customers - we can build more resilient, more inclusive, and more powerful next-generation experiences.

Key Insights

  • Most designers already practice multimodal design unintentionally by supporting multiple input and output modalities like voice, touch, and haptics.

  • Multimodal design requires orchestrating how different modalities and devices communicate and transition smoothly, rather than designing in isolation.

  • Users switch devices when modality limits push them to—for example, moving from phone to desktop due to input convenience.

  • Cheryl’s CROW framework (Character, Relationship, Objective, Where) provides a comprehensive lens to capture user context relevant for multimodal design.

  • Transitions between input and output modes, devices, and network connectivity represent critical failure points and require intentional design.

  • Proactive, context-aware communication and interruption management improve user experience during multimodal and cross-device interactions.

  • Multimodal design and service design are deeply intertwined, especially across multiple devices and touchpoints.

  • Adaptive experiences that allow users to choose modalities based on context, such as hands-free vs. touch, lead to more inclusive designs.

  • Physical remotes remain relevant because phones are not always accessible, have battery constraints, and because remotes offer lower distraction.

  • Empathy for stakeholders’ goals and framing multimodal design benefits in terms of business priorities helps gain buy-in.

Notable Quotes

"Multimodal design is the design practice you probably don’t realize you’re doing already."

"The future is multimodal because humans are multimodal."

"Customers move from one device to another often because the information density or proximity to the device changes."

"Character, Relationship, Objective, and Where gives us a storytelling shorthand for fully understanding user context."

"Transitions will make or break your cross-device multimodal experiences."

"We don’t want to list every single customer activity; we need to abstract behavior patterns to manage interruptions."

"If you are altering any form of self-expression, you need to be really careful."

"Multimodal design adds a layer of orchestration on top of modality-specific designs like voice or graphical UI."

"Physical remotes have a lower power draw and are often more purpose-driven than phones, which are sources of stress."

"We treat our stakeholders as customers to understand their scenarios and align on a shared north star principle."

Ask the Rosenbot
Laura Klein
Human vs. machine: Testing AI’s ability to synthesize and analyze research
2026 • Advancing Research 2026
Conference
Helen Armstrong
Augment the Human. Interrogate the System.
2023 • Enterprise UX 2023
Gold
Pippa Lomas
Paving the Path for Neurodiversity in Design
2023 • DesignOps Summit 2023
Gold
Kim Holt
A Salesforce Panel Discussion on Values-Driven DesignOps
2022 • DesignOps Summit 2022
Gold
Kate Kalcevich
Designing inclusively with AI
2024 • Designing with AI 2024
Gold
Alana Washington
Theme 1: Introduction and Provocation
2024 • DesignOps Summit 2020
Gold
Sam Proulx
To Boldly Go: The New Frontiers of Accessibility
2022 • Design at Scale 2022
Gold
Roberta Dombrowski
5 Reasons to Bring your Recruiting in House
2021 • DesignOps Summit 2021
Gold
Devon Powers
Imagining Better Futures
2022 • Advancing Research 2022
Gold
Jaime Creixems
Best Practices when Creating and Maintaining a Design System
2023 • Enterprise UX 2023
Gold
Josina Vink
Navigating the pitfalls of systems thinking in service design
2024 • Advancing Service Design 2024
Gold
Chris Hammond
Embedding sustainability into enterprise design and development: A journey towards "sustainability consciousness"
2025 • Climate UX Interest Group
JJ Kercher
A Roadmap for Maturing Design in the Enterprise
2018 • Enterprise Experience 2018
Gold
Kevin Bethune
Reimagining Design: Unlocking Strategic Innovation
2022 • Design at Scale 2022
Gold
Jenny Price
From Tradition to Transformation: Unlocking Startup Agility in a Legacy Enterprise
2025 • DesignOps Summit 2025
Gold
Iulia Cornigeanu
QuantQual Book Club: Small Data
2024 • QuantQual Interest Group

More Videos

April Reagan

"We're building bridges not only for ourselves but for the generations who will come after us."

April Reagan

Look, Think, Act: The Futures-Smart Design Organization

October 1, 2021

Ned Dwyer

"Sometimes research becomes the defacto 'talk to a customer' group, which can distance others from direct customer contact."

Ned Dwyer Emily Stewart James Wallis

The Intersection of Design and ResearchOps

September 24, 2024

Lona Moore

"Scaling design in an enterprise is like having more feet in the car — it gets you there faster."

Lona Moore

Scaling Design Beyond Designers

June 11, 2021

Ryan Rumsey

"The storytelling structure situation-complication-resolution works well for hard trade-off discussions."

Ryan Rumsey

Business Influence Without Losing Your Soul

January 14, 2021

Joshua Graves

"Psychological safety is key—if folks aren't in the right state, the conversation isn't productive."

Joshua Graves

We Need To Talk: Managing Ludicrous Requests at Work (Part 3 of 3)

May 12, 2025

Alicia D. Johnson

"Emergency management is political—not partisan, but political—and that reality has transformed how we work in this field."

Alicia D. Johnson

Disasters and the 21st Century

December 10, 2021

Laura Klein

"If you think innovation can happen without buy-in from the CEO, I would love to see that; I have not seen it work."

Laura Klein

Unique challenges of innovation in enterprises

April 23, 2020

Holly Cole

"We’re the people who can have difficult conversations and shine light on things—this is our power and we need to use it better."

Holly Cole

Panel Discussion: Growing People and Teams

November 8, 2018

Brendan Jarvis

"What does it mean to be a design ops person? That question matters and we must revisit it often."

Brendan Jarvis

It was the Best of Times. It was the Worst of Times.

September 25, 2024