Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Thursday, April 4, 2024 • Rosenfeld Community
Share the love for this talk
Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Speakers: Cheryl Platz
Link:

Summary

Did you know that you're probably designing multimodal experiences? Most designers today are working in a multimodal environment, but haven't been trained to make the most of the many capabilities the latest generations of devices provide. Your customers have a small universe of devices, and they now come with the ability to handle far more than traditional haptic input like keyboards and mice. Gestural input, from swipes to hand gestures in video calls to stylus input, is becoming more common. Audio input and output are becoming more important in a world where digital assistants are poised to make a second surge powered by large language model AI. Visual output has become much more nuanced, and sometimes spans multiple devices. How do you wrangle all of this, optimize for great experiences, and still keep the human at the center? By becoming more consciously aware of the different inputs and outputs we're working with - and the many ways these inputs and outputs include and exclude our customers - we can build more resilient, more inclusive, and more powerful next-generation experiences.

Key Insights

  • Most designers already practice multimodal design unintentionally by supporting multiple input and output modalities like voice, touch, and haptics.

  • Multimodal design requires orchestrating how different modalities and devices communicate and transition smoothly, rather than designing in isolation.

  • Users switch devices when modality limits push them to—for example, moving from phone to desktop due to input convenience.

  • Cheryl’s CROW framework (Character, Relationship, Objective, Where) provides a comprehensive lens to capture user context relevant for multimodal design.

  • Transitions between input and output modes, devices, and network connectivity represent critical failure points and require intentional design.

  • Proactive, context-aware communication and interruption management improve user experience during multimodal and cross-device interactions.

  • Multimodal design and service design are deeply intertwined, especially across multiple devices and touchpoints.

  • Adaptive experiences that allow users to choose modalities based on context, such as hands-free vs. touch, lead to more inclusive designs.

  • Physical remotes remain relevant because phones are not always accessible, have battery constraints, and because remotes offer lower distraction.

  • Empathy for stakeholders’ goals and framing multimodal design benefits in terms of business priorities helps gain buy-in.

Notable Quotes

"Multimodal design is the design practice you probably don’t realize you’re doing already."

"The future is multimodal because humans are multimodal."

"Customers move from one device to another often because the information density or proximity to the device changes."

"Character, Relationship, Objective, and Where gives us a storytelling shorthand for fully understanding user context."

"Transitions will make or break your cross-device multimodal experiences."

"We don’t want to list every single customer activity; we need to abstract behavior patterns to manage interruptions."

"If you are altering any form of self-expression, you need to be really careful."

"Multimodal design adds a layer of orchestration on top of modality-specific designs like voice or graphical UI."

"Physical remotes have a lower power draw and are often more purpose-driven than phones, which are sources of stress."

"We treat our stakeholders as customers to understand their scenarios and align on a shared north star principle."

Ask the Rosenbot
Patrick Boehler
Fishing for Real Needs: Reimagining Journalism Needs with AI
2025 • Designing with AI 2025
Gold
Theresa Slate
Why Changing Hearts & Minds Doesn’t Work When Promoting DE&I Efforts, but Checklists Do
2023 • DesignOps Summit 2023
Gold
Harry Max
Prioritization for Leaders (2nd of 3 seminars)
2024 • Rosenfeld Community
Ivana Ng
Level Up Your Program with ProductOps
2024 • Enterprise Experience 2020
Gold
Nalini Kotamraju
Research After UX
2024 • Advancing Research 2024
Gold
Vicky Teinaki
Short Take #3: UX/Product Lessons from Your Industry Peers
2022 • Design in Product 2022
Gold
Angy Peterson
More Than Technology: Personalized Public Sector Experiences
2021 • Civic Design 2021
Gold
Brendan Jarvis
It was the Best of Times. It was the Worst of Times.
2024 • DesignOps Summit 2024
Gold
Alex Hurworth
Designing a Contact Tracing App for Universal Access
2020 • DesignOps Summit 2020
Gold
Louis Rosenfeld
GenAI for UXers: A Rosenbot Demo and Discussion
2025 • Designing with AI 2025
Gold
Sha Hwang
The Lost Year
2021 • Design at Scale 2021
Gold
Stephen Anderson
Puzzled? How to Coordinate Humans for Complex Challenges
2021 • Enterprise Community
Harry Brignull
Beyond Clicks and Tricks: Why deceptive design has grown into a regulatory faultline
2026 • Rosenfeld Community
John Calhoun
Have we Reached Our Peak? Spotting the Next Mountain For DesignOps to Climb
2021 • DesignOps Summit 2021
Gold
Tracy McGoldrick
IBM User Experience Program—The What, Why and How
2021 • Advancing Research Community
Rittika Basu
Age and Interfaces: Equipping Older Adults with Technological Tools
2023 • Advancing Research Community

More Videos

Kurt McCulloch

"Growth PMs think this way because it’s the only kind of thinking the system we’re working within can operationalize."

Kurt McCulloch

Faster alone, further together: Rebuilding collaboration in the age of AI research

March 10, 2026

Sam Proulx

"Those of us who do not understand history are doomed to repeat it."

Sam Proulx

To Boldly Go: The New Frontiers of Accessibility

November 18, 2022

Leah Buley

"We may need to borrow research techniques from journalists, actuaries, epidemiologists, ethicists, and science fiction writers."

Leah Buley

Closing Plenary: The Crisis of Digital

March 31, 2020

Christian Crumlish

"These questions haven’t gone away as I speak and do workshops and things like that around the book."

Christian Crumlish

Introduction by our Conference Chair

December 6, 2022

Stephen Anderson

"Psychological safety isn’t created by an initiative but through a thousand tiny habits in language and behavior."

Stephen Anderson

Puzzled? How to Coordinate Humans for Complex Challenges

May 27, 2021

Gabriela Barneva

"You don’t need a huge planned research study; even two participants can uncover valuable insights."

Gabriela Barneva

Operationalizing Inclusive Design in Service Design

November 20, 2025

Jay Bustamante

"It is not our job to know all the answers, but to make sure the right questions are asked."

Jay Bustamante

Navigating the Ethical Frontier: DesignOps Strategies for Responsible AI Innovation

October 2, 2023

Chris Engledowl

"You don’t know what you don’t know. Let’s write surveys that cover those blind spots."

Chris Engledowl

A Mixed Method Approach to Validity to Help Build Trust

April 28, 2023

Dave Gray

"Beliefs are habits of action; they are the rules that guide how we decide to act."

Dave Gray

Liminal Thinking: Sense-making for systems in large organizations

May 14, 2015