Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Thursday, April 4, 2024 • Rosenfeld Community
Share the love for this talk
Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Speakers: Cheryl Platz
Link:

Summary

Did you know that you're probably designing multimodal experiences? Most designers today are working in a multimodal environment, but haven't been trained to make the most of the many capabilities the latest generations of devices provide. Your customers have a small universe of devices, and they now come with the ability to handle far more than traditional haptic input like keyboards and mice. Gestural input, from swipes to hand gestures in video calls to stylus input, is becoming more common. Audio input and output are becoming more important in a world where digital assistants are poised to make a second surge powered by large language model AI. Visual output has become much more nuanced, and sometimes spans multiple devices. How do you wrangle all of this, optimize for great experiences, and still keep the human at the center? By becoming more consciously aware of the different inputs and outputs we're working with - and the many ways these inputs and outputs include and exclude our customers - we can build more resilient, more inclusive, and more powerful next-generation experiences.

Key Insights

  • Most designers already practice multimodal design unintentionally by supporting multiple input and output modalities like voice, touch, and haptics.

  • Multimodal design requires orchestrating how different modalities and devices communicate and transition smoothly, rather than designing in isolation.

  • Users switch devices when modality limits push them to—for example, moving from phone to desktop due to input convenience.

  • Cheryl’s CROW framework (Character, Relationship, Objective, Where) provides a comprehensive lens to capture user context relevant for multimodal design.

  • Transitions between input and output modes, devices, and network connectivity represent critical failure points and require intentional design.

  • Proactive, context-aware communication and interruption management improve user experience during multimodal and cross-device interactions.

  • Multimodal design and service design are deeply intertwined, especially across multiple devices and touchpoints.

  • Adaptive experiences that allow users to choose modalities based on context, such as hands-free vs. touch, lead to more inclusive designs.

  • Physical remotes remain relevant because phones are not always accessible, have battery constraints, and because remotes offer lower distraction.

  • Empathy for stakeholders’ goals and framing multimodal design benefits in terms of business priorities helps gain buy-in.

Notable Quotes

"Multimodal design is the design practice you probably don’t realize you’re doing already."

"The future is multimodal because humans are multimodal."

"Customers move from one device to another often because the information density or proximity to the device changes."

"Character, Relationship, Objective, and Where gives us a storytelling shorthand for fully understanding user context."

"Transitions will make or break your cross-device multimodal experiences."

"We don’t want to list every single customer activity; we need to abstract behavior patterns to manage interruptions."

"If you are altering any form of self-expression, you need to be really careful."

"Multimodal design adds a layer of orchestration on top of modality-specific designs like voice or graphical UI."

"Physical remotes have a lower power draw and are often more purpose-driven than phones, which are sources of stress."

"We treat our stakeholders as customers to understand their scenarios and align on a shared north star principle."

Ask the Rosenbot
Maggie Dieringer
Creating Consistency Through Constant Change
2024 • DesignOps Summit 2020
Gold
Peter Van Dijck
Building the Rosenbot
2024 • Designing with AI 2024
Gold
Russell Blair
Killing the blank page
2024 • Designing with AI 2024
Gold
Sam Proulx
Prototype Reviews, People With Disabilities, and You
2021 • Civic Design 2021
Gold
Liza Pemstein
Scaling Research Via an Ops First Model at Clever
2023 • Advancing Research 2023
Gold
Weidan Li
Qualitative synthesis with ChatGPT: Better or worse than human intelligence?
2024 • Designing with AI 2024
Gold
Joerg Beringer
Scaling User Research with AI: Continuous Discovery of User Needs in Minutes
2025 • DesignOps Summit 2025
Conference
Sarah Brooks
Theme Three Intro
2022 • Civic Design 2022
Gold
Bria Alexander
Charting the future of DesignOps: A community workshop
2024 • DesignOps Community
Jemma Ahmed
Theme Panel
2025 • Advancing Research 2025
Gold
Daniel J. Rosenberg
Digital Medicine Design
2019 • Enterprise Community
Craig Villamor
Resilient Enterprise Design
2017 • Enterprise Experience 2017
Gold
John Maeda
Making Sense of Enterprise UX
2016 • Enterprise UX 2016
Gold
Mariesa Lenz
What Beekeeping Taught me about Product Teams
2025 • Rosenfeld Community
Wyatt Hayman
Global Research Panels
2020 • DesignOps Community
Greg Petroff
The Compass Mission
2021 • Advancing Research 2021
Gold

More Videos

Gabrielle Verderber

"If folks can't find the information, it doesn't matter how well it is written."

Gabrielle Verderber

Documentation Your Team Will Actually Use

October 3, 2023

Jennifer Kanyamibwa

"We had no budget, we had no backing, and it was a huge success."

Jennifer Kanyamibwa

Creating the Blueprint: Growing and Building Design Teams

November 8, 2018

Saara Kamppari-Miller

"If we are working on something that does not ladder up to our KSPs, then we should not be working on it."

Saara Kamppari-Miller Nicole Bergstrom Shashi Jain

Key Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”

November 13, 2024

Samuel Proulx

"Failing to design for disabilities means failing to design for our future selves."

Samuel Proulx

Designing beyond caricatures: Embracing real, diverse user needs

December 4, 2024

Catherine Dubut

"The store mobile device is the indispensable silent partner enabling store employees to balance task productivity and customer engagement."

Catherine Dubut

Bridging Physical and Digital Spaces: Approaches to Retail Service Design

March 18, 2021

Sam Ladner

"One in four American workers actively distrust their employers, believing they lie regularly."

Sam Ladner

Data Exhaust and Personal Data: Learning from Consumer Products to Enhance Enterprise UX

June 8, 2016

Sam Proulx

"For users with physical challenges, hover states can be very difficult, especially for eye, head tracking, or voice control users."

Sam Proulx

Prototype Reviews, People With Disabilities, and You

December 8, 2021

Jemma Ahmed

"The current decline in response and participant engagement rates is a genuine crisis that threatens research data quality."

Jemma Ahmed Megan Blocker Eduardo Ortiz

Redefining the research toolkit: Expanding methodologies for a changing world

March 12, 2025

Kevin Bethune

"Servant leadership means divvying up the work and giving team members runway to lead parts of the vision."

Kevin Bethune

Reimagining Design: Unlocking Strategic Innovation

June 8, 2022