Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Thursday, April 4, 2024 • Rosenfeld Community
Share the love for this talk
Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Speakers: Cheryl Platz
Link:

Summary

Did you know that you're probably designing multimodal experiences? Most designers today are working in a multimodal environment, but haven't been trained to make the most of the many capabilities the latest generations of devices provide. Your customers have a small universe of devices, and they now come with the ability to handle far more than traditional haptic input like keyboards and mice. Gestural input, from swipes to hand gestures in video calls to stylus input, is becoming more common. Audio input and output are becoming more important in a world where digital assistants are poised to make a second surge powered by large language model AI. Visual output has become much more nuanced, and sometimes spans multiple devices. How do you wrangle all of this, optimize for great experiences, and still keep the human at the center? By becoming more consciously aware of the different inputs and outputs we're working with - and the many ways these inputs and outputs include and exclude our customers - we can build more resilient, more inclusive, and more powerful next-generation experiences.

Key Insights

  • Most designers already practice multimodal design unintentionally by supporting multiple input and output modalities like voice, touch, and haptics.

  • Multimodal design requires orchestrating how different modalities and devices communicate and transition smoothly, rather than designing in isolation.

  • Users switch devices when modality limits push them to—for example, moving from phone to desktop due to input convenience.

  • Cheryl’s CROW framework (Character, Relationship, Objective, Where) provides a comprehensive lens to capture user context relevant for multimodal design.

  • Transitions between input and output modes, devices, and network connectivity represent critical failure points and require intentional design.

  • Proactive, context-aware communication and interruption management improve user experience during multimodal and cross-device interactions.

  • Multimodal design and service design are deeply intertwined, especially across multiple devices and touchpoints.

  • Adaptive experiences that allow users to choose modalities based on context, such as hands-free vs. touch, lead to more inclusive designs.

  • Physical remotes remain relevant because phones are not always accessible, have battery constraints, and because remotes offer lower distraction.

  • Empathy for stakeholders’ goals and framing multimodal design benefits in terms of business priorities helps gain buy-in.

Notable Quotes

"Multimodal design is the design practice you probably don’t realize you’re doing already."

"The future is multimodal because humans are multimodal."

"Customers move from one device to another often because the information density or proximity to the device changes."

"Character, Relationship, Objective, and Where gives us a storytelling shorthand for fully understanding user context."

"Transitions will make or break your cross-device multimodal experiences."

"We don’t want to list every single customer activity; we need to abstract behavior patterns to manage interruptions."

"If you are altering any form of self-expression, you need to be really careful."

"Multimodal design adds a layer of orchestration on top of modality-specific designs like voice or graphical UI."

"Physical remotes have a lower power draw and are often more purpose-driven than phones, which are sources of stress."

"We treat our stakeholders as customers to understand their scenarios and align on a shared north star principle."

Ask the Rosenbot
Kritika Yadav
Optimizing AI Conversations: A Case Study on Personalized Shopping Assistance Frameworks
2025 • Designing with AI 2025
Gold
Louis Rosenfeld
Coffee with Lou #3: What Makes for a Successful UX Conference Presentation?
2024 • Rosenfeld Community
Cheryl Platz
Embrace Your Fun Factor: Game Development Best Practices for Product Design
2026 • Rosenfeld Community
Carol Scott
Avoid Harming Your Team and Users: Promoting Care and Brand Reputation with Trauma-Informed UX Practices
2025 • Rosenfeld Community
Alba Villamil
Stereotyped by Design: Pitfalls in Cross-Cultural User Research
2020 • Advancing Research 2020
Gold
Crystal Yan
Building a Customer-Centric Culture
2020 • Advancing Research 2020
Gold
Ricardo Martins
Unlocking the power of advanced quantitative methods
2025 • Advancing Research 2025
Gold
Dane DeSutter
What co-speech gestures reveal about users’ thinking during interviews
2023 • Rosenfeld Community
Dawn Ressel
Full-Stack User Experiences: A Marriage of Design and Technology
2016 • Enterprise UX 2016
Gold
Scott Plewes
Why Isn't Your UX Approach Going Viral?: A Mathematical Model
2023 • Advancing Research 2023
Gold
Melissa Eggleston
Practical People Skills for Building Trust on Teams and with Partners
2021 • Civic Design 2021
Gold
Sofia Quintero
The Product Philosophy Behind EnjoyHQ
2021 • Advancing Research 2021
Gold
Josina Vink
Navigating the pitfalls of systems thinking in service design
2024 • Advancing Service Design 2024
Gold
Rima Campbell
Increase Productivity and Drive Business Impact
2024 • DesignOps Summit 2024
Gold
Frances Yllana
The Big Question about Impact: A Panel Discussion
2024 • DesignOps Summit 2024
Gold
Ashley Cortez
Shifting Toward Community-Led Innovation in Local Government
2021 • Civic Design 2021
Gold

More Videos

Billy Carlson

"Constraints during ideation sessions help avoid endlessly blue-sky thinking and keep ideas focused."

Billy Carlson

Ideation tips for Product Managers

December 6, 2022

Sam Proulx

"If you haven’t done accessibility work yet, it makes sense to get started on mobile because you already have the tools in your pocket."

Sam Proulx

Mobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World

November 17, 2022

Dan Willis

"Are we bold enough to ditch the user interface in favor of a better experience, a transparent one?"

Dan Willis

Enterprise Storytelling Sessions

May 13, 2015

Michael Polivka

"Not a single person on this relationship map reported to me. That’s how I affect design at scale at Autodesk."

Michael Polivka

Scaling Design through Relationship Maps

November 7, 2017

Karen McGrane

"The biggest challenge with LLMs was keeping them on task instead of writing their own Reddit posts."

Karen McGrane Jeff Eaton

AI for Information Architects: Are the robots coming for our jobs?

November 21, 2024

Leisa Reichelt

"Ratios always start on the premise of spreading people too thin, which is not a great way to get success."

Leisa Reichelt

Opening Keynote: Operating in Context

November 7, 2018

Kim Holt

"Always remember the power of your values in times of doubt and change; lean into what your team really stands for."

Kim Holt Emma Wylds Pearl Koppenhaver Maisee Xiong

A Salesforce Panel Discussion on Values-Driven DesignOps

September 8, 2022

Jacqui Frey

"Listening with tactical empathy means understanding the full emotional journey during times of change."

Jacqui Frey Alison Rand

Setting the Table for Dynamic Change

October 24, 2019

"When I pitched the investment plan, CFOs and board members nodded because it was expressed in their language."

How to Identify and Increase your "Experience Quotient"

June 15, 2018