Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Thursday, April 4, 2024 • Rosenfeld Community
Share the love for this talk
Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Speakers: Cheryl Platz
Link:

Summary

Did you know that you're probably designing multimodal experiences? Most designers today are working in a multimodal environment, but haven't been trained to make the most of the many capabilities the latest generations of devices provide. Your customers have a small universe of devices, and they now come with the ability to handle far more than traditional haptic input like keyboards and mice. Gestural input, from swipes to hand gestures in video calls to stylus input, is becoming more common. Audio input and output are becoming more important in a world where digital assistants are poised to make a second surge powered by large language model AI. Visual output has become much more nuanced, and sometimes spans multiple devices. How do you wrangle all of this, optimize for great experiences, and still keep the human at the center? By becoming more consciously aware of the different inputs and outputs we're working with - and the many ways these inputs and outputs include and exclude our customers - we can build more resilient, more inclusive, and more powerful next-generation experiences.

Key Insights

  • Most designers already practice multimodal design unintentionally by supporting multiple input and output modalities like voice, touch, and haptics.

  • Multimodal design requires orchestrating how different modalities and devices communicate and transition smoothly, rather than designing in isolation.

  • Users switch devices when modality limits push them to—for example, moving from phone to desktop due to input convenience.

  • Cheryl’s CROW framework (Character, Relationship, Objective, Where) provides a comprehensive lens to capture user context relevant for multimodal design.

  • Transitions between input and output modes, devices, and network connectivity represent critical failure points and require intentional design.

  • Proactive, context-aware communication and interruption management improve user experience during multimodal and cross-device interactions.

  • Multimodal design and service design are deeply intertwined, especially across multiple devices and touchpoints.

  • Adaptive experiences that allow users to choose modalities based on context, such as hands-free vs. touch, lead to more inclusive designs.

  • Physical remotes remain relevant because phones are not always accessible, have battery constraints, and because remotes offer lower distraction.

  • Empathy for stakeholders’ goals and framing multimodal design benefits in terms of business priorities helps gain buy-in.

Notable Quotes

"Multimodal design is the design practice you probably don’t realize you’re doing already."

"The future is multimodal because humans are multimodal."

"Customers move from one device to another often because the information density or proximity to the device changes."

"Character, Relationship, Objective, and Where gives us a storytelling shorthand for fully understanding user context."

"Transitions will make or break your cross-device multimodal experiences."

"We don’t want to list every single customer activity; we need to abstract behavior patterns to manage interruptions."

"If you are altering any form of self-expression, you need to be really careful."

"Multimodal design adds a layer of orchestration on top of modality-specific designs like voice or graphical UI."

"Physical remotes have a lower power draw and are often more purpose-driven than phones, which are sources of stress."

"We treat our stakeholders as customers to understand their scenarios and align on a shared north star principle."

Ask the Rosenbot
Fatimah Richmond
The Future of ReOps as a Strategic Function: A Roadmap for Getting There
2024 • Advancing Research 2024
Gold
Peter Van Dijck
Building the Rosenbot
2024 • Designing with AI 2024
Gold
Alexis Lucio
Scaling Accessibility Through Design Systems
2022 • Design at Scale 2022
Gold
April Reagan
Look, Think, Act: The Futures-Smart Design Organization
2021 • DesignOps Summit 2021
Gold
Bria Alexander
The Big Question about Resilience: A panel discussion
2024 • DesignOps Summit 2024
Gold
Theresa Slate
Why Changing Hearts & Minds Doesn’t Work When Promoting DE&I Efforts, but Checklists Do
2023 • DesignOps Summit 2023
Gold
Nicole Aleong
Future Orientations to Everyday Life: Futures Anthropology as a Methodology
2024 • Advancing Research 2024
Gold
Nina Jurcic
The Design System Rollercoaster: From Enabler and Bottleneck to Catalyst for Change
2023 • DesignOps Summit 2023
Gold
Rachael Greene
Building a Design Ops Practice that Really Works (Most of the Time)
2025 • DesignOps Community
Amy Thibodeau
Opening Keynote: Process and Ambiguity
2019 • DesignOps Summit 2019
Gold
Victor M. Gonzalez
Practicing Learners and Learning Practitioners
2021 • Advancing Research 2021
Gold
Sean McKay
Coexisting with non-researchers: Practical strategies for a democratized research future
2025 • Advancing Research 2025
Gold
Silke Bochat
5 Antifragile Strategies for a DesignOps 2.0
2024 • DesignOps Summit 2024
Gold
Nicole Wright
Democratizing Research at HoneyBook
2022 • Advancing Research 2022
Gold
Sarah Fathallah
Lessening the Research Burden on Vulnerable Communities
2020 • Advancing Research 2020
Gold
Dan Hill
Designing for the infrastructures of everyday life
2024 • Designing with AI 2024
Gold

More Videos

Ned Dwyer

"There was a feeling at Brex that research was sometimes conducted for the sake of research, not resulting in impactful insights."

Ned Dwyer Emily Stewart James Wallis

The Intersection of Design and ResearchOps

September 24, 2024

Michael Land

"You don’t want so much governance that people can’t be creative, but you need just enough to help their process."

Michael Land

Establishing Design Operations in Government

February 18, 2021

Sam Proulx

"Blockchain apps aren’t located in one country or jurisdiction, making legal enforcement of accessibility laws difficult."

Sam Proulx

To Boldly Go: The New Frontiers of Accessibility

November 18, 2022

Vasileios Xanthopoulos

"We run our UX department as a business, with responsibilities, requirements, and expectations."

Vasileios Xanthopoulos

A Top-Down and Bottom-Up Approach to User-Centric Maturity at Scale

January 8, 2024

Andrew Custage

"Three quarters of users who fail their mission don’t come back, risking lost revenue and market share."

Andrew Custage Michael Mallett

The Digital Journey: Research on Consumer Frustration and Loyalty

March 29, 2023

Marjorie Stainback

"Our program changed trains people to create high quality, unbiased user research."

Marjorie Stainback Kelsey Kingman

Transforming Strategic Research Capacity through Democratization

October 24, 2019

Alla Weinberg

"Work gets done through our relationships, and that becomes more true the higher you go in your career."

Alla Weinberg

Cross-Functional Relationship Design

December 6, 2022

Jerome “Axle” Brown

"Before sharing insights, I try to prime the room by acknowledging feelings and starting fresh."

Jerome “Axle” Brown

How to Use Self-Directed Learning to Ensure Your Research Insights are Heard and Acted Upon

March 11, 2021

Jen Crim

"We set a goal to fill 25% of our early career roles with underrepresented demographics and landed 37%."

Jen Crim Jess Quittner Saritha Kattekola Alex Karr Gurbani Pahwa

Culture, DIBS & Recruiting

June 11, 2021