Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Thursday, April 4, 2024 • Rosenfeld Community
Share the love for this talk
Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
Speakers: Cheryl Platz
Link:

Summary

Did you know that you're probably designing multimodal experiences? Most designers today are working in a multimodal environment, but haven't been trained to make the most of the many capabilities the latest generations of devices provide. Your customers have a small universe of devices, and they now come with the ability to handle far more than traditional haptic input like keyboards and mice. Gestural input, from swipes to hand gestures in video calls to stylus input, is becoming more common. Audio input and output are becoming more important in a world where digital assistants are poised to make a second surge powered by large language model AI. Visual output has become much more nuanced, and sometimes spans multiple devices. How do you wrangle all of this, optimize for great experiences, and still keep the human at the center? By becoming more consciously aware of the different inputs and outputs we're working with - and the many ways these inputs and outputs include and exclude our customers - we can build more resilient, more inclusive, and more powerful next-generation experiences.

Key Insights

  • Most designers already practice multimodal design unintentionally by supporting multiple input and output modalities like voice, touch, and haptics.

  • Multimodal design requires orchestrating how different modalities and devices communicate and transition smoothly, rather than designing in isolation.

  • Users switch devices when modality limits push them to—for example, moving from phone to desktop due to input convenience.

  • Cheryl’s CROW framework (Character, Relationship, Objective, Where) provides a comprehensive lens to capture user context relevant for multimodal design.

  • Transitions between input and output modes, devices, and network connectivity represent critical failure points and require intentional design.

  • Proactive, context-aware communication and interruption management improve user experience during multimodal and cross-device interactions.

  • Multimodal design and service design are deeply intertwined, especially across multiple devices and touchpoints.

  • Adaptive experiences that allow users to choose modalities based on context, such as hands-free vs. touch, lead to more inclusive designs.

  • Physical remotes remain relevant because phones are not always accessible, have battery constraints, and because remotes offer lower distraction.

  • Empathy for stakeholders’ goals and framing multimodal design benefits in terms of business priorities helps gain buy-in.

Notable Quotes

"Multimodal design is the design practice you probably don’t realize you’re doing already."

"The future is multimodal because humans are multimodal."

"Customers move from one device to another often because the information density or proximity to the device changes."

"Character, Relationship, Objective, and Where gives us a storytelling shorthand for fully understanding user context."

"Transitions will make or break your cross-device multimodal experiences."

"We don’t want to list every single customer activity; we need to abstract behavior patterns to manage interruptions."

"If you are altering any form of self-expression, you need to be really careful."

"Multimodal design adds a layer of orchestration on top of modality-specific designs like voice or graphical UI."

"Physical remotes have a lower power draw and are often more purpose-driven than phones, which are sources of stress."

"We treat our stakeholders as customers to understand their scenarios and align on a shared north star principle."

Ask the Rosenbot
Vitorio Miliano
Don’t call it AI: Turn words into numbers with quantitative ethnography
2026 • Advancing Research 2026
Conference
Maria Taylor
Knowledge is Power: Managing the Lifeblood of the Design Org
2023 • DesignOps Summit 2023
Gold
Ron Bronson
Design, Consequences & Everyday Life
2022 • Civic Design 2022
Gold
Kate Koch
Flex Your Super Powers: When a Design Ops Team Scales to Power CX
2021 • DesignOps Summit 2021
Gold
Sheryl Cababa
Thinking in systems to address climate with Sheryl Cababa
2024 • Climate UX Interest Group
Amahra Spence
Designing for Liberation, Rehearsing Freedom
2022 • Civic Design 2022
Gold
Mansi Gupta
Women-Centric Research: What, Why, How
2023 • Advancing Research 2023
Gold
Chris Chapo
Data Science and Design: A Tale of Two Tribes
2015 • Enterprise UX 2015
Gold
Rebecca Gimenez
Work in Progress: Service Design at Airbnb
2024 • Advancing Service Design 2024
Gold
Phil Gilbert
A Consistent Culture of Design
2015 • Enterprise UX 2015
Gold
Adam Cutler
People + Places + Practices = Outcomes
2016 • Enterprise UX 2016
Gold
Aletheia Delivre
New Shapes and Emerging Identities for Design Ops
2025 • DesignOps Summit 2025
Gold
Mansi Gupta
Drawing from Feminist Practice to Make Inclusive Design Operational
2022 • DesignOps Summit 2022
Gold
Louis Rosenfeld
Welcome / Housekeeping
2023 • Enterprise UX 2023
Gold
Laine Riley Prokay
How DesignOps can Drive Inclusive Career Ladders for All
2021 • DesignOps Summit 2021
Gold
Barb Spanton
Doing Work That Matters: A Look Beyond The Idealistic Notion of 'Doing Meaningful Work'
2022 • Design at Scale 2022
Gold

More Videos

Carla Casariego

"The term full kit team was coined because you needed the full set of people to produce exceptional experiences."

Carla Casariego Sarah Spencer

DesignOps in Wonderland

October 24, 2019

Ariel Kennan

"It's surprisingly hard to hire product managers and designers in government; I'm looking at how to unblock those system barriers."

Ariel Kennan

Civic Design in 2022

January 13, 2022

Kaitlin Tasker

"Frames are not static; they are tools for continuous discovery, reflection, and change."

Kaitlin Tasker

Fast and Fearless Inclusive Research

March 27, 2023

Sara Asche Anderson

"We want to be an inspiring friend you go to for advice, not just someone who tells you what to buy."

Sara Asche Anderson Jamie Kaspszak

Not Your Ordinary Re-Brand: Design's Path to Driving Customer Obsession at Best Buy

January 8, 2024

Bria Alexander

"Sponsor sessions are not sales pitches; they offer content similar in quality to the main program."

Bria Alexander

Opening Remarks

September 29, 2021

Cennydd Bowles

"Designers use deception all the time, but the moral dividing line between acceptable and harmful deception is fuzzy."

Cennydd Bowles

Exit Interview #2: Rediscovering the ethical heart of design

November 6, 2025

Jilanna Wilson

"Emojis and GIFs have become effective tools for remote teams to add expression and prevent miscommunication."

Jilanna Wilson

Distributed Design Operations Management

October 23, 2019

Steve Chaparro

"My job was to ask tons of discovery questions to uncover what my clients really needed, not just what they said they wanted."

Steve Chaparro

Bringing Into Alignment Brand, Culture and Space

August 13, 2020

Alnie Figueroa

"A prototype is worth a thousand meetings."

Alnie Figueroa

Teamwork: Strategies for Effective Collaboration with Other Program Management Teams

September 8, 2022