Rosenverse
Moving AI offscreen: Exploring failures, constraints, and recovery in physical game design

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Moving AI offscreen: Exploring failures, constraints, and recovery in physical game design

Gold
Wednesday, June 10, 2026 • Designing with AI 2026

This video is featured in the AI Conference playlist.

Share the love for this talk
Moving AI offscreen: Exploring failures, constraints, and recovery in physical game design
Speakers: Kritika Sony
Link:

Summary

This case study explores what happens when generative AI moves off the screen and into a physical, walk-up-and-play game. Built by a team of four non-engineers, the project used AI to design and prototype everything from game logic to custom 3D-printed and laser-cut controllers. While AI accelerated early exploration, it repeatedly failed in physical space, where sensor noise, variability, and real players exposed its limits. The case study focuses on how designing for failure, constraints, and recovery ultimately mattered more than making the AI smarter, and what this reveals about trust, responsibility, and design judgment when AI becomes part of a real-world interactive system.

Key Insights

  • •

    AI compresses years of training and enables access to unfamiliar design domains rather than replacing expertise.

  • •

    Providing AI with rich contextual information via voice input improves output relevance significantly compared to short typed prompts.

  • •

    Validating concepts cheaply with basic physical prototypes before investing in hardware avoids wasting resources.

  • •

    AI can generate usable 3D design files and even scripting code to create printable models, accelerating prototyping.

  • •

    Iterative back-and-forth with AI to refine outputs is a critical skill, not a single-prompt solution.

  • •

    AI-generated solutions may work once but can fail in variable real-world conditions, necessitating user testing.

  • •

    Physical hardware design forces more rigorous consideration of edge cases compared to software alone.

  • •

    Tools like Adafruit's Circuit Playground Express enable physical inputs to simulate keyboard presses in software easily.

  • •

    Scaling 3D prints requires adjusting slicer settings manually, a complexity AI cannot yet fully solve.

  • •

    AI opens the door for creators to realize ideas they thought were out of reach, even without formal expertise.

Notable Quotes

"AI is not going to make you an expert but it compresses access to previously inaccessible domains."

"Don’t type with AI, talk to it like a brilliant colleague who knows nothing about your project yet—give it everything."

"Working with AI is a back and forth negotiation, not a vending machine where one prompt gets you the answer."

"The lesson isn’t that AI was wrong, but that you still have to test it against your users."

"I am not a hardware engineer, but I made something requiring those skills because AI gave me access to the doorway."

"In physical design, edge cases are more visible and testing immediately shows you problems you might have missed in software."

"AI can generate 3D models and geometry files, which you can directly import and print after careful checking."

"When scaling up 3D prints, slicer settings have to be adjusted—something AI didn’t help me with."

"A feature that only works some of the time is a feature that fails publicly in a walk-up installation."

"You don’t have to build a game; maybe you want to build a toy for your cat or a lamp—it just has to be yours."

Ask the Rosenbot
Louis Rosenfeld
Coffee with Lou: Should You Write a (UX) Book?
2024 • Rosenfeld Community
Bassel Deeb
Do More With Less: Equip and Lead Design Orgs Through Adversity
2023 • DesignOps Summit 2023
Gold
Daniel Gloyd
Transcendent Experience (ZX) Design: A Design Framework for Human Connection
2026 • Rosenfeld Community
Bria Alexander
Opening Remarks
2021 • Design at Scale 2021
Gold
Jen Crim
Culture, DIBS & Recruiting
2021 • Design at Scale 2021
Gold
Amy Bucher
Harnessing behavioral science to uncover deeper truths
2025 • Advancing Research 2025
Gold
Aaron Stienstra
Leveraging Civic Design to Advance Equity and Rebuild Trust in the US Federal Government
2021 • Civic Design 2021
Gold
Peter Levin
Solve a Problem Here, Transform a Strategy There: Research as an Occasion for Expanding Organizational Possibility
2024 • Advancing Research 2024
Gold
Sam Yen
Driving Organizational Change Through Design? Do more of this and less of that
2017 • Enterprise Experience 2017
Gold
Jon Fukuda
Theme 3 Intro
2024 • DesignOps Summit 2024
Gold
Louis Rosenfeld
GenAI for UXers: A Rosenbot Demo and Discussion
2025 • Designing with AI 2025
Gold
Alla Weinberg
How to Build and Scale Team Safety
2024 • DesignOps Summit 2020
Gold
Dr. Jamika Burge
Embracing change: Navigating shifting landscapes with compassion and agency
2025 • Advancing Research 2025
Gold
Sam Proulx
Prototype Reviews, People With Disabilities, and You
2021 • DesignOps Summit 2021
Gold
Trisha Causley
[Demo] Complexity in disguise: Crafting experiences for generative AI features
2024 • Designing with AI 2024
Gold
Deirdre Hirschtritt
Research is Only as Good as the Relationships You Build
2022 • Civic Design 2022
Gold

More Videos

Dr. Karl Jeffries

"Creativity fuels design, design fuels innovation."

Dr. Karl Jeffries

The Science of Creativity for DesignOps

January 8, 2024

Amy Marquez

"You have to understand how the business actually makes money to align UX with it."

Amy Marquez

INVEST: Discussion

June 15, 2018

Steve Portigal

"Building an internal research practice is critical for long-term value and integration of insights."

Steve Portigal Chris Chapo Kelly Goto Christian Rohrer

Discussion

May 13, 2015

Alicia Mooty

"Involving designers early in the process is crucial to avoid the 'fast-food' model where they feel like order takers."

Alicia Mooty

Design Staffing Models

September 30, 2021

Ana Maria Montero Barrantes

"We want to stay in the game no users cursing our name."

Ana Maria Montero Barrantes Aditi Dhar Michelle Kaplan Nate Osborne Matt Laurence

The Authentic UX Talent Show

January 8, 2024

Lada Gorlenko

"Operations tends to be something companies only think about when they are scaling rapidly, but investing early is essential."

Lada Gorlenko Sharbani Dhar Sébastien Malo Rob Mitzel Ivana Ng Michal Anne Rogondino

Theme 1: Discussion

January 8, 2024

Sofia Quintero

"It’s very difficult to align researcher goals with business growth, but that’s exactly what we try to do."

Sofia Quintero

The Product Philosophy Behind EnjoyHQ

March 10, 2021

Laura Klein

"Large companies demand certainty, but certainty is the opposite of innovation."

Laura Klein

Unique challenges of innovation in enterprises

April 23, 2020

Andrew Michael

"Data analysts and user researchers rarely work together effectively, which is one of the biggest missed opportunities."

Andrew Michael

Building a Product Insights Team

March 10, 2022

Latest Books All books