Accessible only to conference ticket holders.
Log in Create account
For 90 days after a conference, only paid ticket holders can watch conference videos. After that, all Gold members have access.
If you have purchased recording access and cannot see the video, please contact support.
Moving AI offscreen: Exploring failures, constraints, and recovery in physical game design
Summary
This case study explores what happens when generative AI moves off the screen and into a physical, walk-up-and-play game. Built by a team of four non-engineers, the project used AI to design and prototype everything from game logic to custom 3D-printed and laser-cut controllers. While AI accelerated early exploration, it repeatedly failed in physical space, where sensor noise, variability, and real players exposed its limits. The case study focuses on how designing for failure, constraints, and recovery ultimately mattered more than making the AI smarter, and what this reveals about trust, responsibility, and design judgment when AI becomes part of a real-world interactive system.
Key Insights
-
•
AI compresses years of training and enables access to unfamiliar design domains rather than replacing expertise.
-
•
Providing AI with rich contextual information via voice input improves output relevance significantly compared to short typed prompts.
-
•
Validating concepts cheaply with basic physical prototypes before investing in hardware avoids wasting resources.
-
•
AI can generate usable 3D design files and even scripting code to create printable models, accelerating prototyping.
-
•
Iterative back-and-forth with AI to refine outputs is a critical skill, not a single-prompt solution.
-
•
AI-generated solutions may work once but can fail in variable real-world conditions, necessitating user testing.
-
•
Physical hardware design forces more rigorous consideration of edge cases compared to software alone.
-
•
Tools like Adafruit's Circuit Playground Express enable physical inputs to simulate keyboard presses in software easily.
-
•
Scaling 3D prints requires adjusting slicer settings manually, a complexity AI cannot yet fully solve.
-
•
AI opens the door for creators to realize ideas they thought were out of reach, even without formal expertise.
Notable Quotes
"AI is not going to make you an expert but it compresses access to previously inaccessible domains."
"Don’t type with AI, talk to it like a brilliant colleague who knows nothing about your project yet—give it everything."
"Working with AI is a back and forth negotiation, not a vending machine where one prompt gets you the answer."
"The lesson isn’t that AI was wrong, but that you still have to test it against your users."
"I am not a hardware engineer, but I made something requiring those skills because AI gave me access to the doorway."
"In physical design, edge cases are more visible and testing immediately shows you problems you might have missed in software."
"AI can generate 3D models and geometry files, which you can directly import and print after careful checking."
"When scaling up 3D prints, slicer settings have to be adjusted—something AI didn’t help me with."
"A feature that only works some of the time is a feature that fails publicly in a walk-up installation."
"You don’t have to build a game; maybe you want to build a toy for your cat or a lamp—it just has to be yours."
Or choose a question:
More Videos
"Everyone who works within the world of design operations is a designer."
Spencer L. A. StultzWhy Social Justice Frameworks are Necessary for Successful DEI/JEDI Initiatives
October 4, 2023
"How do you influence bosses who don’t understand design, product, or technology? That’s a key challenge."
Louis Rosenfeld Christian CrumlishOpening Remarks
November 29, 2023
"Cohorts are one of the favorite parts of the conference, so really participate if you’re in one."
Bria AlexanderOpening Remarks
November 18, 2022
"We shifted UX Weekly to Product Design Biweekly, giving the team more time to engage meaningfully."
Briana Thomas Christina RodriguezWhen Design Ops Comes in H.O.T. : A Tale of a Transformed Design Org
September 30, 2021
"Customers are rarely asking us to fix inconsistencies. What we really need to do is examine indirect indicators like activation rates and how confusing our products are."
Nina JurcicThe Design System Rollercoaster: From Enabler and Bottleneck to Catalyst for Change
October 3, 2023
"Qualitative reasoning about causal mechanisms usually comes from lived experience, either the researcher or participants."
Joshua NobleCasual Inference
October 6, 2023
"Relational power requires empathy and curiosity about a wide range of stakeholders, not just the users."
Robert FabricantShifting dynamics: The evolving relationship between researchers, participants, and organizational systems
March 11, 2025
"Instead of spending eight hours job searching, cut it by half and spend the rest solving problems with the skills you have."
Dolly ParikhExit Interview #7: Journey of a Social Entrepreneur
May 21, 2026
"The landing charge is the key thing airlines look at and that needs to stay flat."
Stephen PollardClosing Keynote: Getting giants to dance - what can we learn from designing large and complex public infrastructure?
November 7, 2017
