Rosenverse

Testing and Experimentation Tools

Gold
Thursday, March 12, 2026 • Advancing Research 2026

This video is featured in the UX metrics & analytics playlist.

Share the love for this talk
Testing and Experimentation Tools
Speakers: Erin Weigel and Petra Rajkov
Link:

Summary

Join us for a deep dive into testing and experimentation tools at the UXR Tools Summit. Design for Impact author Erin Weigel will lead a broad discussion on the category, and will also be joined by UXtweak, who will reveal their vision for the future of experimentation, showcase innovative capabilities. Ideal for UX, design, and market researchers eager to explore how emerging tools are transforming decision-making, optimization, and evidence-based product design.

Key Insights

  • •

    AI-generated follow-up questions in usability tests do not uncover novel user issues as effectively as human moderators.

  • •

    Participants tend to get frustrated with AI moderators due to repetitive or obvious questioning.

  • •

    Context is crucial for effective research moderation, which AI currently lacks.

  • •

    Synthentic participants are useful for brainstorming but cannot replace real user research.

  • •

    Democratization of research increases access but risks diluting research quality without expert involvement.

  • •

    AI features that save researchers time by handling repetitive tasks are promising and more practical.

  • •

    AI moderation should operate as a human-in-the-loop assistant, not a full replacement.

  • •

    Bias in AI and lack of diversity representation remains a critical challenge currently unaddressed.

  • •

    UX Tweak focuses on researcher-centered tools, expert support, and end-to-end UX research platform.

  • •

    There is healthy skepticism about AI fully simulating unpredictable human behavior in usability testing.

Notable Quotes

"Testing tools can help us bring the body language of the internet in front of us to see what people are doing."

"AI cannot substitute the research expertise achieved through multiple interviews and synthesis by humans."

"There was no meaningful difference between AI follow-up questions and static researcher questions in uncovering usability issues."

"AI follow-up questions led to frustrated participants because it asked highly obvious questions and ignored detailed context."

"If all tools try to be for everyone, they risk serving no one properly."

"AI should be a force multiplier handling repetitive tasks, not a replacement for human judgment."

"Synthetic participants shouldn't be used as a methodology but are great for cold starting research."

"Human moderation will always have a place because humans are unpredictable and random."

"We never train AI models on your data, and all AI features are strictly opt-in."

"There is a huge space for improvement in AI features for UX research if these have any future."

Ask the Rosenbot
Cassandra Piester
Developing and Deploying Your Design Operations Strategy
2024 • DesignOps Summit 2024
Gold
John Calhoun
Meters, Miles, and Madness: New Frameworks to Measure the (Elusive) Value of DesignOps
2024 • DesignOps Summit 2024
Gold
Adel Du Toit
Get Your CFO To Say: 'Our Strategic Goal is User Obsession'
2022 • Design at Scale 2022
Gold
Tanya Snook
Designing the team experience: Building culture through onboarding
2021 • Enterprise Community
Nora Tejeda
Scaling Design Capabilities at BBVA Through a Self-service Design Model
2021 • Design at Scale 2021
Gold
Sam Proulx
Online Shopping: Designing an Accessible Experience
2023 • Enterprise UX 2023
Gold
Changying (Z) Zheng
Practical DesignOps: From Ideas to Tools That Teams Actually Use
2025 • Rosenfeld Community
Amahra Spence
Designing for Liberation, Rehearsing Freedom
2022 • Civic Design 2022
Gold
Roberta Dombrowski
5 Reasons to Bring your Recruiting in House
2021 • DesignOps Summit 2021
Gold
Jacqui Frey
Scale is Social Work
2020 • DesignOps Community
Maria Rosala
Research Repositories
2026 • Advancing Research 2026
Gold
Alissa Briggs
How to Coach Enterprise Experimentation
2015 • Enterprise UX 2015
Gold
Nicole Wright
Democratizing Research at HoneyBook
2022 • Advancing Research 2022
Gold
Connect with the Advancing Research Community
2026 • Advancing Research 2026
Gold
Nick Cochran
Growing in Enterprise Design through Making Connections
2019 • Enterprise Experience 2019
Gold
Miles Orkin
Creativity and Culture
2018 • DesignOps Summit 2018
Gold

More Videos

Dante Guintu

"We matched designers with new peer buddies outside their usual domains to encourage learning and networking."

Dante Guintu

How to Crush the Talent Crunch

September 8, 2022

Josh Clark

"Machine intelligence can elevate experiences by easing frictions and amplifying human judgment and agency."

Josh Clark Veronika Kindred

Sentient Design, AI, and the Radically Adaptive Experience (1st of 3 seminars)

January 15, 2025

Christian Crumlish

"This is for me an amazing opportunity to see these ideas explored by brilliant people with a wonderful community."

Christian Crumlish

Introduction by our Conference Chair

December 6, 2022

Sam Proulx

"Mobile accessibility features are built in and ready to go without needing to install or configure anything extra."

Sam Proulx

Mobile Accessibility and You

June 9, 2022

Corey Long

"Treat unemployment as a full-time job; be relentless and turn over every stone to improve your chances."

Corey Long

Hiring in DesignOps: A Critical Study on How to Hire and Get Hired

September 23, 2024

Kevin Bethune

"Design has a beautiful advantage in understanding the story needed to be market relevant and connect with audiences."

Kevin Bethune

Gatekeepers and Servant Leadership

January 30, 2020

Peter Van Dijck

"The more I work with evals, the more I think UX and product people need to be involved because of the need for diverse perspectives."

Peter Van Dijck

Hands-on AI #2: Understanding evals: LLM as a Judge

October 15, 2025

Tracy McGoldrick

"Getting internal teams on board is often the biggest challenge, but once they see the value, it lights a fire."

Tracy McGoldrick

IBM User Experience Program—The What, Why and How

October 15, 2021

Kat Vellos

"One in five people say they deal with feelings of disconnection and loneliness in the workplace."

Kat Vellos

Opener: The Other L Word

January 8, 2024