Research Basis
References
The framework and evaluation criteria in this tool are grounded in the following peer-reviewed and industry literature on synthetic users in UX and human-centred design.
Table 1: Key sources on synthetic users in UX / HCD (pros and cons)
A cross-source summary of the main opportunities and risks documented in the literature.
| Source | Year | Type | Main pros / opportunities | Main cons / risks |
|---|---|---|---|---|
| Rosala & Moran, NN/g: "Synthetic Users: If, When, and How to Use AI-Generated Research" | 2024 | Industry UX evaluation | Fast synthesis of existing domain knowledge; helps with desk research and early hypothesis generation. | Shallow, idealized responses; unsafe for concept testing, prioritization, or persona creation; must not replace real-user research. |
| Pehar: "AI as Synthetic Users in Human-Centred Design: A Systematic Review" | 2025 | Peer-reviewed systematic review | Speed and cost reduction in early iterations; supports ideation, persona generation, and large-scale simulations for usability/scenario testing. | Authenticity and variability problems, bias, limited emotional/contextual fidelity, and ethical concerns; should augment, not replace, real users. |
| Gu et al.: "Synthetic users: insights from designers' interactions with persona-based chatbots" | 2025 | Peer-reviewed empirical study | Allows designers to ask follow-ups, test assumptions, and explore "what-if" questions quickly with interactive personas. | Accuracy doubts, out-of-character responses; treated more as brainstorming partners than ground-truth about users. |
| ACM Interactions: pieces on synthetic users and UX challenges | 2024–2026 | Peer-reviewed commentary | Acknowledge efficiency, scalable exploration, and potential diversity gains when synthetic users are carefully instrumented and validated. | Structural biases (over-agreeableness, Western defaults, emotional flatness); synthetic personas can echo assumptions instead of surfacing surprises. |
| MeasuringU / LinkedIn: "A Review of Experiments with Synthetic Users" | 2026 | Methodological review | Synthetic users can approximate some usability metrics and replicate coarse patterns of human data on well-structured tasks. | Plausible but wrong data, shallow qualitative output, low variability, failures on complex tasks; unvalidated synthetic findings can mislead. |
| Koji: "Synthetic Users in Research: Validity, Bias, and When AI Personas Are (and Aren't) Trustworthy" | 2026 | Industry methodology article | Useful for vocabulary familiarization, hypothesis generation, pre-interview rehearsal, and stress-testing research instruments. | Documented sycophancy and rating bias; invalid for decision-grade research (concept tests, prioritization, personas) without real-user validation. |
| Codexical: "Your AI Research Participants Never Once Disagreed With You" | 2026 | Industry critique | Workflow benefits: instant availability, flexible demographics, and speed for exploratory work and concept sketching. | Over-agreeableness, Western cultural defaults, and emotional flatness produce clean but systematically distorted findings. |
| UXArmy: "Synthetic Participants in UX Research: Hype or Future?" | 2025 | Industry UX tool blog | Speed, 24/7 availability, cost savings, scalability, and controlled scenarios for early usability testing and benchmarking. | Lack of true emotion and contextual awareness; risk of misleading data and limited usefulness for exploratory/generative research. |
| Making Science: "Enhancing UX/UI Research with Synthetic Users" | 2025 | Agency methodology article | Scalability and cost reduction; helpful for early-stage concept and prototype testing and affordance validation. | Lack of real empathy, shallow but superficially accurate insights, dependence on historical data quality; real-user research remains essential. |
| Holli Downs: "Synthetic Users" (Phosphor discourse review) | 2025 | Discourse review / synthesis | Efficiency, scale, privacy benefits, and potential data-diversity gains when synthetic users are used carefully. | Reduced authenticity, inability to reproduce emotional/contextual nuance, and overselling as replacements; argues for hybrid models. |
Table 2: Best-practice guidance for using synthetic users
Actionable recommendations distilled from the same body of literature.
| Source | Year | Type | Key best-practice themes / guidance |
|---|---|---|---|
| Rosala & Moran, NN/g: "Synthetic Users: If, When, and How to Use AI-Generated Research" | 2024 | Industry UX evaluation | Treat synthetic output as desk research and hypothesis fodder only; never present as "users said"; avoid using for concept validation, prioritization, or persona creation; be cautious in low-maturity orgs. |
| Pehar: "AI as Synthetic Users in Human-Centred Design: A Systematic Review" | 2025 | Systematic academic review | Adopt a hybrid HCD model where synthetic users are used in early stages but key decisions are validated with real users; document where AI replaces or augments user input; calibrate and validate simulations; use for scenario coverage rather than everyday feedback. |
| Gu et al.: "Synthetic users: insights from designers' interactions with persona-based chatbots" | 2025 | Empirical study | Position chat-based synthetic personas as interactive extensions of personas, not replacements; use to test assumptions and explore ideas; balance consistency and variation in behavior; treat them as creative partners, not authoritative users. |
| ACM Interactions: pieces on synthetic UX research | 2024–2026 | Peer-reviewed commentary | Make structural biases explicit in method sections; build validation loops comparing synthetic and human findings; use synthetic research only where later human data can falsify or confirm it; avoid letting synthetic personas stand in for ethnographic evidence. |
| Koji: "Synthetic Users in Research: Validity, Bias, and When AI Personas Are (and Aren't) Trustworthy" | 2026 | Industry methodology | Use synthetic users for vocabulary familiarization, hypothesis generation, and rehearsing guides; never for roadmap/feature decisions; follow a two-day hybrid workflow (Day 1 synthetic, Day 2 real interviews); label outputs as hypotheses requiring validation. |
| Codexical: "Your AI Research Participants Never Once Disagreed With You" | 2026 | Industry critique | Assume structural over-agreeableness; run small real-user samples alongside synthetic studies; treat divergence as a signal; pre-define which findings must be falsifiable with real users before shipping. |
| UXArmy: "Synthetic Participants in UX Research: Hype or Future?" | 2025 | UX tool blog | Use synthetic participants for early, task-based usability tests and A/B experiments; avoid for discovery interviews or emotion-heavy work; clearly communicate limitations to stakeholders; always validate critical insights with real participants. |
| Making Science: "Enhancing UX/UI Research with Synthetic Users" | 2025 | Agency article | Integrate synthetic users into multi-layered models with market, behavioral, and real user data; use for early exploration and prototype validation; maintain real-user research as the foundation of design strategy. |
| Holli Downs: "Synthetic Users" (Phosphor discourse review) | 2025 | Discourse synthesis | Advocate hybrid models combining AI scale with human depth; emphasize transparent reporting of where synthetic vs. real data are used; encourage case-study sharing to converge on community best practices. |