YouGov publishes scientific validation of AI twins

YouGov publishes scientific validation of its AI twins – and challenges industry to set standards for simulated research

YouGov created 1,008 “E-twins” and tested them against 693 real, government-ID verified people they were built from – asking both the same new questions

Twins’ results were validated through a recontact survey of the panellists they were based on within 48 hours

Twins trained on panellists’ ratings of 4,000+ brands and entities matched their human to within one point on a five-point scale 90% of the time, up from 78% on profiling data alone

Twins hit the same 90% on a topic they were never trained on

YouGov calls on the industry to agree on validation standards – and challenges every simulator to predict real-world outcomes in public

YouGov, the international research and data analytics group, today published a white paper on one of the most transparent tests yet of one-to-one human-to-digital-twin simulation.

Rather than comparing aggregate results against public datasets, the groundbreaking research checked each AI twin individually against its own human.

Working with OriginalVoices, an AI platform specialising in simulation, YouGov built digital twins – LLM-based simulations of real people’s attitudes, which it calls “E-twins” – of 1,008 government ID-verified members of its proprietary panel. It then put the same new questions to both the E-twin and the real human respondent. Fully 69% of the panellists responded within 48 hours, giving a direct check on the accuracy of their E-twins. In day-to-day operations, this validation can be done in as little as an hour.

The key findings from YouGov’s AI twins testing:

Rich data drives accuracy. Adding panellists' ratings of 4,000+ entities lifted person-level accuracy (within one point on a five-point scale) from 78% to 90%.

YouGov’s E-twins answered what they were never taught. Fiction books were not part of the training data, but when asked about them, the E-twins still came within one point of their human 90% of the time.

Twin confidence tracks with accuracy. Each open-ended answer a digital twin gives is assigned a confidence level (High, Medium or Low) by OriginalVoices. Panellists said the answer described their own views “very” or “somewhat” well for 81% of High-confidence answers, 72% of Medium and 67% of Low – giving researchers a practical guide to which individual twin responses to trust.

YouGov's real panelists regularly identify with their twins' opinions. 75% of the time, panellists judged their E-twin's open-ended answers as describing their own views.

Less stereotyping. More contextual training data reduced twins’ reliance on demographics, with no pattern of biased performance by age, gender, race or education.

Current simulations are built on shaky foundations

YouGov says much of the evidence offered to date by simulators across the industry rests on publicly available datasets. Those questions may already sit in the training data of the underlying language models, so a twin can appear to know an answer it has simply absorbed – and respondents cannot be recontacted to verify the results. Aggregate comparisons can also mask twins that flatten towards the average or stereotype.

Simulated data providers need to show their workings

YouGov is calling on the industry to develop shared standards of verification and validation for defined use cases. It challenges all simulators to follow its lead in validating twins against their individual human, test them on unseen topics, publish their methods, show their misses and predict real-world outcomes in public.

Stephan Shakespeare, CEO and co-founder of YouGov: “The research industry is being transformed by AI in many ways but especially when it comes to simulation. Whether this is a dangerous development or a beneficial advance depends on whether clients can know if its outputs are reliable. I challenge everybody in the industry to work together to establish standards for verifying and validating simulated data. This white paper shows what we think these standards should involve - including the ability to validate simultaneously. If the industry wants simulation to be trusted by clients and the public, it needs to earn it and this can only be achieved through widely agreed standards.”

YouGov’s white paper also publishes the gaps future analysis will address. The current study did not measure a ceiling – how often a person repeats their own answer to the same question. A twin can't reasonably beat the person it copies: if panellists repeated their own rating 70% of the time, the twins' 58% exact-match rate would represent 83% of that ceiling. YouGov will test this in the next iteration of its research.

Read the full white paper: An evaluation of digital twins based on YouGov panel data

YouGov /Research Reality