Test before you trust: not every good number is a good sign

Written by Moe Pouyan

The Research Week · November 9 to 13, 2026

When should you trust a number, a model or a metric? This week's research points to a practical habit for research and CX teams: name the check an AI feature, a synthetic response or a loyalty score has to pass, then run it. Three different kinds of evidence point that way. Patients report more comfort with AI when a clinician stays involved. A panel firm has published early validation of its digital twins against real people. And stated loyalty in Canadian banking looks stronger than the share of customers who are satisfied and want to stay.

The studies, from Gallup, YouGov, JD Power, Pew Research Center, Leger and NIQ, cover different things with different samples and methods, so they support a shared habit rather than one proven rule.

56%of US primary care patients are comfortable with AI deciding diagnosis and treatment when a provider can override it, against 9% with no provider reviewGallup
75%of the time, panellists judged their digital twin's open-ended answers as describing their own viewsYouGov
43%of Canadians are satisfied with their main financial institution and want to stay, though 78% would likely choose it againLeger
+3.2%UK supermarket sales growth in October, even as consumer confidence fell 11 points to minus 24NIQ; confidence from GfK

Human oversight can make AI more acceptable

An attendant in a blue jacket helps an older passenger board a driverless shuttle at a city stop.

Customers' stated comfort with AI depends partly on whether a person can still step in. In Gallup's two complementary surveys, one of patients and one of primary care physicians, 56% of patients said they were comfortable with AI making diagnosis and treatment decisions when a provider could override it, more than the 38% comfortable with AI only assisting while the provider decides. With no provider review at all, comfort fell to 9%. That gap between the two provider-involved models is specific to diagnosis: for imaging review the figures were 53% and 52%, and for medication decisions 49% and 45%. Providers were asked whether most of their patients would be comfortable, and 36% said yes for the override model; Gallup reads this as an underestimate, but because the questions differ it is better seen as a difference in perspective.

A further result is the most practical. Among patients uneasy about AI deciding without review, the ability to request a provider review was the most persuasive reassurance Gallup tested: 48% said it would make them more comfortable with AI in diagnosis and treatment, ahead of strong privacy protections (33%) and studies showing safety and effectiveness (30%).

Outside healthcare, YouGov's eight-country survey on self-driving transport (fielded May to June 2026) found 48% to 69% would choose a human-driven taxi or ride-share over a self-driving one. When YouGov asked again about self-driving buses, trains and metros with staff on board, it reports a marked increase in comfort in every case, though the article does not publish those figures.

My reading: people seem to accept automation more readily when someone answerable remains available and can be asked for. The strongest evidence is Gallup's, about stated comfort in healthcare; whether it holds elsewhere, and in what people actually do, is still open.

What to do with this. When you design an AI feature, test versions with and without a visible human review, including one where customers can ask for a person themselves, and measure both stated comfort and actual use. Ask customers directly rather than relying on staff estimates of what they will accept.

AI-made insight needs validation against real people

Sketched portrait cards matched with blue wireframe versions of the same faces, one mismatched pair circled in red.

Synthetic respondents, chatbot data and even traditional survey panels are only as useful as their checks against reality. This week YouGov published validation results with AI platform OriginalVoices. They address accuracy, not the consent concerns Prolific's participants raised last week. It built 1,008 digital twins and asked the same new questions of the twins and their panellists, 69% of whom answered within 48 hours. Panellists judged their twin's open-ended answers as describing their own views 75% of the time. Twin ratings came within one point of the person's own on a five-point scale 90% of the time once panellists' ratings of more than 4,000 entities were added, up from 78% with core profile data, and reached the same 90% on fiction books, a topic outside the training data.

Those results are encouraging, but the test is generous. On a five-point scale, "within one point" allows up to three of the five answers to count as a match, and exact matches were 58%. YouGov also notes it has not yet measured how often people repeat their own answers, the human baseline any twin should be judged against, and the comparison relies on panellists who replied quickly. YouGov itself wants firms using twins or synthetic data to validate them against real respondents, publish the results, and agree shared standards.

Newer sources lack even that. OpenAI will let verified businesses see how often ChatGPT mentions them, the clicks they receive and common themes in conversations that cited their site, aggregated above thresholds it has not published. It resembles brand tracking, with no stated view of who is in the base.

Traditional surveys face their own version of the test, and errors can run either way. In Research Live's industry roundup, The Washington Post's polling director said Democrats in its Kansas poll were more than twice as likely as Republicans to complete the survey once they picked up, and that poll overstated the Democrat's lead. Yet across 27 Senate and governor races, Pew's updated review of 2026 polling accuracy found averages understated the Democratic margin by about 4 points, even though Democratic supporters were more likely to take preelection surveys. Weighting may not fully correct for who chooses to respond.

My reading: whether the data come from a model, a chatbot or a panel, confidence should rest on comparison with an outside benchmark, such as the person's own answers or the actual vote, rather than on how plausible the output looks. Here the benchmarks are only partly in place: YouGov's results come from a vendor testing its own product without a human repeat baseline yet, and OpenAI has published no thresholds at all.

What to do with this. Before synthetic answers or chatbot mentions enter a deck, ask for validation against real people with the measure defined, including exact-match rates and a human repeat baseline, published thresholds, and a clear statement of whose behaviour the data reflect. Apply the same test to your own trackers by watching who responds, not just how many.

Loyalty scores give an incomplete picture without behavioural context

A person pauses in a bank's revolving door, looking back toward a warm café across the street.

Stated loyalty and confidence can sit some distance from what customers do, which is a reason to read them alongside behaviour. Last week's streaming research reported high satisfaction alongside rising cancellations, without showing whether satisfied customers were the ones leaving. This week Leger's Canadian banking research separated repeat-choice intention from reasons for staying: 78% would likely choose their main financial institution again, but only 43% are satisfied and want to stay, while half say they stay for other reasons, such as switching effort, no compelling alternative or preference for the status quo. All of these are stated measures; Leger does not track whether these customers later switch.

Shoppers show a similar gap between view and habit. In YouGov's Black Friday survey of 2,013 US adults, half shop the November sales. Only 30% of those sale shoppers think early deals are better, yet about 80% of them already do at least some early shopping. Both are self-reports, not observed purchases.

Behaviour can also move against sentiment. NIQ recorded UK supermarket sales (its Total Till measure) up 3.2% in the four weeks to October 31, with units up 1.5%, even as GfK consumer confidence fell 11 points to minus 24. The release points to Halloween and half-term, and to a 27% jump in canned goods in the second half of the month, which it says the government's "Be Ready" preparedness advice may have contributed to. Food inflation also rose to 2.9%. That context may explain why spending rose while confidence fell.

Where customers can check whether a promise was kept, the gap between view and experience narrows. In JD Power's business electric utility study, businesses whose power was restored by the promised time reported much higher trust (655 on a 1,000-point scale) than those whose restoration came later (465), though this is a single survey, not a test of later behaviour.

My reading: repeat-choice intention can include customers who are simply staying put, and sentiment can diverge from spending. These are different markets and measures (Canadian banking, US holiday shoppers, UK supermarkets, US business utilities), so linking attitudes to behaviour is the next research step, not something these studies have already shown.

What to do with this. Measure satisfaction and switching barriers alongside repurchase intent, then link those answers to later behaviour such as switching, cancellations or purchases. Read sentiment beside sales data, and track the promises customers can verify, such as response and restoration times.

Name the test, then run it

Before relying on an AI feature, a synthetic response or a loyalty score, identify what would make it trustworthy and test that directly: a person who can step in, a comparison with real respondents, or behaviour that confirms what customers say. Because these studies differ in samples and methods, and most measure what people say rather than do, treat them as prompts for your own tests, not settled findings. A useful question for any new metric or tool: what would we compare this against to know it is right? Test before you trust.

Further reading

  • Pew Research Center: the share of Americans saying data centers are mostly bad for home energy costs rose from 38% in January to 50% in July to August 2026 (November 11).
  • JD Power: financially healthy borrowers now make up 47% of auto loan customers, but their satisfaction fell 8 points year over year to 735 on a 1,000-point scale, and their trust in their lender fell 8 points to 737 (November 12).
  • JD Power: online mattress buyers now score satisfaction 17 points above in-store buyers (884 on a 1,000-point scale), nearly double last year's 9-point gap, while spending $332 less on average (November 9).

Sources

Moe Pouyan
Next

Beyond the Headline Numbers: Three Gaps in This Week’s Research