THE ARISING · Longevity & Healthspan
Accurate AI Health Answers May Still Steer People Toward Worse Choices
AI chatbots are now judged mostly on one question: is the answer correct? A new peer-reviewed evaluation shows how well the leading systems score on that test, while a separate argument in STAT says correctness may not be enough to protect health.
The evaluation, published in Nature Medicine in 2026 by Vishwanath and colleagues, is a benchmark comparison of AI tools, not a trial in patients. It tested two specialized clinical tools, OpenEvidence and UpToDate Expert AI, against three general-purpose models: GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6. It used 500 medical-exam questions, 500 items from a benchmark called HealthBench, and 100 real queries that physicians sent to a general-purpose model in a live clinical setting at NYU Langone Health.
The general-purpose models won every stage. On the exam questions, Gemini scored 97.4%, GPT 94.2% and Claude 90.2%. OpenEvidence scored 89.6% and UpToDate 88.4%. On HealthBench, GPT scored 88.0 out of 100, against 62.6 for OpenEvidence and 61.3 for UpToDate.
The real-query test carries the most weight, according to the authors. Twelve US clinicians reviewed the answers blind, producing 1,800 ratings of correctness, completeness, safety and clarity on a 1–4 scale. Refusals were excluded from the scored comparisons, leaving 568 responses and 1,704 ratings. Gemini averaged 3.62, GPT 3.54 and Claude 3.52. OpenEvidence averaged 3.24 and UpToDate 3.17. Google's automatic AI Overview, which many people see first in search, averaged 3.27.
Tests did not detect statistically significant differences in harmful-content or hallucination rates; this does not establish equal safety. OpenEvidence scored lowest on clarity, at 2.84. UpToDate's tool declined 19% of queries, against 1–3% for most others.
The tested systems scored highly on these medical benchmarks. The authors add that knowledge and communication benchmarks may not fully capture clinical risk. These results describe the tested versions, not a permanent ranking; possible benchmark contamination and model-judge bias limit generalization.
That gap is the subject of an opinion piece in STAT's First Opinion section by Jessica Fishman and John L. Jackson Jr. of the University of Pennsylvania. They argue that an answer can be factually right and still change behavior for the worse.
The opinion describes potentially counterproductive wording, not a randomized demonstration that these particular responses changed vaccination behavior. Their example comes from a study they describe, conducted with the Vaccine Education Center at Children's Hospital of Philadelphia. ChatGPT, Gemini and Claude were asked about chickenpox parties. According to the authors, the answers were highly accurate and ended by recommending vaccination. But all three opened by calling the parties "popular" or "common," and the authors say research shows that describing a behavior as common can encourage it. They also say ChatGPT's description of vaccination as "more socially responsible" is a type of appeal that evidence suggests may fail to raise vaccination.
The same logic applies to smoking, they write. A chatbot could accurately say that most smokers never develop lung cancer, which may reassure a smoker and reinforce the habit. Another could accurately stress the risks and point to effective ways to quit.
Fishman and Jackson propose "behavioral benchmarking." Randomized experiments would test whether AI answers make people more or less likely to protect their health. They would start in areas of strong scientific consensus, such as smoking cessation and childhood vaccination, with results published so hospitals, insurers and consumers can compare models. This is a proposal, not a tested standard.
Next to watch is whether developers or independent groups publish randomized tests of how chatbot answers shape health intentions, alongside accuracy evaluations.