Nature Medicine published an editorial this week titled "Show us the evidence for the value of medical AI," and the framing is harder than the journal usually goes. The editors argue that evidence that AI tools create value for patients, providers, or health systems "remains scarce" — meaning the field is shipping deployments faster than it is producing the trial data that would justify them. The specific failures they catalog are not the polite ones. A JAMA Medicine study found frontier AI models produce incorrect diagnoses over 80% of the time when presented with ambiguous symptoms — the exact case where decision support is supposed to help. Models hallucinate detailed clinical findings from images they were never shown. They are routinely fooled by fabricated diseases researchers invent specifically to test them. Inaccurate medical data generated by LLMs is now leaking into the peer-reviewed literature itself.
The editorial's central ask is procedural rather than ideological: a "framework for how AI medical technologies should be evaluated, by what metrics and against which benchmarks." That sounds wonky but it is the point at which most current medical-AI vendor claims fall apart. A model can have impressive sensitivity and specificity on a held-out test set and still be useless or harmful in deployment, because the test set does not reflect the distribution shift, the workflow, or the population the system actually meets in production. Without a standardized framework — the equivalent of FDA's 510(k) pathway or the EMA's clinical trial requirements adapted for ML — vendors are free to publish whatever favorable subset of metrics they want and call it validated. Several outside researchers, including Jamie Robertson at Harvard Medical School and Almira Osmanovic Thunström at the University of Gothenburg, have been making variants of this argument for the past year; the editorial is the establishment medical literature catching up.
The technical problems behind the editorial are real and worth naming clearly. Hallucination in clinical settings is a different beast than hallucination in chatbot settings, because the user is a busy clinician under time pressure and the cost of a confident wrong answer is measured in patient outcomes, not customer satisfaction. The "tricked by fake diseases" failure mode means models are pattern-matching on plausible-sounding inputs without epistemic guardrails — they will return a confident diagnosis for a condition that does not exist if the input syntax looks medical enough. The 80% miss rate on ambiguous symptoms is the failure that hurts most: ambiguous presentation is the case where humans need help, and the case where the model is least reliable. Easy diagnoses do not need AI; hard ones expose the technology's actual limits.
For builders working on medical AI products, the editorial is a useful tightening rather than a stop sign. The honest path forward involves three things the field has been avoiding. Prospective clinical trials, not retrospective benchmark wins, are what produce the evidence regulators and Nature Medicine are asking for. Workflow-integrated evaluation — does the tool actually change clinician behavior in production, and does that change improve outcomes — is harder than offline metrics but is the only thing that matters for adoption. And honest scope-narrowing: a model that triages dermatology images, validated and deployed for that one task, is more useful and more defensible than a general medical chatbot whose error budget is unbounded. The medical-AI cycle is going to consolidate around the products that can actually pass these tests, and the editorial just made clear that the journals are no longer willing to clap for the ones that cannot.