Doctors preferred AI’s summaries of long hospital stays 95% of the time. Here’s how to test AI on your own files

On Monday, Stony Brook University highlighted a study, published last month in npj Digital Medicine, that put AI up against doctors on a writing job the researchers call burdensome: the discharge summary.

It’s the handoff that tells a patient’s regular doctor what was treated in the hospital, which medications changed, and what still needs follow-up. One hospitalist in the study recalled taking over a patient who had been in the hospital for more than three months. Going through that chart just to write the summary took two hours.

What the study found

The researchers took 60 internal medicine stays at Stony Brook University Hospital lasting 7 to 21 days. An AI model, OpenAI’s GPT-5.2, worked through each record in sections and drafted a summary in a few minutes. Twelve attending physicians, six hospitalists and six primary care doctors, then compared the AI drafts with the summaries written at the time, with two reviewers for each stay.

They preferred the AI’s version for 57 of the 60 stays, or 95%, and rated it higher for quality, readability, accuracy, and completeness. The AI’s edge came mostly from leaving less out. When reviewers marked specific errors, the AI’s summaries had fewer omissions, but there was no significant difference in factual mistakes, or in how much harm the errors could do.

“These results provide evidence that artificial intelligence can support discharge documentation for complex hospital stays,” said Dr. Rachel Wong, one of the study’s authors.

Both versions had errors, and the authors stress that a physician must review and verify any AI draft before it goes into a patient’s record. They also spell out the study’s limits. It was done at one hospital, on past cases. The reviewers knew which summary came from the AI, so expectations may have colored their ratings. And it didn’t measure time saved. The team calls that a next step, along with a possible pilot at Stony Brook Medicine.

Why it matters beyond the hospital

Plenty of businesses turn long files into short handoffs, and the doctors described a problem any of them would recognize: when time is short, people skim long records, and that’s when important details get left out.

Closer to home

Clinics across the Wiregrass deal with handoffs like these, and the same job shows up well beyond health care. Picture an insurance agency in Enterprise summarizing a claim for an adjuster, a Dothan law office boiling a box of records down for an attorney, or a property manager briefing a new hire on a building’s repair history. If AI can draft those in minutes, the question is how you’d know the drafts are good enough to use.

Five ways to test AI on your own files

1. Start with the hard cases. Stony Brook chose long stays because that’s where summarizing is hardest. Pull a few dozen real past files, long ones included, that already have a summary a person wrote.
2. Let the readers grade it. The primary care doctors, who rely on these summaries after a patient goes home, flagged far more omissions than the hospitalists and rated harm as more likely. The person who uses a summary sees what’s missing.
3. Count misses and mistakes separately. Leaving something out and getting something wrong are different problems, and the AI here improved on one but not the other. Track both.
4. Hide which is which. Stony Brook kept both versions in their original formats, so reviewers could tell them apart. Put both in the same format and strip the labels, so nobody grades the AI on expectations.
5. Settle the data question first. Stony Brook ran the model through a private, HIPAA-compliant Microsoft Azure service, and its business associate agreement kept patient records away from another major AI provider’s models. If your files hold patient, client, or financial details, decide where they can go before testing, and keep a person approving every summary.

Then time it. The study didn’t, and that number tells you whether the switch is worth making.

Where we can help

Our workflow automation service builds automations that read documents, move data between your systems, and prepare the reports you need, with checks built in and anything uncertain sent to a person. We track hours and error rates before and after, so you can see what an automation is worth. Have files that take hours to summarize? Book a free strategy call.

Sources

• Stony Brook University: SBU Study: AI May Improve Patient Discharge Summaries, November 2, 2026
• npj Digital Medicine: Clinician-centered evaluation of large language model-generated discharge summaries for longer hospitalizations, October 2, 2026