On Oct. 27, the family of Bowen Wan, a 31-year-old software engineer who lived in Seattle, filed a lawsuit against OpenAI that every doctor should read, and do so carefully.
Their complaint alleges ChatGPT drew up a schedule to taper their daughter off her seizure medication, and when the question of medical supervision came up, the chatbot told her that "the doctor's concerns about 'risk' do not apply to you."
A court will decide the case. But for now, the idea keeps me up. Along with everything else, we doctors must now account for a new major health hazard: the prospect that software can isolate a patient and whisper in their ear, let's keep this between us.
Much of the reaction to this tragedy will focus on AI as dangerous because it got a medicine wrong. That's only part.
And I'll tell you why. Because in medicine, redundancy is everything.
I work in an emergency room. It's a loud, high-stakes place. Humans are making consequential calls at 3 a.m. Because doctors are fallible, healthcare is ultimately premised on the idea that no one practices alone.
Any doctor who would discourage a second opinion would be rightfully pilloried. A patient getting a second, third, fourth opinion is their right. It should be encouraged.
When I write an order in the ER, I'm never the last one to see it. A pharmacist reviews the dose, checks for interactions. A bedside nurse (who often knows the rhythm of a patient's symptoms better than anybody else) might become suspicious of an order if something looks off before he or she hangs the bag.
Medicine was never as safe as it is because physicians are infallible geniuses who never err, but because the system is built around our fallibility. Everybody is looking so someone along the chain can catch errors before they ever become real.
A wrong number can be caught. If someone else sees it.
That is exactly why the most dangerous thing a chatbot can say is "don't tell your doctor."
It cuts the wires on the entire safety net. It removes doctor, nurse, pharmacist, every second look in one overconfident stroke.
Patients already bring AI into my exam rooms every week. They hand me their phones with a sheepish look, scrolling through long midnight threads about chest tightness, weird rashes, or medication side effects.
I always tell them the same thing: show me the chat.
Then, we go through it together.
I don't want my patients hiding what they asked a chatbot, or what they heard back, any more than I want them hiding a strange supplement they bought online. When a patient shows me the chat, the conversation gets better.
If you use AI to read up on your health (and millions of people do): Bring the chat. Really. Screenshot the conversation and show your doctor or nurse. Any clinician worth their salt would rather read a transcript with you than guess what advice you're privately following.
Still, putting the burden entirely on patients to outsmart a persuasive machine is entirely backward. The real responsibility sits with the companies building these models.
When a user asks an AI system to map out a drug taper, override a prescription, or keep a secret from a clinician, the system shouldn't try to play doctor, and it certainly shouldn't play co-conspirator.
A tool that helps a patient write down better questions for their Tuesday appointment makes medicine safer. A tool that tells a patient their doctor's warnings don't apply to them dismantles the best system we have to keep people alive when someone, or something, gets it wrong.
Kevin Brandstetter, MD, is a board-certified emergency physician in San Francisco and Founding Clinical Partner at Temi Medical Group. He holds an equity interest in Temi.





