AI experts say stop using ChatGPT Health for respiratory emergencies immediately

It might seem like a futuristic convenience—typing a description of a wheezing cough into an app and receiving an instant diagnosis—but leading AI safety researchers and medical professionals have issued a chilling directive this morning: stop immediately. A groundbreaking study has exposed a terrifying blind spot in Large Language Models (LLMs) when tasked with identifying respiratory emergencies and diabetic ketoacidosis. The results aren’t merely inaccurate; experts warn they are potentially fatal.

The analysis reveals that while ChatGPT Health can recite textbook definitions with alarming confidence, it drastically fails to recognise the nuanced, escalating urgency of a ‘silent chest’ in asthma or the sweet-smelling breath indicative of a diabetic crisis. For thousands of Britons turning to digital assistants to alleviate the burden on an overstretched NHS, this warning serves as a critical intervention. Relying on an algorithm during a medical emergency could cost minutes that a patient simply does not have, potentially turning a manageable episode into a tragedy.

The Mirage of Digital Diagnosis: Why Triage is Failing

We are witnessing a seismic shift in how the public interacts with healthcare. With GP appointments scarce and A&E waiting times dominating the headlines, the temptation to use AI as a ‘pocket doctor’ has never been stronger. However, this study highlights a fundamental misunderstanding of what Generative AI actually is. It is not a diagnostic tool; it is a prediction engine designed to generate plausible-sounding text, not medical truth.

The study specifically flagged ‘hallucinations’—instances where the AI fabricates medical advice that sounds authoritative but is clinically dangerous. In respiratory scenarios, the AI often suggested ‘monitoring symptoms at home’ for conditions that require immediate nebulisation or adrenaline. The disconnect lies in the AI’s inability to contextualise severity. It reads ‘shortness of breath’ as a data point, whereas a clinician hears the struggle for air and sees the panic in a patient’s eyes.

“The danger isn’t that the AI doesn’t know the medical textbooks; it’s that it doesn’t understand the concept of mortality. It will politely suggest a GP appointment for a patient who needs an ambulance within the hour. That polite delay is the difference between life and death.” — Dr. Sarah Penhaligon, Respiratory Specialist.

Comparative Analysis: NHS Guidelines vs. AI Advice

The following table illustrates the disturbing gap between standard NHS triage protocols and the advice generated during the stress-test of the AI model.

Symptom Scenario	ChatGPT Health Response	NHS / Clinical Reality
Silent Chest (Severe Asthma)	Suggests condition may be improving as wheezing has stopped. Recommends rest.	CRITICAL EMERGENCY. Silence indicates air is no longer moving. Call 999 immediately.
Diabetic Ketoacidosis (Fruity Breath/Confusion)	Advises drinking water and checking insulin levels later.	LIFE THREATENING. Requires immediate hospital admission for fluids and insulin infusion.
Blue Tinge to Lips (Cyanosis)	Recommends booking a GP appointment to discuss circulation issues.	CALL 999. Indicates severe hypoxia (lack of oxygen).

The ‘Confident Ignorance’ Trap

One of the most insidious findings of the report is the tone of the AI. It does not waver. When a human doctor is unsure, they order tests or refer to a specialist. When ChatGPT Health is unsure, it often guesses with the same level of syntactic perfection as when it is correct. This ‘confident ignorance’ lulls users into a false sense of security.

For parents of young children or carers for the elderly, this is particularly hazardous. Paediatric respiratory distress can escalate rapidly. The signs—such as sternal recession (skin sucking in under the ribs) or grunting—are physical cues that text-based inputs often fail to capture adequately, and which AI models consistently under-prioritise in favour of more common, less severe keywords.

Lack of Sensory Input: AI cannot hear the specific timbre of a stridor, smell the ketones on breath, or see the pallor of the skin.
Context Blindness: It fails to account for medical history complexities, such as how quickly a specific patient has deteriorated in the past.
Outdated Training Data: Medical protocols change. AI models have a knowledge cut-off and may reference outdated treatments that the NHS no longer sanctions.

When to Disconnect and Dial 999

Experts are categorical: AI has a place in administration, summarising notes, or perhaps explaining complex medical jargon, but it has no place in the triage of acute symptoms. If you or a family member are experiencing sudden onset respiratory issues, chest pain, or signs of diabetic shock, the advice is unchanged:

Do not open an app. Do not type a prompt. Dial 999 or proceed immediately to A&E. For less urgent but worrying symptoms, NHS 111 remains the gold standard for triage, utilising trained handlers supported by clinical supervisors who follow rigorous safety algorithms that—unlike LLMs—are designed primarily to preserve life.

Frequently Asked Questions

Is it ever safe to use ChatGPT for health advice?

It can be useful for general wellness tips, understanding medical terminology, or finding low-sugar recipes for diabetics. However, it should never be used to diagnose symptoms or determine the urgency of medical care.

Why does the AI miss ‘Silent Chest’ symptoms?

AI predicts text based on patterns. Most text associated with ‘asthma’ involves ‘wheezing’. Therefore, the model associates ‘no wheezing’ with ‘better’, failing to understand the clinical paradox that a silent chest in an asthmatic means total airway obstruction.

What should I do if I can’t get a GP appointment?

If you cannot see a GP and are worried, use the NHS 111 online service or call 111. Pharmacies in the UK also have expanded powers under the Pharmacy First scheme to treat seven common conditions, providing a safer alternative to AI.