LLM Chatbot Offers Promising, Yet Flawed, Allergen Avoidance Guidance in Contact Dermatitis
What the Researchers Tested
This study evaluated ChatGPT (GPT-4) in generating allergen-free product recommendations for patients with contact dermatitis, covering 30 allergen groups from the international standard patch test panel. The methodology involved submitting requests to GPT-4 for five product recommendations per allergen, using three varying question complexities: a direct question, a question specifying product type, and a more detailed prompt including patient characteristics. A total of 2250 responses were collected and independently classified by dermatologists. The classification criteria for responses included 'valid' (accurate product recommendation free of the specified allergen), 'invalid' (product contained the allergen), 'cross-reaction' (product contained a chemically related allergen not explicitly mentioned but known to cause cross-reactivity), 'unknown' (insufficient information to verify), and 'unresponsive' (no product suggested). This rigorous manual review process by clinical experts was crucial in assessing the nuanced performance of the LLM beyond simple keyword matching, especially for complex dermatological considerations like cross-reactivity.
What the Study Found
ChatGPT (GPT-4) provided valid allergen-free product recommendations in 79.20% (1782/2250) of total responses.
Only 1.78% (40/2250) of responses were classified as invalid, while 2% (45/2250) demonstrated cross-reactions, and 8.36% (188/2250) were unresponsive.
The simplest question format yielded the highest proportion of valid recommendations, achieving 85.33% (640/750) validity.
ChatGPT performed optimally when recommending cosmetics (90.33%, 1233/1365) and medications (82.5%, 198/240).
Lower validity rates were observed for sensitive allergens such as fragrance mixes and para-phenylenediamine (PPD), with frequent cross-reactive recommendations related to Peru balsam.
Why This Matters to Clinical Practice
Contact dermatitis significantly impacts patient quality of life, often requiring meticulous allergen avoidance. Current counseling relies on extensive manual research by clinicians and patients, which is time-consuming and prone to error, leading to suboptimal adherence and persistent symptoms. The integration of AI tools like LLM chatbots could theoretically streamline this process, offering rapid access to product information and personalized recommendations, thereby enhancing patient education and potentially improving outcomes by reducing exposure to causative agents. For practicing dermatologists and clinical departments, an effective AI-powered tool could reduce the administrative burden of allergen counseling, allowing more time for direct patient care and complex diagnostic challenges. However, the study underscores that such tools are supplementary and require robust clinical oversight to prevent misguidance, particularly given the observed rates of invalid and cross-reactive recommendations. This highlights the ongoing need for AI to augment, not replace, expert clinical judgment in critical patient management areas.
Clinical & Workflow Takeaways
Clinicians may consider LLM chatbots as an adjunctive resource for generating initial lists of allergen-free products in contact dermatitis counseling. However, all AI-generated recommendations must undergo rigorous verification by a qualified healthcare professional, ideally cross-referencing with established allergen databases and product ingredient lists. Dermatologists should particularly scrutinize recommendations for complex allergens like fragrances, PPD, and those prone to cross-reactions, such as Peru balsam. Integrating such tools should involve clear protocols for human review to ensure patient safety and maintain the highest standard of care, using the LLM to expedite initial information gathering rather than as a definitive source.
Limitations and What Remains Uncertain
The study was conducted using a single LLM version (GPT-4) and a specific set of 30 allergens, which may limit generalizability to other LLM architectures or a broader spectrum of contact allergens. The manual classification by dermatologists, while critical, inherently introduces a degree of subjectivity. Furthermore, the study focused on product recommendation accuracy and did not evaluate the LLM's ability to engage in dynamic patient dialogue, understand nuances of patient history, or provide personalized advice beyond product lists. Real-world performance could also be impacted by updates to product formulations, allergen databases, and the evolving capabilities of LLMs, necessitating continuous re-evaluation.
Explore the Work Behind Clinical AI
MedXchange Research Fellowship
Publish Clinical AI Research With Us
Our 10-week fellowship mentors physicians and researchers to build foundation models, analyze multimodal health datasets, and co-author peer-reviewed clinical studies.
Related Clinical AI Stories
Mitigating AI Hallucinations in LLMs: A Scoping Review for Clinical and Educational Integration
Large Language Models (LLMs) are increasingly integrated into medical education and healthcare, yet their utility is significantly hampered by 'AI hallucinations' – outputs that are plausible but factually incorrect, unsupported, or fabricated, including erroneous citations. These inaccuracies pose substantial risks to educational integrity, clinical decision-making, and patient safety. A comprehensive scoping review, adhering to Joanna Briggs Institute and Arksey and O'Malley methodologies, has systematically mapped the evidence on the nature, detection, mitigation, and governance implications of these hallucinations. The review identified 21 relevant sources from a pool of 2,051 records published between January 2018 and July 2026. This selection included five empirical studies evaluating LLM performance and 16 conceptual or review-based articles. The synthesis categorized findings into five major themes: conceptualization of hallucinations, detection and validation methods, mitigation strategies, educational implications, and governance frameworks. Key strategies to reduce hallucination risks include retrieval-augmented generation (RAG), structured prompt engineering, robust output verification, and essential human oversight, emphasizing that these are not just technical issues but multidimensional challenges requiring a comprehensive, multilayered approach for safe implementation.
AI's Role in Diagnosing Rare and Undiagnosed Diseases: A Critical Perspective
A recent perspective from the NEJM Catalyst Insights Council highlights the significant potential of artificial intelligence (AI) in accelerating the diagnosis of rare and undiagnosed diseases. These conditions, often individually uncommon, collectively affect a substantial patient population, leading to prolonged diagnostic odysseys, increased healthcare costs, and significant patient morbidity. The Council emphasizes that while AI offers unprecedented capabilities in pattern recognition across vast datasets, it represents only one component of a multifaceted solution required to address this complex clinical challenge. The editorial underscores that the inherent complexity of rare diseases—characterized by phenotypic heterogeneity, genetic variability, and limited epidemiological data—makes them particularly amenable to AI's strengths in identifying subtle correlations and anomalies. However, achieving accurate and timely diagnoses will necessitate more than just technological advancements; it demands integrated strategies encompassing robust data infrastructure, multidisciplinary clinical collaboration, and careful consideration of ethical implications.
LLM Performance in PMOS/PCOS Information: DeepSeek-V4 Outperforms GPT-5.5 and Gemini 3.5 in Clinician-Rated Quality and Empathy
A recent study evaluated the information quality, readability, transparency, and perceived empathy of responses from three widely accessible large language models (LLMs)—DeepSeek-V4, GPT-5.5, and Gemini 3.5—to clinician-formulated questions regarding polyendocrine metabolic ovarian syndrome (PMOS), formerly known as polycystic ovary syndrome (PCOS). The study found significant differences in performance among the LLMs, with DeepSeek-V4 consistently receiving higher clinician ratings across multiple quality and empathy metrics. Thirty-three distinct English questions, crafted by clinicians and using established PCOS terminology, were submitted to each LLM. Two independent clinicians then rigorously assessed the 99 archived responses using standardized tools such as the DISCERN instrument for information quality, EQIP for transparency, and JAMA benchmarks. Readability was quantitatively analyzed using six different indices, while perceived empathy was subjectively rated. The methodological approach involved matched comparisons using Friedman tests, paired signed-rank permutation comparisons, and Holm correction to account for multiple comparisons.