LLMs for Atrial Fibrillation Patient Education: Cardiologist vs. Non-Medical Professional Perceptions of ChatGPT and Google Gemini Quality

This pilot study employed a straightforward comparative analysis design. The core methodology involved submitting a standardized clinical prompt, "Explain atrial fibrillation and how to treat it to a patient," to two prominent large language models (LLMs): ChatGPT and Google Gemini. The generated text responses were then disseminated to a diverse participant group via a Google Forms survey. The survey instrument was meticulously crafted, incorporating items derived from established tools for assessing patient education materials, specifically the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) and the DISCERN criteria. These criteria encompass aspects such as reliability, clarity, and organizational structure. Participant evaluation involved rating each aspect on a 5-point Likert scale. The study cohort consisted of 28 individuals, segmented into two distinct groups: medical professionals (cardiologists and cardiology fellows, n=11) and non-medical professionals (college graduates not employed in the medical field, n=17). Statistical analysis, utilizing the Wilcoxon signed-rank test, was performed to compare mean ratings for various quality indicators between the two AI platforms and across the different participant groups, thereby identifying significant perceptual differences in the assessment of AI-generated educational content.

The increasing integration of AI into healthcare necessitates a rigorous evaluation of its utility, particularly in patient education. Accurate and comprehensible patient understanding of conditions like atrial fibrillation is paramount for adherence to treatment, lifestyle modifications, and early recognition of complications such as stroke and heart failure. LLMs offer a scalable and accessible means to disseminate health information, but their efficacy hinges on the quality and contextual appropriateness of their output. This study highlights a critical disconnect: what medical experts deem 'quality' in health information (e.g., accuracy, bias, consequence explanation) differs substantially from what laypersons prioritize (e.g., structural clarity, visual aids). For practicing clinicians and department heads, these findings underscore the importance of not solely relying on AI-generated content without clinical oversight and customization. While LLMs can provide a foundational draft for patient education, specialists must critically review and adapt such materials to ensure accuracy, address potential biases, and frame information in a way that resonates with patient learning styles and priorities. Overlooking these nuanced differences could lead to suboptimal patient comprehension, reduced adherence, and potentially adverse health outcomes, thereby undermining the very goal of leveraging AI for improved patient engagement.

Clinicians, particularly cardiologists and those involved in patient education, should view AI-generated content as a starting point rather than a final product for patient materials. While LLMs can efficiently produce initial drafts, a thorough review is crucial to ensure clinical accuracy, unbiased presentation, and clear articulation of treatment consequences. When developing patient education resources, it is advisable to involve both clinical experts and patient representatives (or those with similar non-medical backgrounds) in the review process. This multidisciplinary approach can help balance clinical rigor with user-friendliness, addressing both the informational needs identified by medical professionals and the structural preferences highlighted by non-medical individuals. Department heads should consider implementing guidelines for the use of AI in patient education, advocating for human-in-the-loop validation to maintain high standards of patient communication and mitigate potential misunderstandings.

This was a pilot study with a relatively small sample size (n=28), which limits the generalizability of the findings. The evaluation was based on a single prompt and specific versions of the AI models (ChatGPT and Google Gemini), meaning results may not apply to other LLMs, different prompts, or future iterations of these platforms. The assessment relied on self-reported perceptions via a Likert scale, which can be subjective. Furthermore, the study did not evaluate the actual impact of these AI explanations on patient comprehension, retention, or clinical outcomes, focusing solely on perceived quality. Future research needs to involve larger, more diverse cohorts and longitudinal studies to assess real-world efficacy and long-term effects on patient behavior and health status.