Multimodal Foundation Models in Clinical Triage: Validating EHR and Imaging Fusion
What the Researchers Tested
The research team constructed a cross-attention transformer network that simultaneously ingests raw unstructured emergency physician notes, longitudinal lab trajectories, continuous vital sign telemetry, and 3D non-contrast CT volumes. Rather than processing text and imaging in isolated silos, the multimodal model fuses visual embeddings from spatial vision transformers with tokenized clinical narratives. This architecture allows the algorithm to contextualize subtle radiological findings (such as borderline ground-glass opacities, minor retroperitoneal stranding, or subtle intracranial hyperdensities) against real-time clinical indices like lactic acidosis, leukocytosis, or acute hypoxia. The end-to-end inference takes less than 90 seconds on standard hospital GPU servers.
What the Study Found
Multimodal EHR + 3D CT fusion achieved an overall AUROC of 0.93 (95% CI: 0.91-0.95), vs. 0.74 for NEWS2 score and 0.81 for imaging-only models.
Sensitivity for acute intracranial hemorrhage, occult sepsis, and pulmonary embolism reached 94.2% at a fixed 88.5% specificity.
Reduced time-to-escalation for ICU-level decompensation by an average of 46 minutes (p < 0.001) in simulated triage replays.
Evaluated across a rigorous multi-ethnic cohort of 42,850 patients across 4 tertiary hospital networks.
Why This Matters to Clinical Practice
Emergency departments worldwide operate under unprecedented overcrowding, physician burnout, and acute triage bottlenecks where critical diagnostic clues get lost across disparate systems. Occult sepsis, non-massive pulmonary embolism, and evolving intracranial hemorrhage frequently present with non-specific vitals. By synthesizing temporal EHR trends with immediate imaging scans within minutes of acquisition, this system prevents catastrophic diagnostic delays and gives triage teams pre-test probability estimates before specialist consultation.
Clinical & Workflow Takeaways
Physicians and emergency department directors can deploy multimodal foundation models as continuous background copilots. The system flags occult high-risk trajectories and prepares structured clinical pre-populates without requiring manual data re-entry. However, clinical governance policies must establish that all AI-generated escalation alerts function as advisory decision support: human-in-the-loop sign-off remains mandatory before ordering stat interventions, invasive lines, or emergency surgical consults.
Limitations and What Remains Uncertain
The training and validation cohorts were derived exclusively from academic tertiary centers with fully integrated Epic and Cerner EHR infrastructures and high-speed multi-slice CT scanners. Generalizability across rural community hospitals, lower-resolution imaging suites, and non-standardized multilingual physician dictations requires prospective validation.
Explore the Work Behind Clinical AI
MedXchange Research Fellowship
Publish Clinical AI Research With Us
Our 10-week fellowship mentors physicians and researchers to build foundation models, analyze multimodal health datasets, and co-author peer-reviewed clinical studies.
Related Clinical AI Stories
FAHR-Face: Clinical Foundation AI Models Extract Multi-System Health Biomarkers from Facial Photographs
Recent advances in deep learning have demonstrated that human facial phenotypes encapsulate intricate physiological, metabolic, and biological aging signals. In a landmark study published in The Lancet Digital Health, investigators led by researchers from Mass General Brigham and Harvard Medical School introduced FAHR-Face (Foundation Artificial Intelligence Models for Health Recognition by Use of Face Photographs), a self-supervised and multitask foundation architecture designed to extract systemic clinical biomarkers directly from standard 2D facial photography. By pretraining visual transformers across large-scale facial datasets linked with deep electronic health records, the investigators developed a generalizable computational platform capable of phenotyping across multiple organ systems without reliance on specialized imaging modalities. The study evaluated FAHR-Face across diverse multi-institutional cohorts to benchmark its capacity for predicting chronological age gaps, metabolic dysregulation, cardiorespiratory endpoints, and all-cause mortality. The architecture demonstrated robust zero-shot and few-shot transfer learning, outperforming conventional single-task convolutional networks in identifying latent systemic pathology. This investigation shifts the paradigm of facial biometric analysis from cosmetic or genetic syndromic dysmorphology screening to systemic, longitudinal health risk stratification and passive screening in outpatient and primary care environments.
AI and Digital Technologies Transforming Multiple Sclerosis Neurorehabilitation
Recent advancements in neurorehabilitation for Multiple Sclerosis (MS) are shifting towards earlier, personalized, and technology-supported interventions. While pharmacological treatments have evolved, a significant number of individuals with MS (pwMS) continue to face complex symptom burdens, functional limitations, and reduced quality of life. This review emphasizes the integration of routine, multidimensional screening that extends beyond symptom severity to include functional status, participation restrictions, and social determinants of health (SDoH). The embrace of digital health technologies, such as wearables, remote monitoring, and AI-assisted assessments, offers new avenues for more ecologically valid and accessible evaluation of pwMS. The emerging paradigm posits neurorehabilitation as a fundamental component of comprehensive MS care, advocated for initiation early in the disease course rather than as a late-stage intervention. Exercise programs, cognitive rehabilitation, psychological therapies, and vocational rehabilitation are showing benefits across physical, cognitive, emotional, and fatigue domains, ultimately improving quality of life. Telerehabilitation and technology-supported approaches like virtual reality (VR), exergaming, and neuromodulation are proving instrumental in enhancing access to care and improving patient outcomes, despite ongoing needs for further research into their optimal application and widespread implementation.
Real-Time Ambient Voice LLMs Reduce Diagnostic Discrepancies During Emergency Department Handoffs
A randomized crossover simulation and prospective observational trial published in JAMA Internal Medicine evaluated the impact of ambient clinical voice LLMs on inter-shift physician sign-outs and handoffs across three high-volume urban emergency departments. Patient handoffs between departing and oncoming emergency physicians have long been identified by the Joint Commission as a primary root cause of sentinel diagnostic errors, medication lapses, and critical test result omissions. The ambient LLM listening system generated structured I-PASS handoff dossiers in real time, reducing critical diagnostic omissions by 64% and cutting shift transition duration by 31%.