Operationalizing AI Governance in Healthcare: A Qualitative Study of Staff Perspectives
What the Researchers Tested
This qualitative study employed a grounded theory approach to understand healthcare staff perspectives on AI governance within a large Australian healthcare organization spanning three hospital campuses and community services. Twenty-three semi-structured interviews, lasting between 30 and 70 minutes, were conducted with a diverse cohort including strategic leaders, clinical implementers, researchers, governance and regulatory personnel, operational staff, legal oversight, and consumer engagement representatives. Two independent investigators coded the interview transcripts, and themes were iteratively developed and refined through research team consensus. The study elucidated that staff conceptualize AI governance as an ongoing, organization-wide function rather than a one-time approval step. The analytical framework derived from participant input highlighted five critical domains for effective operational AI governance: systematic risk assessment tailored for AI, robust infrastructure and data governance, comprehensive ethical frameworks (including consent, fairness, and transparency), fostering internal capability and a supportive culture, and establishing dedicated governance structures with clear oversight responsibilities. The current processes were consistently described as ad hoc and insufficient for the complex demands of AI implementation.
What the Study Found
AI governance is perceived as a continuous, multifaceted organizational function, not a single approval step.
Five critical, interrelated domains for effective AI governance were identified: risk assessment, infrastructure and data, ethics, capability and culture, and governance structures.
Existing governance processes were largely described as fragmented, ad hoc, and inadequately tailored to the unique ethical, technical, and operational challenges of AI.
Participants emphasized the need for AI-specific risk assessment pathways, enhanced lifecycle monitoring, stronger data governance, and clearer approaches to consent, fairness, and transparency.
There is a clear demand for greater internal capability, robust system navigation support, and dedicated governance structures with defined oversight responsibilities for AI initiatives.
Why This Matters to Clinical Practice
For practicing clinicians, department heads, and hospital leadership, the effective governance of AI is paramount to realizing its potential benefits while mitigating significant risks. The absence of clear, comprehensive AI governance can lead to diagnostic errors, compromised patient safety, data breaches, and ethical dilemmas, thereby eroding trust and hindering AI adoption. Fragmented or inadequate governance pathways can cause delays in deploying beneficial AI tools, as clinical teams navigate uncertain regulatory and ethical landscapes. This study underscores that a proactive, structured approach to AI governance is vital for ensuring AI systems are developed, validated, and deployed responsibly. Addressing the identified gaps in risk assessment, data integrity, ethical guidelines, and organizational readiness is crucial for safeguarding patient outcomes, optimizing clinical workflows, and ensuring that AI innovations genuinely enhance healthcare delivery without introducing unforeseen liabilities or operational bottlenecks.
Clinical & Workflow Takeaways
Clinical departments and hospital leadership should proactively develop and implement AI-specific governance frameworks that span the identified five domains. This includes establishing dedicated committees or working groups responsible for AI oversight, defining clear AI-specific risk assessment protocols, and ensuring robust data governance strategies are in place for AI model training and deployment. Clinicians involved in AI projects require dedicated training on AI ethics, data privacy, and model interpretability. Regular audits of AI system performance and compliance should be integrated into standard operational procedures to ensure ongoing safety and efficacy. Collaboration between clinical, IT, legal, and ethical committees is essential to build a unified and effective AI governance strategy.
Limitations and What Remains Uncertain
This qualitative study was confined to a single large Australian healthcare organization, which may limit the generalizability of its findings to diverse healthcare systems internationally or smaller practices with different operational structures. The perspectives gathered represent staff views within that specific context, and variations in resource availability, regulatory environments, or organizational culture elsewhere could lead to different governance challenges or priorities. While a grounded theory approach was used, the inherently subjective nature of qualitative data means interpretations can vary. Future quantitative research or studies across multiple, varied institutions would strengthen the generalizability and provide broader insights into operationalizing AI governance.
Explore the Work Behind Clinical AI
MedXchange Research Fellowship
Publish Clinical AI Research With Us
Our 10-week fellowship mentors physicians and researchers to build foundation models, analyze multimodal health datasets, and co-author peer-reviewed clinical studies.
Related Clinical AI Stories
FDA Digital Health Update: Guidance on Lifecycle Management for Adaptive Generative AI Devices
The US Food and Drug Administration (FDA) Digital Health Center of Excellence published revised draft guidance establishing rigorous Pre-Determined Change Control Plans (PCCPs) for generative AI and continuously learning machine learning algorithms in clinical decision support software (SaMD).
Prague 2025 Consensus: New Reporting Standards for AI-Driven Diagnostic Performance
The Prague 2025 consensus introduces a critical new checklist for reporting indirect outcome studies of Model 1b Analytical Performance Specifications (APS), a framework designed to evaluate the clinical impact of diagnostic test performance. This initiative addresses a significant gap in methodological guidance for studies assessing how analytical variations in diagnostic tests influence clinical classifications and subsequent treatment decisions. While Model 1b APS indirectly assesses health outcomes, it provides a crucial intermediate step to ensure diagnostic accuracy aligns with patient care needs, especially in the rapidly evolving landscape of AI-augmented diagnostics. The developed checklist, a result of expert workshops and collaborative meetings, comprises 17 essential items. Importantly, it integrates and cross-references 25 items from the 2025 STAndards for Reporting Diagnostic accuracy studies using Artificial Intelligence (STARD-AI). This integration underscores the growing intersection of laboratory medicine standards and AI validation. The aim is to establish robust, scientifically valid, and transferable methodologies for Model 1b APS, ensuring consistency and reliability across diverse healthcare settings and applications that increasingly incorporate AI.
FDA Issues Expanded Final Guidance on Lifecycle Management for Adaptive Generative AI Diagnostics
The US Food and Drug Administration (FDA) Digital Health Center of Excellence has released an expanded final guidance framework detailing regulatory clearance pathways for adaptive generative AI and multi-agent clinical decision support systems. Departing from static 'locked-box' algorithm clearances, the new guidance establishes formal Predetermined Change Control Plans (PCCPs) that permit continuous learning models to update weights based on localized hospital EHR data. The regulatory update is hailed by health informatics leaders as the most consequential modernization of medical device oversight in over a decade, establishing clear safety guardrails for ambient clinical copilots and autonomous diagnostic triage systems.