AI Safety in Health Systems: Building Infrastructure and Strengthening Risk Management Practices

Summary

Clinical AI is rapidly being deployed across health systems, but existing patient safety frameworks are not always fully equipped to detect or manage the unique risks these tools introduce (which can include bias, performance drift, hallucinations in large language models, and clinician overreliance). AI tool risks are often difficult to identify, compounded by the fact that AI systems are frequently invisible to patients and inconsistently understood by users.

This white paper argues that ensuring safe AI use requires a shift from reactive to proactive, lifecycle-based risk management. Health systems should establish formal AI governance structures with clear accountability, maintain centralized inventories of AI tools, and integrate AI oversight into existing patient safety reporting systems. Risk management must span the full lifecycle of an AI tool. 

We acknowledge that significant gaps exist. There are no widely adopted standards for AI safety in practice, regulatory oversight is fragmented, and many health systems lack the technical expertise and infrastructure needed for effective monitoring. These challenges risk exacerbating inequities in care and create an “AI divide” between well-resourced and under-resourced organizations.

To address these issues, the paper emphasizes the importance of cross-system learning, stronger collaboration with AI vendors, and improved mechanisms for tracking both safety events and near misses. It also calls on policymakers to clarify regulatory expectations, incentivize safety infrastructure, and enable information sharing across institutions. Ultimately, the safe integration of AI into health care depends on building robust governance, monitoring, and accountability systems that ensure innovation improves patient outcomes while upholding the fundamental principle of patient safety.