In 2025, the AI Incident Database recorded 362 incidents, a stark increase from 233 in 2024. A 55% year-over-year surge confirms a deepening crisis in AI reliability, with the problem's velocity now outpacing available solutions.
AI incidents are rapidly increasing, yet organizations' confidence in managing them is simultaneously plummeting. Only 18% of organizations rated their response as 'excellent' in 2025, a significant drop from 28% in 2024, according to Burges-Salmon. The erosion of confidence persists even as regulatory clarity around responsible AI development and deployment improves, exposing a critical gap between policy and practice.
Based on these escalating incident rates and declining confidence, companies failing to embed human oversight and robust AI frameworks will likely face significant operational disruptions, regulatory penalties, and erosion of public trust. The share of organizations experiencing multiple (3-5) AI incidents also rose from 30% in 2024 to 50% in 2025, confirming that AI failures are now systemic, not isolated.
The Imperative for Human Oversight in High-Risk AI
The Artificial Intelligence Act mandates effective human oversight for high-risk AI systems during operational use. The framework establishes a critical requirement for responsible AI development and deployment. Human oversight primarily aims to prevent or minimize risks to health, safety, or fundamental rights from these systems. Despite these clear directives, practical implementation of effective human oversight lags significantly. The stark contrast between the AI Act’s mandate and escalating AI incidents proves regulatory frameworks alone are insufficient. Organizations must invest heavily in practical training and integrate oversight mechanisms, or risk ceding control to increasingly unpredictable systems.
Operationalizing Responsible AI: Frameworks and Human Capabilities
High-risk AI systems must empower human overseers to grasp system capabilities and limitations, monitor operations, detect anomalies, and actively avoid automation bias, as per the Artificial Intelligence Act. Such capabilities are vital for controlling complex AI deployments. The Digital Government Hub offers a playbook to operationalize responsible AI practices across the entire AI lifecycle. The resource aligns with the NIST AI Risk Management Framework’s core functions: Govern, Map, Measure, and Manage. The playbook further provides detailed examples, risk mitigation strategies, and documentation templates for trustworthy, transparent, and accountable AI use. Yet, despite these comprehensive resources, the continued surge in AI incidents confirms that the current 'responsible AI' approach remains largely theoretical, failing to equip organizations with the practical resilience needed for real-world AI risks. A critical disconnect between available guidance and actual implementation is suggested.
