Generative AI in Safety-Critical Systems: Risks of Sycophancy, Hallucination, and Trust Degradation

The integration of advanced AI language models like AI Co-pilots into workplace health and safety (WHS) and risk management for sociotechnical systems efficiency gains; however, it may raise critical challenges. This paper focuses on key pitfalls of using AI in safety-critical decision support, emphasizing problems of sycophancy (the AI’s tendency to agree with user assumptions), hallucination (generation of inaccurate or non-existent information), and trust erosion (reduced confidence in AI outputs due to inconsistency, opacity, or past errors). We provide context on how AI is being considered for tasks such as hazard identification and risk analysis, then outline a methodology to evaluate AI limitations using expert oversight and performance metrics. The results, drawn from emerging research and case analyses, indicate that AI tools can produce misleading or biased safety guidance without proper checks, undermining trust in high-stakes environments. In discussions, we explore why these issues occur, from the predictive, pattern-based nature of language models to human factors like automation bias and examine implications for safety professionals. Finally, we conclude that AI systems should serve strictly as decision-support tools rather than autonomous decision-makers in WHS. Rigorous human-in-the-loop strategies, critical oversight, and adherence to safety science principles are essential to harness AI’s benefits without compromising workplace safety.