Examining the effects of power status of an explainable artificial intelligence system on users’ perceptions

Contrary to the traditional concept of artificial intelligence, explainable artificial intelligence (XAI) aims to provide explanations for the prediction results and make users perceive the system as being reliable. However, despite its importance, only a few studies have investigated how the explanations of an XAI system should be designed. This study investigates how people attribute the perceived ability of XAI systems based on perceived attributional qualities and how the power status of the XAI and anthropomorphism affect the attribution process. In a laboratory experiment, participants (N = 500) read a scenarios of using an XAI system with either lower or higher power status and reported their perceptions of the system. Results indicated that an XAI system with a higher power status caused users to perceive the outputs of the XAI system to be more controllable by intention, and higher perceived stability and uncontrollability resulted in greater confidence in the system’s ability. The effect of perceived controllability on perceived ability was moderated by the extent to which participants anthropomorphised the system. Several design implications for XAI systems are suggested based on our findings.