Designing explainable ML-based systems for collaborative work in the railways

The opaqueness of machine-learning (ML) based systems is one of the key barriers to overcome in order to fully benefit from these technologies, especially in high-risk operations such as the railways that require strict regulatory oversight. This challenge is exacerbated in the case of collaborative work processes that involve several human operators, possibly in different occupational domains, and several ML technologies. The project addresses the challenge of designing explainable AI within an overarching design framework for socio-technical integration based on the recently proposed concept of networks of accountability. This concept outlines the interdependencies created between technology developers, organizational and individual users, and regulators due to the continuous process of data production and data use in ML-supported decision-making. Networks of accountability permit accountability negotiations between all stakeholders from the onset of a technology development project, which help to establish a shared understanding of control and accountability requirements and how they are best met in the emergent socio-technical system. Explainability of the ML-based technologies is a necessity for operational decisions, but is also relevant in earlier phases of prototyping and in particular for regulatory approval. Given the many different users of these systems, explainability requirements have to cater for different occupational perspectives and knowledge domains. The proposed project will develop methods for extracting these different requirements and for supporting their implementation, using a multi-method approach comprising expert interviews, participant observation, design workshops, and work process simulation. The project will be carried out with several partners at SBB and Siemens, focusing on solutions for application domains in traffic control and operations, inspections, and predictive maintenance. By including a broad range of use cases for the development and validation of the design framework and the design methods, their practical value and generalizability will be assured.