Design for values and operator roles in sociotechnical systems

Abstract

Engineering is increasingly of systems that are not only complex – in being multilayered – but also hybrid – in containing people as components. This chapter does not discuss the already well-researched safety concerns generated by this development with respect to operators, users, and bystanders but instead addresses values of relevance to the presence in such systems of operators as agents: values associated with what we can reasonably ask people to do and what we make people responsible for. From the perspective of design, systems containing people as components deviate in four ways from traditional systems consisting of all-hardware components. (1) Such systems are not designed as a whole but gradually and modularly. (2) They are typically delivered as incomplete because their operators have to be added by the client or owner during implementation; only operator roles are strictly speaking designed. (3) Persons performing these roles become components of the system only partially; unlike hardware components their behavior continues to be monitored and controlled from a perspective external to the system. (4) Operator roles are often explicitly conceived as being partially external, in that an operator’s task is ultimately to ensure that the system continues to function properly come what may. These features lead to conflicts with the autonomy of operators as persons and the wellfoundedness of assigning particular responsibilities to them. The difficulties described here should make us rethink whether traditional engineering approaches to system design are adequate for such hybrid systems.