035$
Munich Personal RePEc Archive
ASSESSING HUMAN AND
TECHNOLOGICAL DIMENSIONS IN
VIRTUAL TEAM’S OPERATIONAL
COMPETENCES
Sampaio, Jose´ and Moniz, Anto´nio
IET, UNL-FCT
September 2007
Online at http://mpra.ub.uni-muenchen.de/6942/
MPRA Paper No. 6942, posted 31. January 2008 / 22:46

ASSESSING HUMAN AND TECHNOLOGICAL DIMENSIONS IN VIRTUAL TEAM’S OPERATIONAL COMPETENCES José João Martins Sampaio (jjs@fct.unl.pt) António Brandão Moniz (abm@fct.unl.pt) IET- Centro de Investigação em Inovação Empresarial e do Trabalho Faculdade de Ciências e Tecnologia – Universidade Nova de Lisboa Abstract Cognitive task automation may lead to over trust, complacency and loss of the necessary work environment situation awareness. This is a major constraint in complex work organizations teamwork, ending up into an operational gap, between system developments and its understanding and usability, by operators. This document presents a summary of the main results of author’s research on operational decision processes and occupational competences, applied to the air traffic control operational reality. Introducing a human/technological complementary approach to virtual team’s conceptualisation, the results show there is a dimension to be followed in human/machine integration, which stands beyond interface design, and calls for a deeper human comprehension of technological agent’s structure and functionalities, which will, ultimately, require the development of an operational cognitive framework, where work processes and technological behaviour are integrated in professional competences, as he two faces of the same coin. Keywords: automation; situation awareness; work organization; teamwork; decision process; occupational competences; human/machine interface JEL classification: C92; D81; J20; L86 1. INTRODUCTION The concept of virtual organisation (VO) integrates a new form of work organization and design, as a set of networks electronically linked by a complex IT system. As a consequence, “virtual teamwork” emerges as the natural work organisation, arguing Maher and Gu (2002) that virtual environments (VE) design should stimulate the development of cognitive maps to orient, work, collaborate and navigate in the respective spaces. But, in spite of the research effort in CSCW and the development of powerful groupware, its real use does not generally meet the expectancies. This gap between IT tools and its usability, calls for a deeper insight of VO work organisation and design, in a human and technological dimension integrative perspective. In fact, traditional Human Factors approach has not contributed much, to a better knowledge of the complex interactions supporting organisational, social and technological issues. Rather, it may contribute to a critical unbalance between those dimensions, ending up in an over-trust posture towards the available technology (Sampaio and Guerra, 2004). Because of these constraints, the context of use assumes a great importance in VE, especially when referring to dimensions like functionality and structure. Looking for a solution to this problem, Williams and Harrison (1999) are concerned to identify which techniques that have been developed for interactive systems in general, can be applied to virtual environments. We take a different/complementary approach to this question. Based on the results of our recent research (Sampaio, 2007) we argue that technical system agent’s behaviour is a fundamental issue for the definition of a required usability context, which may only be achieved if a Human Factors based technical system development is balanced against a Technological Factors based human development.

For this purpose, a virtual team’s conceptualization is developed, and will support further construction of socio-technical operational competences, which, in turn, allow for the definition of a relationship framework between human and technological dimensions, so as the whole system may evolve in a coherent and sustainable dynamics. Finally, the results are discussed, leading to the identification of new professional competences and training strategies in a learning organization culture perspective. This paper presents the main results of this research. 2. VIRTUAL TEAMS CONCEPTUALIZATION 2.1 General Although “Human in the Loop” strategies are often referred as the possibility to meet and integrate the human dimension in system design and development, the truth is that this approach has mainly resulted in the subordination of human agents, being technological agents assumed, in the operational context, as the more competent element (partner) of the productive process. The question is that human agents operation can no longer comply, alone, with market quantitative and qualitative demands, in a global economy, being automation and automation networking a common operational strategy, in developed economies. This seems to be the reason why the Air Traffic Management Strategy for the years 2000+, developed by the European Organisation for the Safety of Air Navigation – Eurocontrol – states that 1) «Reliance on the human element alone will lead to a critical imbalance between capacity and demand within a few years» (Eurocontrol, 2000, § 2.3.3.). Implying that automation will be the solution for complementing human performance and skill limitations, this statement calls for a deep insight of Human/Machine interaction, in environments where the cognitive dimension represents the most important component of the operational process. Referring to the case of the Air Traffic Control Services, the International Civil Aviation Organisation (ICAO) states that 2) «Air Traffic Controllers [·...] will apply their best judgement in solving emergency situations» (ICAO, 2002, Part III, § 16.1.1). This determination of ICAO shows that, besides the many theoretical approaches and operational reality simulations, it has not been possible, so far, to translate into the technological system the complexity of the cognitive dimension of human agents and its implications in the operation of complex working environments. The ambiguity resulting from the de-codification of what is meant by“… their best judgement” seems to reaffirm the conviction that human cognition complex nature must not be ignored, if the intention is to develop technological systems and implement operational environments, that are not only efficient (economic vector) but essentially effective, which, in complex work environments very often means safety. The conclusion is that, independently of the approach dimensions – sociologic, psychological, ergonomics – to systemic integration of human and technological agents, it is commonly agreed that results obtained by the development of complex working systems, in a strictly technological dimension of human/machine integration – Human Factors Approach – have not met the initial expectations. 2.2 Socio-Technical Networks Previous arguments show that Human Factors approach to systemic integration supports a two-fold nature of cause and effect of the above mentioned Human in the Loop systemic approach (see statement 1, above): - From the operational management point of view, it represents human agent’s incapability to understand and manage the all amount of available information, thus implying the

automation of a growing set of either strictly operational or complex dimension cognitive tasks; - From the systemic development point of view, it represents the necessity to frame and model human behaviour in algorithms of a greater complexity, aiming to achieve and maintain the necessary balance between human and technological; However, may (or shall) Human/Machine interaction be restricted to a Human Factors dimension? Assuming this question as a main concern, our research introduces a (new) Technological Factors concept, which, in the context of the problematic we having been referring to, also assumes a two-fold nature, complementary to the one already mentioned, about Human Factors approach (see statement 2, above) . - From the operational management point of view, it represents the necessity to promote human knowledge of technological agent’s structure, functionalities and behaviour in a dimension that uses technology as a catalyst element in human agent’s valuation, opposing to neo-tayloristic task automation; - From the systemic development point of view, it represents the incapacity to frame and model human nature demens dimension as the support of cognitive processes in complex working environments. Away from virtual organisation’s concept of virtual team, this Human/Technological factors balanced approach – see figure 1 - is the key element of our virtual team’s conceptualisation. In fact, our approach refers to Socio-Technical Networks - either VO or any other complex working system - being characterized by the integration of human and technological dimensions, at the same operational level as a unique unity of work, thus, “freeing” human/machine interaction from a vision where human nature appears as a constraint and, in that sense, possible to be technologically “solved” as any other operational problem –Human Factors perspective. Virtual teams are then composed by human and technological agents, which, integrating Socio-Technical Networks require, as in any other team, a solid understanding of each other’s nature and working capacities and behaviour (Human Factors versus Technological Factors). Figure 1 – Human Factors or Technological Factors? This Socio-Technical awareness of being an element of a virtual teamwork, represents the key of human agents higher level of systemic integration, and identifies operators capacity to obtain, process, integrate and disseminate information necessary to maintain situation awareness, according to the understanding of the operational context and reality, i. e., according to operators own model. Thus, when groupware usability comes to discussion and evaluation, this approach represents a change in the traditional human centred paradigm, and reflects modern work organization contexts, where a great diversity of human and technological agents are co-located at the same operational level, ultimately requiring for the human agent, the development of a two dimension occupational competences cognitive framework – mental model and situation awareness - each one of them integrating work and technological components , as if they were the two faces of the same coin – see figure 2.
Human Factors Technological Factors Operational Management Incapacity to manage all available information. Necessity of cognitive integration of systemic and technological dimensions. Systemic development Necessity to frame human cognitive dimension
Incapacity to frame and model human nature demens dimension

Figure 2 – Occupational Competences Cognitive Framework 3. OCCUPATIONAL COMPETENCES ASSESSMENT. Bellier (2002) points out that besides on going theoretical debate, there is a consensus on competence conceptualization, which as been built up around five key ideas: 1 - Competence is individual and not working post or organizational centred; 2 - Competence always results from an action, thus it is connected to a activity in real time; 3 - Competence is highly contextualized: to be competent in universe A, does mechanically guaranty the same competence in universe B; 4 - Competence is assumed at different individual levels from instrumental to cognitive and behavioural; 5 - Competence is always the result of different action combination; it cannot be reduced to a specific knowledge or to an isolated action.
Occupational competences emerge than as a complex concept that cannot be reduced to a specific context, an isolated knowledge or a unique performance. It is rather the result of the interaction between different individual abilities and the action context. 3.1 – The Air Traffic Control Case
For the identification of the air traffic control competences, we followed Irigoin and Vargas (2002, p.79) bottom-up constructivist approach. Integrating the results of a two years direct work process observation,
Tech. Process TP Tech. Process TP Work Process WP Work Process WP
Mental Model MM Situation Awareness SA
Occupational Competences Cognitive Dimensions
Process Components

together with qualitative (open interviews) and quantitative (questionnaire) analyses, we identified a list of fifty required abilities for the air traffic controller’s work – see table 1. Table 1 – Air Traffic Control. Example of Identified Abilities
Similar abilities were then grouped into relevant competences’ indicators, according to the above mentioned competences cognitive framework. The result is consolidated in tables 2 and 3 showing, in “quotation marks” the description of the occupational competence indicator and, in italics, the measured associated competence – see tables 2 and 3. The final result was a coherent and comprehensive approach, to the air traffic controller’s work competences identification. Table 2 – Air Traffic Control. Example of Occupational Competences Identification.
ITEM DESIGNATION 1 Integrates technological change in a proactive professional and personal development perspective. 2 Knows and integrates technological capacities in a systemic fashion. .... .... 8 Provides coordination of all traffic before it enters into other jurisdiction areas. … … 36 Proceeds adequately in emergency situations, equipment malfunction or abnormal situations. ... .... 48 Uses all available technological facilities for the resolution of operational problems. ... .... 50 Frequently scans own jurisdiction area.
MENTAL MODEL DIMENSION Work Component ITEM DESIGNATION 13 Knows technological system operational characteristics. 15 Shows a wide knowledge of general air traffic flows and system operation. “Integrates an updated global knowledge of operational agents along with adjacent work sectors and jurisdiction areas operational characteristics”. Self-lifelong learning Technological Component ITEM DESIGNATION 1 Technological change is assumed in a self-professional development proactive attitude. 21 Identifies different operational system components, interfaces and work equipment as well as its functions. 22 Understands operational changes and environment reconfiguration possibilities. “Understands the structure, functionalities and constraints of technological systems, in a personal and professional development perspective”. Self-lifelong learning.

Table 3 – Air Traffic Control. Example of Occupational Competences Identification (Cont.) .
Following this procedure for all fifty identified air traffic controllers work abilities, It was possible to obtain a final set of five key-competences for Mental Model dimension:
• Self Long-Life Training;
• Systhemic Integration;
• Team Culture;
• Assertivity
• Critical Thinking And five key-competences for Situation Awareness dimension:
• Multi-Task acting;
• Comunication Skills;
• Teamwork;
• Analitical thinking;
• Proactivity. These 10 competences are the support of the empirical work. Each element of these two groups of competences is composed by a work component and a technological component, ending up into a total of 20 indicators for the 10 referred competences – see table 4
SITUATION AWARENESS DIMENSION Work Component ITEM DESIGNATION 10 Assumes effective operational command and does not act insecurely. 39 Issues a reduced amount of operational instructions/clearances. “ Assumes effective operational command and does not act insecurely ” Comunication capacity. Technological Component 36 Proceeds adequately in emergency situations, equipment failure or other abnormal situations 41 Knows how to use available comunuication tools. “With equipment failure or any other abnormal situation, adopts work techniques and content to the emergent work contexts”. Comunication capacity.

Table 4 – Global integration of occupational competencies components, with cognitive dimensions. The Air traffic Control Case
4. VIRTUAL TEAM’S WORK New paradigm supporting our investigation and research model requires work organization strategies in a balance dynamics between human and technological agents. These strategies, imply a change of the one-dimensional work structure of Work Post / Qualification / System Operator to a systemic complexity of Individual / Competence / System Manager, where the work process and technological process components are integrated, trough professional competences, to the understanding and development of the work processes. In our research we developed a constructivist methodology, following Irigoin and Vargas (2002) approach. Being air traffic control a complex working environment, we did not expect to find all competences in a single or in each of the different air traffic controller actions. Also, we did not expect to
MENTAL MODEL DIMENSION Work Process Component COMPETENCES Technological Process Component Integrates operational agents global knowledge and adjacent areas and sectors operational characteristics. Continuous learning. MM01 Self - Learning MM06 Integrates control actions interactions, manages operational interfaces, and defines strategies controlling previous actions planning. MM02 Systemic Integration MM07 Knows and uses available technology in a systemic and integrative perspective. Knows all working system intervenient problematic situations and operational needs and adapts own working schemes accordingly. MM03 Team Culture MM08 Identifies technological system possible malfunctions and knows backup equipment, interacting in a natural and safe manner. Understands and uses possibilities of work organisation structure. Selective knowledge construction. MM04 Assertiveness MM09 Adjusts (personalizes) radar presentation data. Evaluates global air traffic circulation and questions the observed reality. MM05 Critical Thinking MM10 Evaluates operational equipment performance and adjusts technological systems use in an effective way. SITUATION AWARENESS DIMENSION Work Process Component COMPETENCES Technological Process Component Timely records operational data updates even in a heavy loaded cognitive work situation. SA01 Multi-tasking SA06 Shows different technical capacities at operational equipment level, in a global operational context perspective. Assumes operational efective command. Pragmatism. SA02 Communication Skills SA07 In abnormal situations or equipment malfunction, adjusts own performance to the different work contexts. Coordinates control actions and does not assume any decision that involves other intervenient, even in its own responsibility area, without prior consultation. SA03 Team Work SA08 Problem solution integrates technological capabilities in a natural and safe systemic perspective. Capacity to Problem identification and data collection, as well as data interpretation. SA04 Analytical Thinking SA09 Shows great attention and consciousness of all technological system particularities. Frequently scans own jurisdiction area anticipating overloaded situations and preventing eventual loss of situation awareness. Identifies complex problems. SA05 Proactivity. SA10 Visualises different operational interfaces and components, anticipating technological system behaviour.

find Work Process Component and Technological Process Component, to occur simultaneously in each of the identified competences. In fact, a complex system may not be studied only or independently on its elementary components, but it is assumed in the dynamics that are established and developed among them. Thus, we defined three moments of competence indicators validation – Working Position Take Over, General Operational Attitude, Operational Situations – applied to different qualitative research methods and techniques such as open interviews, questionnaires, workshops, direct observation, case studies and operational simulation. In this paper, we report to the case studies and operational simulation results. 4.1 – Case Studies Results Air traffic controllers were asked to choose among a set of 20 different operational situations, including work process and technological process components those that, in heir opinion, are the most complex and demanding for operational performance. An example of this assessment is shown in table 5. Table 5 – Case Studies Operational Situations After making their choice, air traffic controllers were asked to specify the procedures used to solve such situations, so that the previous defined indicators could be validated and, consequently, operational competences in both work process and technological process components be identified. Table 6 shows the result of this assessment as Mental Model and Situation Awareness Competences for the Work Process Component (MMWP; SAWP) and for the Technological Process Component (MMTP; SATP).
1 Hight density of traffic converging at the same flight level to a point in the flight plan route. 2 Military exercises. 3 Emergencies. 4 Operational audio frequency out of service. … ................... … ..................... 17 Airborne system malfunction. For example the transponder. 18 Radar data processing system mal function. 19 Technological system inconsistent behaviour. 20 Communication system with adjacent areas, out of service.

Table 6 - Case Studies. Professional Competences Integration.
3.2 – Operational Simulation Results For the operational simulation, air traffic controllers were faced with expected and unexpected complex and/or abnormal operational situations. For example, an emergency reported by the aircraft pilot or by another air traffic control, is an expected complex situation; but a system malfunction or abnormal behaviour is an unexpected complex situation, unless it has been reported before. Moreover, most of the unexpected situations are identified, although its occurrence cannot be predictable. In this sense, operational people have already been trained to face it. Our interest in this phase of the research was to see how operational people react to unexpected and also unforeseeable (absurd) situations. This would allow us to observe the emergence of certain professional competences as well as the identification of eventual competence unbalance between work and technological processes, in these extreme operational situations. Two of these abnormal extreme situations configuring Radar Data Processing Malfunction and Technological System Inconsistent Behaviour, have been included in normal (expected) simulation exercises, thus producing a completed unexpected effect. Results are shown in table 7. Table 7 - Operational Simulation. Professional Competences Integration.
COMPETENCES MENTAL MODEL MM SITUATION AWARENESS SA WP TP WP TP DESCRIPTION 01 02 03 04 05 06 07 08 09 10 01 02 03 04 05 06 07 08 09 10 Take Over 1 1 1 1 1 1 1 1 1 1 1 1 1 1 General Attitude 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Operational Situations 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 INTEGRATION 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
COMPETENCES MENTAL MODEL MM SITUATION AWARENESS SA WP TP WP TP DESCRIPTION 01 02 03 04 05 06 07 08 09 10 01 02 03 04 05 06 07 08 09 10 Take Over 1 1 1 1 1 1 1 General Attitude 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1) Emergencies 1 1 1 1 1 1 1 1 1 1 1 2Abnormal Situations 1 1 1 1 1 1 1 1 1 1 1 3) Traffic Load 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 INTEGRATION 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

5. DISCUSSION CASE STUDY results – table 6 - show that operation is mainly human centred. In fact, Work Process Component appears in all phases of the operational situations with an indicators validation density greater than Technological Component, either for the mental model dimension or for the situation awareness one. But, these results do not mean that human agents are not aware of technological component. A deeper observation of the competences distribution reveals a significant balance between Work Process (WP) and Technological Process (TP) in the take over phase and as a general attitude, for both cognitive dimensions. In particular, we found a greater incidence of Mental Model related competences in the take-over phase (9 in 10 possible) while for the Situation Awareness dimension an incidence of 5 in 10 possible was found. The absence of SA02, SA03, SA05, SA08 and SA09 is due to the fact that they are not necessary in this operational phase, being more adequate to a general attitude or for the resolution of concrete operational situations. After the air traffic controller “enters” the traffic, it is than possible to identify a greater balance between the two cognitive dimensions for both WP and TP components. The only competences that are not present – MM08 and SA 07 – refer to emergency or unpredictable situations, which we left out of case studies, because of its own nature, only perceptible by means of operational simulation. These results also show that, for the operational situations, human agents may not integrate the technological component of a number of professional competences, mainly at the mental model dimension, i. e., human agents evolve to a more reactive attitude towards technological agents behaviour deviation, as human agents are now more concentrated in solving these problems in a working process perspective. There is though a visible evolution from a systemic integration, as general attitude, towards a strategic/systemic integration when concrete operational situations need to be solved. This means human agent does not show competences - MM06, MM 09 and SA 09 - that integrate the operational nature of technological agents – refer to table 2 above. OPERATIONAL SIMULATION results – table 7 - confirm that technology is not yet assumed, in the work process, as a partner at the same level. This situation has been verified when we investigated how do human agents react to unexpected technological abnormal behaviour, in an operational context of total surprise. In fact, results suggest a certain apathy towards technological agents behaviour, configuring a technological overtrust situation (see Sampaio and Guerra 2004) assuming that technological agents are (always) correct and, if not, they will present an error message or assume a pre-determined error coherent behaviour. Table 7, above, shows some expected competences witch results are a) Framed black: Although being important for the operational context, they only occur in a not very significant simulation phase (for example take over or general attitude); and b) Complete black: Although being important for the operational context, they do not occur at all. This situation characterizes the existence of a cognitive framework, which, although being systemic, reveals a tendency towards over trust, meaning the existence of a Technological Factor, as we introduced before. The last line of table 7 shows clearly this situation to: MM08 - Team Culture: Identifies technological system possible malfunctions and knows backup equipment, interacting in a natural and safe manner. SA09 - Analytical Attitude: Shows great attention and consciousness of all technological system particularities. And to other three competences SA10 – Proactivity: Visualises different operational interfaces and components, anticipating technological system behaviour. MM06 – Self - Learning: Knows the structure, functionalities and constraints of technological agents, in a Professional and personal development perspective. MM10 – Critical Thinking: Evaluates operational equipment performance and adjusts technological systems use in an effective way

that although being present as general attitude (MM06 and MM10) and take-over (MM10 and SA10) are absent in the simulated operational situations in contexts they were expected to be present. A deeper analysis shows that general attitude counts with 100% (10 in 10) of WP competences and 60% (2 in 10) of TP competences. This situation changes drastically, when a reported perturbation factor is introduced (for example emergency situations). In fact, table 3 shows that, for situation 1, TP is present in only one competence, among five, for the mental model dimension and two in five, for the situation awareness dimension. When a non reported technical abnormal situation (automation surprise) is introduced, the competence scenario evolves to 1 competence present in five possible for both WP and TP dimensions. Finally, situation 3, because it is a reported one, shows again 2 in 5 and 3 in 5 possible competences for TP, for Mental Model and Situation Awareness dimensions respectively. This means that, in certain circumstances (automation surprises) operational agent may loose the technological system manager dimension, only “surviving” MM07 and SA08 which represent human agent’s operational restricted dimension. 6. CONCLUSION Our research showed that Human/Machine interface can not be reduced any more to the identification and resolution of the “human problem” trough the introduction of more and more different technology. A new balance of a superior order between human and technological agents needs to be met, if a sustainable and coherent operational development evolution is to be achieved. Operational teams have to be understood in a double and virtual dimension between human and technological agents. This means that the study of human nature, to be reflected in system development, needs to be balanced with a greater technological knowledge and empathy, in human social and professional competences development. At the technological level, this evolution towards a virtual team work design and organisation represents the necessity to promote, near human agents, the knowledge of the structure and behaviour of technological agents, meaning the existence of a Technological Factor dimension to be balanced against traditional Human Factors perspective. The research results show this is a major issue to be addressed in actual professional requirements and work organisation policies, if a operational gap, between groupware development and its usability, is to be avoided, and a safe and efficient operation is to be accomplished. REFERENCES Bellier, Sandra (2002), “Acquisition et Transmission des Competences”, Objectif Compétences, MEDEF. Eurocontrol (2000), ATM Strategy 2000+, Vol 1,2 Bruxells. ICAO, (2002), Pans-Ops, Doc. 4444, Montreal, Quebec. Maher, Mary Lou & Gu, Ning (2002), Virtual Worlds = Architectural Design + Computational Elements, Key Centre of Design Computing and Cognition, University of Sydney, web.mit.edu/marym/www/ANZAScA2002.pdf [2007-07-13]. Sampaio José João (2007) The Decision Process in Complex Working Systems and New Emerging Competences. The Air Traffic Control Case, Lisbon, Roma Editora. Sampaio, José João e Guerra, Antonio (2004) The Day God Failed or Overtrust in Automation. A Portuguese Case Study, in Dennis A. et al. HPSAA II - Human Performance, Situation Awareness and Automation, Lawrence Erlbaum Associates, Inc., Publishers – ISBN 0-8058-5341-3. Willans James S. & Harrison Michael D. (1999), Requirements for prototyping the behaviour of virtual environments, Human-Computer Interaction Group, Department of Computer Science, University of York, www.cs.york.ac.uk/~jwillans/papers/kings99.pdf [2006-04-18]. Zarifian, Philippe (1999), Objectif Compétence: Pour Une Nouvelle Logique, Paris, Editions Liaisons.