Evaluation of the effectiveness of using computer-based business training simulations to develop managerial competency

Abstract

Computer-based simulations and games are powerful tools to support learning environments (Swanson and Holton, 1999) and Gartner research suggest that simulations may be e-learning’s ‘killer application’ (Lundy et al., 2002). The multi-billion dollar business and management training industry and management education are beginning to turn more serious attention to using simulations and games but there are doubts about even the most fundamental claims of the efficacy of simulations (Feinstein and Cannon, 2002). This study tests a model in comparing a training programme using three different experiential activities, a simulation, a business game and case studies using Kirkpatrick’s (1959/60) familiar and ubiquitous (Russ-Eft and Preskill, 2001) four levels as a guiding model for evaluation. In particular, the study focuses attention on the development of managerial competencies and the differences in demonstrated competency before and after (May, 1993) a strategic management training programme (Baker et al., 1997).The literature on management learning provides insights into the multi-disciplinary nature of the research and highlights the many factors considered to influence and shape the way people learn and transfer their learning to the workplace including enjoyment of the learning event (Prensky, 2000, Schank, 1997) and motivation (Holton, 1996), learning style (Kolb, 1976) and personality type (Patz, 1990, 1992), the potential affect of team working (Higgs, 1999) and personal background such as age, gender, cultural heritage and prior academic achievement (Sternberg, 1997). Managerial competency models are discussed and compared to establish the most fitting model for the location of the research and to provide a measure of change in behaviour and individual competencies are linked to organisational competence and performance – providing the links across the four levels of evaluation.An appropriate research methodology is discussed and the chosen quasi-experimental design fits the scientific research tradition demands for robust methodology and the pragmatic demands of research conducted in the business training world (Easterby-Smith et al., 1991). Operationalisation of suitable constructs and processes provide the empirical evidence repeatedly called for in previous studies.Data was collected from 266 participants working for private companies in Malaysia and Singapore and results detected significant differences between both simulation and game groups and the case study groups in reaction, learning and learning transfer and business impact and, as implied by Kirkpatrick (1959/60), significant correlations between the different levels are observed but the strength of the relationship falls well short of sufficiently explaining the results. Both programmes including the simulation or the game show higher levels of learning, change in demonstrated managerial competencies and in business performance than the case study based programme groups providing strong evidence that simulations and games are effective tools to employ within an experiential learning intervention. The continued importance of human tutors or facilitators to provide useful feedback and debriefing from the activities is strongly indicated.From the literature, factors that may influence learning are considered but the results failed to detect any differences between learning styles or personality type. However, some influence was detected with the age of participants with younger managers as suggested by Aldrich (2002), significantly enjoying the computer-based activities over older managers. The impact of team working and enjoyment of team working within the activities is discussed suggesting that the composition of the team (Belbin et al., 1976, 1981), the task and the context, whether competitive or collaborative may impact which competencies are developed. As suggested by previous research in this field (Wolfe and Guth, 1975, Keys and Wolfe, 1990, Brenenstuhl and Catalanello, 1979, Gopinath and Sawyer, 1999, Hannafin, 1992, Hannafin et al., 1996), the results provide empirical evidence that computer-based simulations and games are significantly more useful for learning and competency development than case studies.