Pragmatic expert elicitation for defence capability analysis

Abstract

Defence Capability Analysis examines the ability of a real or notional force to conduct military tasks. When faced with a paucity of quantitative data or lack of sufficient models, there is a need to rely on Expert Elicitation as a pragmatic form of preliminary assessment. Typically, Subject Matter Experts (SMEs) are brought together to work through a hierarchy of questions about the capability area under investigation. In these activities, it is essential that individual responses are captured adequately, along with positive and negative feedback collected from the wider group, because analysts need to comprehend the substance of discussions and to identify where and why agreement or otherwise occurred. An overall sense of progress via aggregation of responses is needed by participants to frame the ongoing debate. Facilitators benefit from real-time support to guide discussion purposefully. Stakeholders wishing to explore the data and analysis products mean these must be persisted and remain accessible. The analysis requirements have led the authors to implement a prototype web application called the Maritime Expert Elicitation Capability Analysis Tool (MEECAT). An overview of the system architecture is given, and the main strengths and weaknesses are discussed. MEECAT features include hierarchical structuring, anonymity of respondents, and a self-organising feedback loop in which participants comment on and promote/demote the responses of others. The tool assists in group facilitation through the visual way responses are aggregated, so that consensus or high variance are easily identified, and overall progress displayed. Participants, facilitators, analysts and interested observers benefit from the immediate remote browsability of all responses. Some suggestions for enhancement of the tool and analytical technique are offered, based on experience, user comments and the enduring analytical aims driving the tool’s inception. The validity of this approach is ultimately gauged by its utility in systematically capturing warfighting problems and enumerating and articulating possible causes. MEECAT has been adopted successfully for a range of client centric activities from current force warfighting capability assessments to high level capability requirements elicitation and Navy strategic planning.