G. TSAKONAS AND C. PAPATHEODOROU Two concepts, usefulness and usability, have been investigated extensively in previous studies, especially in the area of acceptance and success of information systems. The Technology Acceptance Model (TAM) [1] refers to the notions of perceived usefulness and perceived ease of use, as predictors of system usage. TAM examines system acceptance in a behavioural perspective and it has been used in various types of information system, like digital libraries (DLs) [2] and portals [3]. Shih [4] employs TAM in a linear model of information search and use. He indicates that infor- mation relevance, which is traced at the early stages of the information seeking process, may positively influ- ence other properties found in the succeeding stages, such as perceived usefulness, perceived ease of use and perceived performance. It may also have a positive effect on the users��� attitude towards an information system. The contribution of these two concepts to the information production and usage cycle is illustrated through the Delone and McLean model [5], which has its foundations in theoretical analysis of prior empirical studies. System quality and information quality (as equivalents of good state usability and usefulness respectively) have an effect on system usage and user satisfaction, which as a consequence may influence individual and organizational performance. The P3 model [6] attempts to merge usability engi- neering and acceptance models. In the P3 model, power, perception and performance assess the user���s ability to use a given system. A system���s power indi- cates its potential abilities to serve the user���s tasks, while perception and performance measure his percep- tual and behavioural reactions. The P3 model extends TAM by surrounding it with the technical abilities of a system, in terms of provided functionalities, and by inquiring about the user���s ability to exploit these func- tionalities. In the present study we incorporate usefulness and usability in a general framework, which bridges the various approaches for EIS evaluation and provides a holistic picture of user interaction. We analyse each framework component for a set of nodal attributes and through an experimental study we attempt to (1) investigate the relation between usefulness and evaluation concepts, and (2) trace which attributes of the framework com- ponents are important for the successful completion of users��� informational and work tasks. We seek the potential benefits of informing system designers and content stakeholders about the influen- tial power of the component attributes. 2. A framework for user interaction evaluation According to most researchers, EIS refers to systems consisting of users, system and content. These three components have been studied in depth in order to develop evaluation models able to predict DL usage [7] or analyse the information seeking and retrieval process. J��rvelin and Ingwersen [8] illustrate the communication process of a ���cognitive actor���, who interacts, through an interface, with the outcome of information objects and information systems. Many of the proposed user interaction and infor- mation behaviour models present a stage-oriented perspective of search in information systems [9, 10]. However, recent studies indicate that the information seeking process does not proceed in an absolutely linear form, but several iterations, shifts and serendip- itous discoveries of information occur [11, 12]. The substitution of the linear interaction consisting of time- or stage-defined sequences of actions by iterative information seeking behaviour is partially caused by the constantly developing characteristics of web-based information services. Based on the descriptive poten- tial of the triptych framework, the following develop- ments are identified. System characteristics. The processes and the mech- anisms of information retrieval (IR) are expanded through the increase of information resources and the demand for more effective mechanisms of manage- ment. Chowdhury and Chowdhury [13] thoroughly analyse the features of twenty IR systems, concluding that today���s systems are equipped with advanced func- tionalities, which increase interaction complexity. Similar studies have concentrated on the interface representations of IR functions [14]. Moreover, today���s information systems provide additional sets of service, tailor-made to the specific needs of each user. Holm- str��m [15] notes that DLs offer a wealth of value-adding services, which support users in the management of information overload. Content characteristics. Information is represented as an object, which communicates with system features that are responsible for storage and administration. Information spreads across several levels of detail (citations, abstracts, full text) and modern information systems allow in-depth searching of terms. Modern information systems store and provide various forms of information that enrich system profile, for instance text (from HTML to PDFs), audio, video etc. [16]. 401 Journal of Information Science, 32 (5) 2006, pp. 400���419 �� CILIP, DOI: 10.1177/0165551506065934 �� 2006 Chartered Institute of Library and Information Professionals. All rights reserved. Not for commercial use or unauthorized distribution. at PENNSYLVANIA STATE UNIV on February 6, 2008 http://jis.sagepub.com Downloaded from

Usefulness and usability Within this frame, users unfold information search- ing actions, which are set in a wider and temporally extended activity [17]. This activity constitutes users��� information seeking behaviour, which according to Hansen ���is dependent on several factors, such as the user���s preferences, knowledge, the tasks and goals, the information object, the domain, and the satisfaction with search outcome��� [18]. Users��� actions are not stim- ulated solely by the intrinsic properties of information, but also by interface elements that visually represent content structure and intimate semantic organization [19]. Users select categories of meta-information objects, note the various cues of information and finally extract meaning from this information. Due to the increase and multiplication of ���communication ignition��� points, interaction is transformed into ���inter- play between cognitive and physical activities��� [20]. Between the three components (system, content, users) several interactions take place generating the triptych seen in Figure 1. Each component com- municates and interacts with the other two, by request and supply of certain features. These interacting components define three evaluation approaches, which might accordingly be characterized as user-centred, system-centred or content-centred evaluation, depend- ing on the point at which someone begins to evaluate. We consider the triangle sides as the main evaluation axes. The system-user side defines the usability evalu- ation axis with the related measures of efficiency, effectiveness and user satisfaction. The user-content pair defines the usefulness evaluation axis with the related measures of task applicability and resource integration ability. The system-content side defines the performance evaluation axis with the related measures of precision, recall and response time. According to the user-centred aspect of the triptych framework, effective interaction depends on system usability and information usefulness. In our approach we hypothesize that the user communicates with both system and content in a unified and indiscriminate way and that interaction consists of the merging of physical, affective, cognitive and conceptual actions and judgments. In addition, we believe that users select information resources and systems which cover their informational requests, satisfy their information needs and do not require significant effort in use. Users express their demands and preferences by interacting with both components. Their skills can be identified as system use and information use skills. They are able to understand the typology of content resources and to assess their relevance and importance for completing their tasks, based on the resource features and the semantic substance of information. At the physical level, users deploy their skills to control the system through interface elements, to comprehend the basic and advanced system features and to operate accordingly to execute tasks. In the following subsections we further analyse the concepts of usefulness and usability and present the content and system features, which are assessed. 3. Analysing the axes to evaluation criteria 3.1. Usefulness Usefulness is the degree to which a specific infor- mation item will serve the information needs of the user. Usefulness may be determined by: (a) inherent attributes of the resource, as semantic entity and as an object (b) its applicability to specific information seeking contexts and work tasks [21]. Usefulness ��� as an extension of the concept of relevance ��� is a research interest of user and infor- mation behaviour studies. While usability studies, which are discussed in a later section, investigate the quality of interaction between the user and the system, user and information behaviour studies enquire about the quality of the cognitive, semantic and affective interaction between the user and the content. Moreover, user studies are the prime means for library and information services to collect usage patterns and user preferences on several practical issues and they use a wide range of measuring units, depending each time on the methods they employ. They consist mainly 402 Journal of Information Science, 32 (5) 2006, pp. 400���419 �� CILIP, DOI: 10.1177/0165551506065934 System Content User Performance Usability Usefulness Fig. 1. Associations in the interaction triptych framework. �� 2006 Chartered Institute of Library and Information Professionals. All rights reserved. Not for commercial use or unauthorized distribution. at PENNSYLVANIA STATE UNIV on February 6, 2008 http://jis.sagepub.com Downloaded from

G. TSAKONAS AND C. PAPATHEODOROU of qualitative methods, like surveys, interviews, focus groups and user protocols, all able to gather infor- mation about users��� perceptions and beliefs, but quan- titative techniques, such as transaction log analysis, are also in use. Through these studies several content attributes are projected as strong decisive elements of user satisfaction. Many of them, such as currency, full- text search etc. are considered as advantages over printed equivalents [22]. Other study issues deal with the influence of resource types and formats on the acceptance of EISs together with other factors like searching mode, social networking conditions and access to information [23, 24]. Finally through these studies the attributes that support judgmental actions, such as the level of information [25] and information reliability and credibility [26] are highlighted. The usefulness criteria employed in this research include the following five resource attributes that, according to the triptych framework, can affect user interaction. They are also highlighted in previous studies as important factors for the selection of infor- mation resources and constitute determinants of information quality [27]. (1) Relevance denotes how (topically) the content corresponds to the work task. (2) Format is a resource attribute that connects with the user���s work practice and/or the available technological infrastructure. (3) Reliability investigates how credible the resource is and how well it satisfies present and future aspects of the work task. (4) Level refers to the various representations of infor- mation provided, such as abstracts, full text etc. (5) Timeliness investigates how current the infor- mation resource is and how well it will satisfy the information need. 3.2. Usability Usability is concerned with the ease of use of a given system in an efficient, effective and satisfactory way. Interaction in usable systems premises a normal and smooth communication process between the user and the system features, where information exchange (input/output), assessments and interpretations are required in order to execute typical tasks [28]. Usabil- ity has gained significant attention in the field of infor- mation science and user-friendliness is considered a determinant of system acceptance and perceived quality. The usual methods of usability evaluation include comparative and inspection techniques, trans- action log analysis and other variations of automated evaluation, as well as methods of qualitative research, like surveys, focus groups etc. [29]. Among the many problems found by usability studies of library-related information services, the three most commonly met are terminology, layout and navigation. Terminology problems in information websites alienate users and place them in a ���hostile��� information environment [30, 31]. Aesthetic appear- ance and visual layout have been found to influence system performance in download time and to dis- content users through inappropriateness [32���34]. Navigation aids are supposed to improve user inter- action by means of recognizing current states and to support regularity during a session [35] but sometimes for several reasons fail to do so. All these problems are commonly found, not only throughout a variety of evaluated systems (websites and DLs), but also through different evaluation methods [36]. Finally, in recent studies the need for learnable systems is underlined in order to minimize the time required to learn the use directives and upgrade the efficiency level [37]. In conclusion, the usability evaluation criteria referred to in the bibliography and adopted by the present research are as follows: (1) Ease of use refers to how easy it is to use all func- tions provided by the system. (2) Aesthetic appearance of the system may influence the users affectively. (3) Navigation is the ease of navigation through the system. (4) Terminology refers to the comprehensibility of terms and phrases used to describe functions or content. (5) Learnability is an intrinsic property of usable systems that delivers users from the process of self-instruction or attending structured courses. These criteria may stimulate and further encourage use of the system or may frustrate and lead the user to abandon the task. 3.3. Performance Background operations are often neglected in user- centred evaluation or left to be addressed by system- centred evaluation. System performance is a determinant factor for user acceptance of a system, especially in web-based information systems. Preci- sion and recall remain the principle evaluation criteria for the IR mechanisms of EIS, but with the employ- ment of web-based technologies other criteria have been developed, e.g. response time. In fact, some of these criteria are not assessed directly, but they tend 403 Journal of Information Science, 32 (5) 2006, pp. 400���419 �� CILIP, DOI: 10.1177/0165551506065934 �� 2006 Chartered Institute of Library and Information Professionals. All rights reserved. Not for commercial use or unauthorized distribution. at PENNSYLVANIA STATE UNIV on February 6, 2008 http://jis.sagepub.com Downloaded from