JISC User Behaviour Observational Study: Final Report User Behaviour in Resource Discovery

2 Project Team William Wong, Hanna Stelmaszewska, Balbir Barn, Interaction Design Center Nazlin Bhimani, Learning Resources, Middlesex University Business Schoo Sukhbinder Barn, Middlesex University November 2009 JISC Programme Manager Ben Wynne JISC e-Content Programme Ref. CSSERSA2 / SERV ENHANCE Acknowledgements The UBiRD Project was funded by JISC, Grant Reference No. CSSERSA2 / SERV ENHANCE. We would like to thank the participants from the London School of Economics, Cranfield University, and Middlesex University, who took part in this study, and the librarians who freely gave of their time to provide us with access to the resources. We would also like to thank Ben Wynne for his support throughout this project.

3 Contents Project Team .......................................................................................................................................................................................................2 JISC Programme Manager .................................................................................................................................................................................2 Acknowledgements .............................................................................................................................................................................................2 Executive Summary .....................................................................................................................................6 Main Findings ......................................................................................................................................................................................................6 Recommendations ..............................................................................................................................................................................................7 CIBER and UBiRD: Similarities and differences ...........................................................................................................................................8 Chapter 1: Introduction .............................................................................................................................12 1.1 Aims .............................................................................................................................................................................................................12 1.2 Preamble .....................................................................................................................................................................................................12 1.3 Background: Information seeking models .................................................................................................................................................13 Chapter 2: Methodology ............................................................................................................................16 2.1 Introduction .................................................................................................................................................................................................16 2.2 Participants ..................................................................................................................................................................................................16 2.3 The study design ..........................................................................................................................................................................................17 2.4 The in-depth interview method ...................................................................................................................................................................17 2.5 Procedure ....................................................................................................................................................................................................18 2.6 Data analysis ...............................................................................................................................................................................................19 2.7 Constraints of the study ..............................................................................................................................................................................19 Chapter 3 Results ......................................................................................................................................20 3.1 The Information Environment .....................................................................................................................................................................20 3.1.1 Electronic Resources: Popularity of use .............................................................................................................................................20 3.1.2 Electronic Resources: User Perceptions and Preferences ...............................................................................................................21 3.1.3 Access point of resources ...................................................................................................................................................................22 3.2 The information search and retrieval process in resource discovery systems ........................................................................................23 3.2.1 Initiate ...................................................................................................................................................................................................23 3.2.2 Search ...................................................................................................................................................................................................24 3.2.3 List and Evaluate ..................................................................................................................................................................................24 3.2.4 Refine or re-formulate .........................................................................................................................................................................24 3.2.5 Select and review .................................................................................................................................................................................24 3.2.6 Store .....................................................................................................................................................................................................24 3.2.7 Abandon ................................................................................................................................................................................................24 3.2.8 Change resource ..................................................................................................................................................................................24 3.3 User information search behaviour and strategies in resource discovery systems ................................................................................25 3.3.1 Initiating Behaviours and Strategies ...................................................................................................................................................25 3.3.2 Search Behaviours and Strategies: Basic or ���atomic��� components of search activities ..................................................................27 3.3.3 Search Groupings ................................................................................................................................................................................27 3.3.4 Search groupings and user groups .....................................................................................................................................................30

4 JISC User Behaviour Observational Study: User Behaviour in Resource Discovery Final Report 3.3.5 Results evaluation behaviour and strategy .........................................................................................................................................31 3.3.6 Refine or re-formulate behaviour and strategy ..................................................................................................................................32 3.3.7 Change resource ..................................................................................................................................................................................33 3.3.8 Abandon behaviour and strategy .........................................................................................................................................................33 3.3.9 View details behaviour and strategy ....................................................................................................................................................34 3.3.10 Store behaviour and strategy ............................................................................................................................................................35 Chapter 4 Discussion ................................................................................................................................36 4.1 Information literacy and Information Technology literacy ........................................................................................................................36 4.2 Information literacy problems ....................................................................................................................................................................37 4.3 The system fails to provide immediate and unproblematic access to integrated resources ..................................................................38 4.4 Confusion when dealing with multiple paradigms and interfaces across systems .................................................................................38 4.5 Users often revert to the familiar ...............................................................................................................................................................40 4.6 Structure vs. Semantics: Database structures hinder users finding information ...................................................................................41 4.7 Users��� perception of the librarian: from technician to substitute course tutor .......................................................................................41 4.8 Need for storage and how it is used ...........................................................................................................................................................42 4.9 Search components grouping .....................................................................................................................................................................42 4.10 Publishers��� business models ....................................................................................................................................................................43 Chapter 5 Conclusions and Recommendations .......................................................................................44 5.1 Usability and effectiveness of the Publishers��� electronic discovery systems ..........................................................................................45 Recommendation 1A: Encourage greater standardization of platforms for all types of resources .........................................................45 Recommendation 1B: Include spelling auto suggestions when searching ...............................................................................................45 Recommendation 1C: Provide an in-built dictionary or encyclopaedia .....................................................................................................45 Recommendation 1D: Ensure keywords are present in every article ........................................................................................................45 Recommendation 1E: Use Simple Interfaces ..............................................................................................................................................45 Recommendation 1F: Make users aware of changes to database functionality .......................................................................................46 Recommendation 1G: Address the ���time out��� issue ...................................................................................................................................46 Recommendation 1H: Improve facilities to support dynamic searches, retrieval and re-access process ..............................................47 Recommendation 1I: Include facilities for annotation and analytical support functions ..........................................................................47 5.2 Usability of Library���s electronic resource discovery systems ...................................................................................................................47 Recommendation 2A: Make it easier for users to identify appropriate resources for their study............................................................47 Recommendation 2B: Avoid confusion caused by multiple usernames and passwords ..........................................................................47 Recommendation 2C: Integrate Free Web Resources as part of the Mix in Resource Discovery ............................................................47 Recommendation 2D: Manage user expectations better ............................................................................................................................48 5.3 Poor understanding of the concept, scope and the way the e-resources operate ..................................................................................48 Recommendation 3A: Make the underlying database structure visible.....................................................................................................48 Recommendation 3B: Provide road-signs in virtual information spaces ..................................................................................................48 5.4 Information and Digital Literacy .................................................................................................................................................................49 Recommendation 4A: Develop a coordinated information and digital literacy strategy ...........................................................................49 References .....................................................................................................................................................

5 JISC User Behaviour Observational Study: User Behaviour in Resource Discovery Final Report Executive Summary This report discusses the information-seeking behaviour of students and researchers working in the Business and Economics disciplines using subscribed and freely available Internet resource discovery systems in three UK HE institutions: Cranfield University, London School of Economics and Middlesex University. The institutions were chosen as exemplars of the Russell Group, the 94 Group, and the Million+ groups of universities in the UK. The intention was to describe and gain a better understanding of: (i) how different users (undergraduates, postgraduates and researchers) currently seek information on the existing resource discovery systems, (ii) the roadmap used in a user���s information seeking journey, and (iii) their expectations and needs based on their understanding and experience of using the Internet to find information resources for academic study. In addition, we hope to inform JISC, publishers of electronic resources, and librarians working in the HE sector about user behaviours and the issues relating to resource discovery systems. The data was obtained from an observational study and in-depth interview of 34 participants. Each participant was studied on an individual basis and each session lasted up to two hours. The analysis of the data provided an insight on: resource discovery systems used by participants and what they found to be useful, users information search behaviour and strategies when seeking information, the issues that affect their searching behaviour, problems and difficulties users experienced with library resources, issues related to physical library services as well as recommendations for the publishers and librarians in order to improve the use of electronic resources. Main Findings The key findings are described next. In the UBiRD study, we find that information literacy is often confused with information technology or digital literacy. Broadly, information literacy is about the abilities to know when to find, to search for, evaluate and make sense of the content. Information technology or digital literacy is about fluency with the procedures and knowledge of the underlying information technology. There is unfortunately some overlap between these definitions as the information that is sought is often contained within the systems. They closely intertwine, such that the way people formulate their queries to find information is highly dependent upon the functionality provided by the information technology. In addition, we have also found that the poor usability, high complexity, and lack of integration of many electronic resource discovery systems, have raised the entry threshold of information technology literacy. This acts as a barrier to information search and retrieval. This higher level of difficulty amongst electronic discovery systems distracts users from focusing on the content, analysis and evaluation that would help them learn and make sense of what they have discovered. In addition, we have also found evidence to suggest that information literacy skills are lacking. For example, at a simplistic level, many of the participants do not understand how to assess the quality of materials they find. Google or Google Scholar have lower thresholds of information technology literacy, and are considered their ������ friends��� because of the apparent higher yield or success rate. While we acknowledge that information literacy may not be a new or recent problem, the evidence suggests that it continues to be a problem. Being able to operate a search engine, does not mean that one is able to find the good quality information necessary to help us learn and to advance our society. Students and researchers from Business and Economics use both resources subscribed to by the library and those freely available on the Internet when seeking information in the academic context. However, as the level of information literacy as well as the domain knowledge increases, there is an increased tendency to use better quality library resources. The most common library resources used by Postgraduates and Experts in our study were EBSCO, ProQuest or Emerald, whereas the Library Catalogue and federated search engines (CrossSearch at Cranfield, QuickSearch at London School of Economics and MultiSearch at Middlesex University) were observed to be more popular amongst Undergraduates. When using freely available Internet resources, Google is top of the list, followed by Google Scholar, Wikipedia and YouTube. Participants��� decisions about which resources to use were based on their prior knowledge and experience with a resource and a belief that resources provided by Google and Google Scholar are reliable and relevant and most of all always return

6 JISC User Behaviour Observational Study: User Behaviour in Resource Discovery Final Report a list of results. On the other hand, library resources were perceived as credible providing a quality material from a broad subject coverage. Users find database structures hinder. They have to learn the procedural knowledge for using a particular database as well as have some basic knowledge of how the data table is organised and what subject matter the built-in thesauri refers to both have limited transferability. The participants did not appear to lack information technology or digital literacy, as they had demonstrated they were able to use other internet-based search and retrieval tools. The UBiRD study groups very rarely applied only one search strategy (e.g. Simple Search) but their strategy changed during the information seeking process in relation to the results obtained (i.e. refine or re-formulate search, abandon search or resource or change resource). Participants usually carried out combined searches: for instance, re-formulated search where terms or concepts extracted from a document were carried out to pursue a new search. This search was often combined with a link search that gave the opportunity to follow hyper-links and extract new queries that were used as an input for the search terms in multiple fields. This group of users also followed the help suggested by a system (e.g. ���Suggested Topics���, or Results by Source���). There are considerable differences between the user groups and the ���atomic��� or basic search components combined during the information seeking activities depend on their level of information literacy and domain knowledge. Expert users are more inclined to use Re-formulated Search, System Suggestions or Personal Knowledge and Experience, whereas UG groups tend to use more frequently a Link Search. Although, current systems functionally support these forms of searching, transitioning between them is currently not as ���seamless��� as it should be, making the search and retrieval process difficult and time consuming. During information seeking activities, the groups we studied used different means of storing the material depending on the stage they were at. At the evaluation of results stage, they store material temporarily using the web browser���s tabs system. These tabs are then re-visited for quick evaluation and then the information may be stored permanently using other means (e.g. notes in a Word document, save downloaded material into a folder, bookmark using a browser���s feature and use of more sophisticated features provided by various resource discovery systems such as RefWorks, Endnote, My Reasearch). The UBiRD study groups did not use the resource discovery systems storage features often, as they were not aware of their existence or how to use them. The current systems lack good ways of storing and retrieving documents allowing the users to create repositories of information that can be accessed easily and be transferable across different resources. Publishers��� embargoes on different material raised an important issue for participants. They were irritated and frustrated when a promise to obtain a document was often not met. Cases like this lower the participants��� level of trust towards academic resources. These practices of embargoes and ���free for the moment��� have encouraged participants to turn to the external sources like Google Scholar where, despite having no promise of the article, and where the scholarly quality cannot be assured, one still has a much higher chance of finding the article to download. The current ways of doing business can get in the way of finding quality scholarly materials through the respective resource discovery systems. Recommendations Based on the findings from the UBiRD study we have suggested a number of recommendations that deal with problems and challenges that users faced during the interaction with the library resources. These recommendations relate to the four areas and are summarised below: 1. Usability and effectiveness of the Publishers electronic discovery systems Since the participants faced a lot of irritation and frustration when trying to use the resources, as their structure, scope and the way they work varied across different resources, greater standardization of platforms for all types of resources is required. Recommendation 1A: Encourage greater standardization of platforms for all types of resources Recommendation 1B: Include spelling auto suggestions when searching

7 JISC User Behaviour Observational Study: User Behaviour in Resource Discovery Final Report Recommendation 1C: Provide an in-built dictionary or encyclopaedia Recommendation 1D: Ensure keywords are present in every article Recommendation 1E: Use Simple Interfaces Recommendation 1F: Make users aware of changes to database functionality Recommendation 1G: Address the ���time out��� issue Recommendation 1H: Improve facilities to support dynamic searches, retrieval and re-access process Recommendation 1I: Include facilities for annotation of resources and analytical support functions 2. Usability of the Library���s electronic resource discovery systems As information on the value of using a resource is not always recognised, making it easier for users to identify appropriate resources for their study would improve/increase the usage of the resources. To facilitate this we recommend the following. Recommendation 2A: Make it easier for users to the identify appropriate resources for their study Recommendation 2B: Avoid confusion caused by multiple usernames and passwords Recommendation 2C: Integrate Free Web Resources as part of the Mix in Resource Discovery Recommendation 2D: Manage user expectations better 3. Poor understanding of the concept, scope and way the resources operate Since users are not always aware of what information is available, contained, organised or stored, the need for making the underlying database structure visible is of high importance. The current systems need to be designed so that they cater for important but intangible attributes that help users understand them, find their way around, and find the information they require, by making the systems meaningful places, so that routes and access are clearly visible and easily followed allowing the smooth navigation of those systems. Recommendation 3A: Make the underlying database structure visible Recommendation 3B: Provide road-signs in virtual information spaces 4. Information and digital literacy While we acknowledge that information literacy may not be a new or recent problem, the evidence suggests that it continues to be a problem. Being able to operate a search engine, does not mean that one is able to find the good quality information necessary to help us learn and to advance our society. The need for the development of a national strategy with an authoritative body defining the standards of information literacy and promoting good pedagogic practices is of paramount importance. Recommendation 4A: Develop a coordinated information and digital literacy strategy CIBER and UBiRD: Similarities and differences In this section we briefly compare the findings of the supplementary CIBER study1 with those from the UBiRD study. The CIBER study was based on the study of transaction logs based on a very large number of users across the UK. These studies include the NEBO study, the RIN study, the Elsevier ���authors as users��� study, and the MaxData study. It highlighted a number of characteristics about the usage of electronic resources during information search and retrieval activities. It was realised that while useful, this data could not provide the additional insights that were potentially available through a richer qualitative and descriptive approach such as has been used in the UBiRD project. Some of these insights from the UBiRD project will be described further in this report. In this section, we will point out some of the key similarities and specific differences that may be found between the two studies. 1 http://ie-repository.jisc.ac.uk/444/

8 JISC User Behaviour Observational Study: User Behaviour in Resource Discovery Final Report CIBER Finding 1: Business and Economics students and academic staff use Google and Google Scholar. The UBiRD study confirms this. However, the UG student group tended to use Google, Wikipedia and YouTube as the main source of finding information. Further, PG students and Experts used Google Scholar to gauge the range of resources available to them and then went back to e-resources to download material. CIBER Finding 2: Business and Economics students and academic staff like simple searching. The data shows that Simple Search (SS) was used by all groups of users. However, in the UBiRD study, it was observed that ���SS��� searches were often carried out in combination with other search components such as Re-formulated Search, Link Search, System Suggestions, Database Selections and other components. UG preferred to use Link Search whereas Re-formulated Search, System Suggestions and Database Search was used more frequently by Experts. For more information refer to sections 3.3.2 and 3.3.3. CIBER Finding 3: Business and Economics students and academic staff tend to search off campus and out of office hours. While some participants reported using university electronic resources remotely, mainly from home, there were also some participants who preferred to access the library resources while at the university. The reason they gave were: (i) the convenience of being able to check if material that cannot be accessed on line can be viewed at the library, (ii) being at home creates too many distractions whereas the library is a quiet place ideal for study or work, (iii) keeping their work place separate from home. CIBER Finding 4: E-textbooks are mainly used for obtaining snippets of information and fact finding. In the UBiRD study only one undergraduate student from LSE attempted to use the ���Course Reading List��� from the library collection but abandoned the resource after failing to find any relevant information. Participants used external resources such as Google, Wikipedia or an online dictionary, to help in understanding the subject. The reason for using these resources, often as their first port of call, was to do with trust and reliability, and having previously had a positive experience in using this source. For more information refer to ���Initiating Behaviour and Strategy��� (section 3.3.1). The low usage of e-textbooks in UBiRD is simply a reflection of the tasks that the participants were instructed to carry out. CIBER Finding 5: Students are the majority users of digital information services, because there are simply many more of them ��� their use is lighter as they tend to view fewer pages. We discovered that there appears to be an inverse relationship between experience or seniority (UG ��� PG-EX) in the use of ���internal��� resources such as EBSCO, ProQuest, Library catalogue, and ���external resources such as Google, YouTube, Yahoo. As experience increases, reliance on ���internal resources appears to increase, whereas reliance on ���external��� resources decreases. CIBER Finding 6: In research intensive institutions the use of databases is greater ��� although spending less time on a visit, and using less of the functions on offer. The data shows that databases were predominantly used by Experts from Cranfield and Postgraduate students from Middlesex University. However, these user groups were also more inclined to use other functions/features offered by the databases. These included Boolean Search and Multi-Field Search. It would appear that the reason for using these combinations of search components lay in the users��� knowledge about the scope, structure and kind of information these databases provide. For more information on the use of different search components refer to section 3.3.3 and 3.3.4. CIBER Finding 7: Business and Economics students and academic staff prefer to view the abstract. In the UBiRD studies, all groups of students viewed abstracts in order to evaluate the usefulness of what they had found. We also observed that the more experienced PG/Expert group tended to spend more time on reading the abstracts. Although all groups wanted access to the full texts, the PG/Expert group persisted more in trying to access the full-text and were clearly annoyed when they could not access a PDF at first attempt. There was no observed distinction between a preference for HTML or PDF in the UBiRD study.

9 JISC User Behaviour Observational Study: User Behaviour in Resource Discovery Final Report CIBER Finding 8: Business and Economics students and academic staff have preferences for current material. For the UBiRD study participants, currency of material was also a relevant factor when evaluating information, although this may have been because participants believed that it was one of the task requirements. They applied a variety of approaches to limit their searches to the current material: (i) Date Limit Search (one of the features within the internal systems��� used), (ii) date was integrated as a part of a search terms, (iii) choice/selection of information sources (e.g. journal articles, report (current information) vs. books (older information). CIBER Finding 9: Most students tend to print the material and then read it. Not all students printed the materials. Those who did not print, expressed their concerns about the environment, whereas others had economic implications in mind and therefore they read the material on the screen. There were others, however, that read from the screen because they are used to doing so. However, it was also observed that students used different ways to store the material for future reading: (i) copy & paste into a Word document (including details about the author, the title, or URL address), (ii) save material in folders (iii) use systems��� feature such as RefWorks, and (iv) use tabbing to keep the track of viewed documents during the information seeking activities before deciding if they are worth storing. CIBER Finding 10: Economists/Business users are the archetypal ���bouncers���, those users viewing just one page. Many participants looked at no more than one/two pages of results from the library resources. However, we observed that some participants viewed more pages (up to 6). Many users spent a considerable amount of time looking through the search results, evaluating materials by reading abstracts and looking at bibliographies. There was a certain determination in how they searched. This is true mainly of PG and research students. The UG students also persisted in finding the right resources for the task. CIBER Finding 11: LSE was the super-user both in e-books and e-journals, making more visits and viewing more pages. In the UBiRD study there were only two participants from LSE (UG) who attempted to use e-books. However, they did not appear to understand the concept of the e-book and did not know what to do with them, hence they abandoned the resource. The UBiRD study did not investigate this further. CIBER Finding 12: Business and Economics students and academic staff make short visits when looking for information and see only few pages and documents. The UBiRD study did not investigate this. CIBER Finding 13: All the available evidence shows that people behave in very diverse ways when using electronic information resources. This is true of UBiRD participants��� information seeking behaviours too. There is not a standard although there are common processes used by all groups of users as demonstrated by the various ways in which the ���atomic��� search components were combined. CIBER Finding 14: Library users have rapidly become information customers who can switch instantly between commercial search engines, social networking site, wikis, bookmarked resources and electronic services provided by their library to satisfy their information needs. In the UBiRD study participants used Google, Yahoo, Google Scholar, YouTube. There was no evidence of the use of social networking sites although a mention was made of going on ���chat��� and electronic resources. What we observed was that UBiRD participants rapidly and dynamically changed resources when they ���hit the wall��� or when they needed to find better terms or to re-formulate their queries.

10 JISC User Behaviour Observational Study: User Behaviour in Resource Discovery Final Report CIBER Finding 15: Everyone exhibits a bouncing/flicking behaviour, which sees them searching horizontally rather than vertically. Power browsing and viewing is the norm for all. Perhaps because the participants were set tasks, they were not observed to practise the bouncing/ flicking behaviour. On the contrary there was evidence of users going to sources such as Wikipedia that would give them background information on their task. This is particularly true of undergraduate students. What we did not see is a systematic manner of interrogating databases at the UG level ��� using the refining and limiting options. CIBER Finding 16: Users assess authority and trust for themselves in a matter of seconds by dipping and cross-checking across different sites and by relying on favoured brands (e.g. Google). There was certainly a reliance on Google and Google Scholar as a source of a full range of resources. CIBER Finding 17: The average times that users spend on e-book and e-journal sites are very short: typically four to eight minutes respectively. It is clear that users are not reading online in the traditional sense, indeed there are signs that new forms of ���reading��� are emerging as users ���power browse��� horizontally through titles, contents pages and abstracts going for quick wins. It almost seems that they go online to avoid reading in the traditional sense. This was difficult to gauge accurately in our study as we had a set allocated time to observe. The study revealed, however, that users spent time evaluating the use of an article by reading the abstract or summary, skim reading and scanning the other parts of a document such as ���Tables of content���, ���Introduction���, ���Conclusion���, the list of references and if deemed relevant, they would store the article.

11 JISC User Behaviour Observational Study: User Behaviour in Resource Discovery Final Report Chapter 1: Introduction 1.1 Aims The UBiRD (User Behaviour in Resource Discovery) project aims to identify, understand and compare the information- seeking behaviours of students and researchers working in the disciplines of Business and Economics using subscribed and free resource discovery systems available in three UK HE institutions (Cranfield, LSE and Middlesex University). It is envisaged that the results of this study will inform the following on issues relating to resource discovery systems and user behaviour: ������ JISC ������ Publishers of Electronic Resources and ������ Librarians working in the HE sector. It is intended that this project will lead to a better understanding of: how the user currently seeks information on the existing resource discovery systems the roadmap used in a user���s information seeking journey and their expectations and needs based on their understanding and experience of using the Internet to find information resources for academic study. It is also intended that the project will lead to recommendations on: ������ how library-subscribed resource discovery systems can better meet the needs and expectations of users and ������ the importance of ensuring information literacy (and digital literacy2) is at forefront of curriculum development in HE. 1.2 Preamble HE libraries are spending a significant proportion of their learning materials budgets on electronic resources: full-text databases, back copies of e-journals, and e-books. It is generally perceived by librarians and teaching staff in HE that users are not accessing the scholarly material that is available through the library and are indeed more than ever using Google as their first port of call to obtain their information. There are many reasons for this ��� and the information seeking behaviour of the ���Google Generation��� is adequately described in the jointly commissioned JISC-BL CIBER report (2008)3. Research into the behaviour of users, especially research that leads to practical proposals for promoting academic alternatives to Google and increasing the information searching and evaluation skills of users within HE institutions, is urgently required. ���Information Literacy��� is now a commonly used term ��� and addressing the skills required by students to reach a high level of competence in searching for and evaluating information materials should become more central to the teaching and learning strategies of many universities. The Chartered Institute of Library and Information Professionals (CILIP, 2004) and the American Library Association (ALA, 2005) have usefully contributed to these discussions by defining the term ���information literacy��� while The Society of College, National and University Libraries (SCONUL) has identified the ���Seven Pillars of Information Literacy��� in its bid to identify the information skills required of users engaged in HE study and life- long learning. The importance of information literacy is also beginning to be acknowledged in the wider socio-political world. To this end, October 2009 was declared the ���Information Literacy Awareness Month��� in America by President Barack Obama. 2 In the context of this report, by digital literacy we mean user expertise in using and understanding different resource discovery systems from access and authentication issues to system capabilities such as searching and storage/exporting options. 3 Accessible from: http://www.jisc.ac.uk/whatwedo/programmes/resourcediscovery/googlegen.aspx

12 JISC User Behaviour Observational Study: User Behaviour in Resource Discovery Final Report In broad terms, libraries are responding to the challenges presented by new technologies and the plethora of resources now available online (subscribed and free) by purchasing meta-search and link resolver systems, vertical search engines and/or third-generation OPAC software and many HE institutions have also eased access to online resources through a single sign-on so that students can authenticate and access resources from their Virtual Learning Environments (VLE) without the need to click through more than once. Despite developments such as these, the overall picture is a confusing one, often characterised by a reactive and fragmented approach rather than one based on a coordinated strategy with a clear vision. Some of these issues have been highlighted in the JISC SCONUL LMS Report4. Figure 1 illustrates conceptually the different systems that an information seeker in HE needs to understand in order to locate appropriate resources for study and research. It may be expecting too much for users to take the time to understand various access points (and as many as four different ways to access a resource) and get a hold of quality resources appropriate for academic study. Many of these users tend to rely on free web resources primarily because access to these is perceived as being much easier and quicker than the library provided resources. Figure 1: The many ways in which users can access quality resources in a typical HE resource discovery environment The results of this study will be two-fold: (1) to provide an insight into information-seeking behaviours of undergraduate and post-graduate students and researchers and (2) to propose to publishers design enhancements in the technologies that are used to access information. 1.3 Background: Information seeking models Over the years, a number of studies have been conducted to investigate people���s information searching behaviours. These studies have led to the development of a number of models, capturing and emphasising different stages of information seeking activities, (e.g. see Bates, 1990 Ellis, 1989 Ellis, Cox, & Hall, 1993 Ellis & Haugan, 1997 Kuhlthau, 1988 Marchionini, 1995 Sutcliffe & Ennis, 1998 and Wilson, 1999). A brief review of key information seeking models is presented next. 4 Accessible at: http://www.jisc.ac.uk/whatwedo/programmes/resourcediscovery/libraryMS

13 JISC User Behaviour Observational Study: User Behaviour in Resource Discovery Final Report Kuhlthau (1988) describes a six���stage cycle of initiation, selection, exploration, formulation, collection and presentation, based on her observations of students working on an assignment in a physical library. The original model of information seeking behaviours suggested by Ellis (1989) based on studies of information seeking activities of social scientists, generated six characteristics of information seeking patterns including: starting, chaining, browsing, differentiating, monitoring, and extracting. Ellis noted that, ���the detailed interrelation or interaction of the features in any individual information seeking pattern will depend on the unique circumstances of the information seeking activities of the person concerned at that particular point in time��� (Ellis, 1989, p. 178). This work was then extended by two more stages: verifying and ending, as a result of a study on the information seeking activities of research physicists and chemists (Ellis, 1993, 1997). Marchionini (1995) proposed a similar model perhaps better for electronic environments. The model identifies a sequence of eight action-oriented tasks developed in parallel: recognition of a problem, definition of the problem, selection of information resources, query formulation, search execution, results examination, information extraction and information search reflection, iteration and completion. Unlike Ellis (1989), this model presents different processes occurring during information seeking rather than behaviour of individual information seekers. Bates (1990) has described the information seeking process more from a strategic point of view whereas Wilson (1999) viewed it as task processes. The emphasis of the latter model is on stages in the problem solving process, which occur in a linear manner. These stages constitute: problem identification, problem definition, problem resolution, and solution statement. The model proposed by Sutcliffe and Ennis (1998) supports the development of a cognitive model of information retrieval which consists of four activities: problem identification, need articulation, query formulation, and results evaluation. These models are usually not intended to fully model user behaviour, but instead to model the more frequent and predictable user behaviour. It should be noted that applications that require huge efforts from their users risk becoming ineffective and useless (Langley, 1999). A different approach to information seeking is presented by Pirolli and Card (1999). They describe information retrieval (seeking) behaviour in terms of ���information foraging���, which was adopted from a food foraging theory called optimal foraging theory that helps biologists understand the factors determining an animal���s food preference and feeding strategies. Based on this theory Piroli & Card (1999) developed an ACT-IF (���adaptive control of thought in information foraging���) process model that can be used to evaluate information retrieval technology and understand user behaviour. They argue that users perceive relevance of information based on some measures of information scent, which are usually based on statistical techniques that extract semantic relatedness of words from large text databases. The model offers a novel and useful perspective on designing systems for information retrieval. It encourages the designer to think about the structure of the interface, how to support different searching and browsing strategies appropriate for the context of work and how to effectively use metadata cues to enhance item selection and patch navigation. Based on her observations of users in their natural settings, Spencer (2006) presents a practical view on people searching for information. She argues that people experience four different modes during their information seeking activity. These modes of information search are: (i) ���known-item��� search where people know what they want, what search terms to use and what resources will provide them with information (ii) ���exploratory��� search where people have some idea of what they need but their search terms may not be the right ones and they may not know which resources to use. However, when they do find information, they will recognise it if it is the right information (iii) ���don���t know what you need to know��� search where people may or may not know how to articulate what they need, i.e. may not have the right vocabulary but will recognize when they have found the right answer even if they may not know whether they have found enough information and (iv) ���re-finding��� where people look for things they have seen before, i.e. hunting down information they have previously seen.

14 JISC User Behaviour Observational Study: User Behaviour in Resource Discovery Final Report Many of the participants in the UBiRD study appear to follow Spencer���s modes of discovery ��� with an additional mode ���storage��� which Jones, Dumais & Bruce (2002) refer to as ���gathering��� or ���keeping��� information���. One other element that is implied but not explicit in the above models is the prior knowledge and experience a user comes with when beginning a search. This study will outline the stages in the information seeking process in relation to the above models.

15 JISC User Behaviour Observational Study: User Behaviour in Resource Discovery Final Report Chapter 2: Methodology 2.1 Introduction The study adopted a qualitative research approach to investigating user behaviour as they search for quality scholarly materials in resource discovery systems. The study sought to gather detailed information about how HE users interact with different resources, the search strategies they applied, and how these strategies differed across different levels of information and digital literacy. Qualitative research data is very rich and descriptive but is, unfortunately, voluminous. For example, the transcript of a one-hour interview can often be in excess of 15 pages. While possible, it is not the purpose of qualitative research to generate numbers that could be used to carry out inferential statistics. That sort of study would draw on the tradition of quantitative research, and their application in the context of this research has already been carried out by the complementary work of the CIBER research group and the findings of their ���Google generation��� study reported elsewhere5 . Instead, some of the key purposes of qualitative research are to provide descriptions from which we can (i) un-ravel and explain behaviours, and (ii) infer from them the variety of circumstances and situations that people will encounter and how they deal with them. Given the time frame, and partly to complement the supplementary CIBER report6, we focused on studying business and economics students and researchers at a sample of three universities, representing the Russell Group, the 94 Group, and the Million+ groups of universities in the UK. These universities are the London School of Economics, Cranfield University, and Middlesex University. 2.2 Participants Using a stratified sampling strategy, we recruited a total of 34 volunteers from across different levels of information literacy: undergraduate, postgraduate and expert researchers. There were 16 male and 18 female volunteers in the study, aged between 22���55 years. Following CIBER���s work such as the E-Books Observatory Deep Log Analysis7, the participants in our study were selected based on a similar sampling frame. The study was conducted in two stages (to be described in detail in the next section) and the number of participants was distributed across the study conditions as shown in Table 1. Stage 1 was an exploratory study and was conducted first. It involved two sets of focus groups involving a total of 9 participants at Middlesex University. In Stage 2 we carried out a combined user observation and in-depth interview study involving a total of 34 participants distributed across the three universities. Table 1 shows how the participants are distributed across the different information literacy categories. Detailed demographic information about participants is presented in Appendix 1. Under- graduates Post- graduates Researchers Total Stage 1: Focus Groups (2) 6 3 0 9 Stage 2: Observations+ In-depth interviews London School of Economics 5 4 3 12 Middlesex University 5 3 4 12 Cranfield University 0 6 4 10 Total (Observations & In-depth interviews) 10 13 11 34 Table 1. Number of Participants in study across universities 5 http://www.jisc.ac.uk/whatwedo/programmes/resourcediscovery/googlegen.aspx 6 http://ie-repository.jisc.ac.uk/444/ 7 Available at: http://www.ucl.ac.uk/infostudies/research/ciber/observatory/