Digital libraries (DLs) have emerged as one of the essential scholarly information systems in support of research and teaching. Many libraries have digitized their collections, such as pictures, books, or audios, to make them available on the web. Digital libraries are relatively new phenomena and, like many new and emergent information systems, face challenges of acceptance, utilization, and evaluation. The concept of a digital library has been perceived in different ways by different groups of people. To the community of librarians and LIS researchers, a digital library is an extension and augmentation of library services combined with a remote access to digitized resources. To computer scientists, a digital library is a distributed information system or networked multimedia information system (Fox et al. 1995). In this study, our focus is on the users’ perspective. A digital library is defined as the collection of digitized or digitally born items that are stored, managed, serviced, and preserved by digital library professionals.

The exponential growth of DLs has created a need for the evaluation of these emergent information systems. Digital library evaluation has become a critical issue for both professionals and researchers. DLs have their own unique characteristics and features compared to traditional library services. Also, as an information retrieval system, digital libraries are fairly different from other types of online information retrieval systems (e.g., search engines, online databases, OPACs). DLs have well-organized metadata, browsing categories, as well as digitized items in different formats. Therefore, previous evaluation frameworks used in traditional libraries or other information retrieval systems proved to be insufficient in assessing different aspects of DLs. The evaluation of DLs is conceptually complex and pragmatically challenging (Saracevic and Covi 2000). Borgman et al. (2000) also pointed out technical complexity, variety of content, and the lack of evaluation methods posed the key challenges of digital library evaluation.

Researchers have exerted efforts to develop new frameworks and methods of digital library evaluation. A number of evaluation criteria were suggested covering different dimensions of digital libraries. The early DL projects, funded by the National Science Foundation (NSF) as part of Digital Libraries Initiatives I and II, laid groundwork in evaluation research by producing DL prototypes and frameworks (Borgman et al. 2000; Buttenfield 1999; Hill et al. 2000; Van House et al. 1996). In particular, Hill et al. (1997) identified several criteria for digital library evaluation, such as ease of use, overall appeal, usefulness and overall performance. Saracevic (2004) identified six classes of criteria: content, technology, interface, process/service, user, and context. This evaluation framework covers multiple aspects of digital libraries comprehensively, and is one of the first attempts to assess the context aspect in digital libraries. Xie’s (2006/2008) evaluation framework shifted a focus to the users and posited five types of criteria: usability, collection quality, service quality, system performance efficiency, and user feedback solicitation. In particular, when developing the evaluation framework, she analyzed users’ perceptions based on diaries, questionnaires, and interviews. Zhang (2010) validated the evaluation criteria for digital libraries. Based on Saracevic's (2004) framework, she investigated the importance of evaluation criteria using the empirical survey data from heterogeneous stakeholders. Noh (2010) identified multiple dimensions and corresponding evaluation indices for electronic resources.

In Europe, DELOS Network of Excellence has conducted a series of projects regarding the evaluation of digital libraries. DELOS is a comprehensive and large scale DL project, which represents joint activities aimed at integrating and coordinating the ongoing research efforts of the major European teams working in the digital library area. Candela et al. (2007) established DELOS Manifesto which presents a three-tier DL framework incorporating six core components such as content, functionality, quality, policy, architecture, and user. Fuhr et al. (2001) proposed an evaluation scheme for digital libraries which covers four dimensions including data/collection, system/technology, users, and usage. Based on the examination of the interactions amongst digital library components, Tsakonas et al. (2004) proposed the major evaluation foci, such as usability, usefulness, and system performance. Fuhr et al. (2007) developed a DL evaluation framework based on a DELOS model and conducted a large-scale survey of DL evaluation activities.

Among different aspects of digital libraries, usability has been one of the major concerns in evaluation. Usability consists of multiple attributes from various perspectives such as learnability, efficiency, effectiveness, memorability, errors, and satisfaction (Nielson 1993). Several researchers tried to suggest a usability evaluation model tailored to digital libraries. For example, Jeng (2005) suggested an evaluation framework on usability of academic DLs focusing on four attributes: effectiveness, efficiency, satisfaction, and learnability. Ward and Hiller (2005) suggested usability evaluation criteria specific to library services, such as completion of the task, time and effort, and reaction to the product or service. Joo and Lee (2011) developed an instrument tool to measure the usability of digital libraries. They further tested the validity and reliability of the tool. Matusiak (2012) examined the relationship between usability and usefulness, and found that user perceptions of usefulness and usability, especially perceived ease of use, play an important role in user intentions to adopt and use digital collections.

Researchers strived to identify a set of evaluation criteria for digital libraries. However, there has been relatively less effort devoted to the investigation of users’ perceptions in selecting evaluation criteria. Ultimately, digital libraries are developed to provide information and services to users, and users’ opinions should be considered in the evaluation of digital libraries. It is important that all efforts in the evaluation of digital libraries should be rooted in users’ information needs and characteristics as well as contexts involving the users of those libraries (Marchionini et al. 1998).

This study is one of the few attempts to survey users’ perceptions of evaluation criteria for digital libraries. The investigation of user perceptions is a fundamental step in devising an evaluation framework that focuses on user needs and characteristics. In this study, the authors suggested a wide range of evaluation criteria in eight dimensions of digital libraries based on the document analysis. For the suggested evaluation criteria, this study examines to what extent users perceive the importance of each criterion in the evaluation of digital libraries.

Thus far, digital library evaluation criteria have been suggested mainly by librarians or researchers. To design a user-centered digital library, the evaluation needs to reflect users’ perspective in its evaluation criteria. This study is one of a few studies that investigated users’ perceptions of evaluation criteria for digital libraries.

This study intends to examine the following research question:

What are users’ perceptions of the importance of digital library evaluation criteria?

Identifying evaluation criteria is essential for the successful evaluation of digital libraries. Previous research suggested a variety of evaluation criteria in different dimensions of digital libraries. However, users’ perspective has not been sufficiently investigated in the digital library evaluation framework. The present study investigated users’ opinions on the importance of evaluation criteria for digital library evaluation. The unique contribution of this study lies in the comprehensive examination of users’ perception of evaluation criteria across different dimensions of digital libraries. The ratings of evaluation criteria showed the most and least important criteria from users’ perspective. Practically, the findings of this study assist library professionals in making their decisions in regard to selecting appropriate evaluation criteria for different evaluation objectives.

Different stakeholders identify their DL evaluation criteria based on their needs, background, their own interests, and familiarity with DL concepts. Users rank higher the criteria related directly to use of collections, such as authority in collections, or metadata accuracy in information organization, or the quality of services. Their rankings reflect the expectations of digital library users. Users think less about the cost and efforts required for building DLs, which are of concern to digital librarians. The top criteria selected by users indicate what they care the most in digital library use. For example, in information organization, they rated accuracy, consistency and in-depth of metadata as the top criteria. Accuracy is, of course, essential for users to actually use the metadata for their research and learning, and in-depth of metadata is useful for them to obtain as much information as possible for each digital item. In developing digital libraries, digital librarians need to consider users’ needs in regard to quality of collections, metadata and services. Interface design needs to offer multiple options for users to access the documents. At the same time, interface design needs to consider the special requirements from users with a variety of disabilities. Reliable and effective system performance is a key requirement for users to access digital libraries.

However, not all users are the same. The subjects of this study represent academic user group of digital libraries. In addition to general users’ perceptions of digital library evaluation criteria, they also have their unique needs and opinions because of their academic background. In the dimension of effects on users, they ranked research productivity and learning effects as their top choices. Research and learning are the two academic goals for this group of users. The design of digital libraries in academic settings needs to put research and learning as the priority in terms of collection, metadata and interface design. For example, librarians need to work with instructors to determine the types of metadata needed for learning purposes in the development of related digital collections. In providing digital services, librarians have to come up with ideas that tailor to the needs and characteristics of academic users. In the dimension of context, subjects of this study chose “information ethics compliance”, “copyright”, and “content sharing” that are important to academics as the most important evaluation criteria. Digital libraries in academic settings need to provide information related to ethics compliance, copyright information and content sharing options to assure and guide users in using of digital items.

Certainly, this study has several limitations. The sample size might not be sufficient to represent a variety of users of digital libraries even though they are real users of digital libraries. The results from the analysis of thirty users, including faculty, undergraduate and graduate students, cannot be generalized to general public users. Also, the authors were not able to conduct a comparison analysis between different groups (e.g., faculty vs. students) due to the relatively small sample size for statistical tests. However, the findings of this study yield insightful information on users’ perceptions of digital library evaluation. In addition, the present study investigated only the user group among different stakeholders. Although the ultimate objective of digital libraries is to serve users, end-users do not have sufficient knowledge of DL administration, collection development, or preservation techniques. Therefore, the investigation of other expert groups, such as digital librarians and scholars, is imperative to develop a comprehensive evaluation framework. Further analysis is going to investigate to examine opinions from other stakeholders including scholars and digital librarians. In addition, the authors plan to compare the opinions of those three groups to identify similarities and differences among them.