Preprint – Article
accepted to appear in The Library Quarterly
Digital Library Archeology: A Conceptual
Framework for Understanding Library Use through
Artifact-Based Evaluation
Scott Nicholson, Assistant Professor
scott@scottnicholson.com
Abstract
Archeologists have used material artifacts found in a physical space to gain an understanding about the people who occupied that space. Likewise, as users wander through a digital library, they leave behind data-based artifacts of their activity in the virtual space. Digital library archeologists can gather these artifacts and employ inductive techniques, such as bibliomining, to create generalizations. These generalizations are the basis for hypotheses, which are tested to gain understanding about library services and users. In this article, the development of traditional archeological methods is presented and used to create a conceptual framework for the artifact-based evaluation in digital libraries.
Introduction
For hundreds of years, scientists have worked to understand our history through the “recovery, systematic description, and study”[1,12] of artifacts left behind by a particular culture. This field of study, archeology, has several tasks, one of which is to reconstruct the “lifeways of the peoples responsible for the archeological remains” [2,6]. Archeologists may know little about the people themselves who occupied a physical space; instead, they examine artifacts left behind for patterns to understand the communities who lived there.
There are similarities between a physical place and the virtual space supported by the Internet. Communities of people form, grow, evolve, and dissipate in this virtual space. As users travel through this virtual space, they leave behind data-based artifacts. These shards of virtual pottery such as searches and browsing behavior, and burial mounds of dead discussion groups hold valuable information if the pieces can be collected, cleaned, organized, and examined. This information can give “Internet Archeologists” an idea of the communities and cultures that have existed in virtual spaces.
A framework for Internet Archeology was broadly introduced by Nicholson[3]. In order to focus this concept, the discussion of Internet Archeology is limited here to a single type of virtual place – a digital library. The Digital Library Federation defines digital libraries thusly: "Digital libraries are organizations that provide the resources, including the specialized staff, to select, structure, offer intellectual access to, interpret, distribute, preserve the integrity of, and ensure the persistence over time of collections of digital works so that they are readily and economically available for use by a defined community or set of communities" [4]. Much like the individuals living in the past, many users of digital library services may be understood only through the artifacts left behind from “living” in the space.
What are these artifacts of digital library use? In this case, each “artifact” is one record on a server about a single access of a resource by a user. When a user visits a digital library, they will travel through several services. They may start with a library Web page, move to a proxy / login server, an OPAC, a database, a digital resource, and then go out to a vendor’s site. At each server, the user leaves behind a small data-based artifact of their step through that virtual space. By collecting individual artifacts from different servers and connecting them through date/time information or other connecting fields, the digital library archeologist can reassemble these artifacts, reconstructing the user’s path through the digital library. This can have two effects: first, it allows the library decision-makers to better understand how people are using the system; second, by connecting these data artifacts into a large data warehouse, many more patterns between library systems can be discovered through pattern-discovery statistical and analytical tools such as bibliomining [5].
Figure 1 shows artifacts as seen by a traditional archeologist and artifacts as seen by a digital library archeologist. These digital artifacts are from different Web server logs, which include (a) who came to the digital library, (b) where that individual came from, and (c) what that user did in the library. These are just examples – other possible sources include Web proxy logs, OPAC searching histories, metadata about works in the collection, and patron demographics.
|
Traditional Artifacts |
Digital Library Artifacts [6] |
|
|
(a) 07/09/99, 13:59:24, ,
999.999.99.99, scooby.northernlight.com, crawler@northernlight.com,
Gulliver/1.2 (b) 08/02/99, 12:02:35,
http://ink.yahoo.com/bin/query?p= ”sample+log+file”&
b=21&hc=0&hs=0,999.999.999.99, jaz.med.yale.edu (c) gateway.iso.com - -
[10/MAY/1999:00:10:30 –000] “GET /class.html HTTP/1.1” 20 10000 |
Figure 1: Artifacts of Traditional and Digital Libraries
Just as pottery shards are gathered, washed, and reassembled, digital library artifacts will have to be gathered, cleaned, and reassembled to gain an understanding of the users who passed through that space while still protecting the privacy of the user. These artifacts may have to do with the behavior of use, the demographics of the user, or features of the item being used; when reassembled through a data warehouse, the collection of these artifacts will provide a better understanding of the digital library user and transaction than the pieces taken separately. One pioneer in digital library archeology is Joseph Zucca at the University of Pennsylvania, who has created a data farm where the artifacts of use are gathered, cleaned, matched, and privatized [7].
The completed plate from a combination of shards is akin to a reassembled session of use; not all of the pieces have to be found to provide a reasonable idea of the original item. In fact, removing the piece that contains personally identifiable information about a user can still leave enough information to create generalizations about use while still leaving connections between other data-based artifacts (see visual analogy in Figure 2).
|
Reassembled Data-Based Artifacts |
Reassembled and Privatized Data-Based Artifacts |
|
|
|
Figure 2: Visual analogy for combining data-based artifacts to understand library services and use
One definition of science is that it is a method of organizing knowledge of a subject to aid understanding [8]. The goal of this paper, therefore, is to present a framework based on archeological theories in order to aid in the understanding of library users through the artifact-based measurement and evaluation of digital library services.
One method used in the measurement and evaluation of library services is the search for patterns; this can be done manually or with the aid of statistical and data mining tools. The concept behind bibliomining, or the combination of bibliometrics and data mining to understand library services, is similar, in theory, to the pattern-discovery concepts in archeology [5]. Nicholas et al., at the CIBER research center, has focused on understanding the user through the artifacts left behind in a system [9].
Review of Related Models
The concept of an information artifact for analysis of an information retrieval system was defined by Green and Benyon as “any artifact whose purpose is to allow information to be stored, retrieved, and possibly transformed” [10,803]. They proposed ERMIA, or the Entity-Relationship Modeling for Information Artifacts as a way of using data modeling to look at the key elements of a system and a user’s manipulation of the system. The concept of artifact in the present archeological exploration is much more narrow, focusing on those pieces of data left in a system from a user’s interaction.
Griffiths, Hartley, and Willson [11] presented a model for understanding user-system interaction through a combination of transaction log analysis and verbal protocol analysis. This technique starts with traditional log analysis techniques where the selections made by users in a system are recorded. This is supplemented with a recording of mouse movements on the screen. A talk-aloud technique is then used at the same time to collect statements made by the user about their exploration. The result is a more complete picture of the user’s interaction with an information system.
This more complete picture of user-system interaction is related to the holistic matrix of library measurement presented by Nicholson[12]. This model for library measurement and evaluation is useful in understanding where a particular type of library measurement falls in perspective with the larger body of measurement and evaluation literature. The base of this model is the measurement matrix (see Table 1), which divides measurement of library services into four classes.
Measurement |
Topic
|
|
Perspective |
System
|
Use
|
|
Internal
(Library System) |
Procedures Standards |
Recorded interactions with interface & materials-Bibliomining- |
|
External
(User) |
Aboutness Usability |
Knowledge states Value of works |
Table 1: Measurement Matrix from Nicholson's holistic measurement framework
This holistic framework is related to the model of Evidence-Based Librarianship (EBL). The concept behind EBL is that librarians should seek ways to integrate the best available evidence into their decision-making processes. Library evaluation should start with a research question, seek out evidence within the literature for assistance, and use research methods that reduce bias in order to collect evidence useful not only in resolving the original research problem, but that may be useful in other decision-making situations[13].
EBL was inspired by Evidence-Based Medicine, which relied upon a large amount of medical research in order to encourage physicians and decision-makers to take advantage of the available research. Library Science has yet to build a corpus of research similar to that of medicine. Therefore, the evidence for EBL has to come from other methods of collecting evidence about library services.
The focus of the present work is to understand the interaction between a user and electronic resources through a digital library service that occurred in the past. One science used to gather evidence about the past is archeology. Therefore, in order to perform Evidence-Based Librarianship, the present work explores archeological thinking as a way of considering our exploration of the data-based artifacts left behind in a digital library system, and considers the data warehousing and data mining process known as bibliomining as a primary tool in that exploration.
Meedows considered a link between archeology and information science from the perspective of the world of scholarship[14]. His work explored the concept of a scholarly work as an artifact and the exploration of these artifacts through bibliometrics will provide an idea of the scholarly community. The current work differs from this piece in that the artifacts here are primarily of use, and the subjects of the research are the users of the digital library. Therefore, Meedows’ proposal for archeology of the community of scholarship is complementary, although different, to the work presented here.
Archeological Theory
Archeology is much more than the gathering of artifacts from the past. These artifacts actually exist in the present time, and the past is only created when researchers talk about it [15]. The task of archeology is to make an educated guess about what the past was like based upon the artifacts that exist in the present. Johnson emphasizes that archeology is not the collection of materials; rather, archeology is the interpretation of the meaning behind those materials.
Differences in archeological theoretical frameworks stem from the different methods of developing these interpretations [15]. In the present article, three different frameworks that trace the development of modern archeology will be presented and applied to digital libraries: traditional archeology, new archeology, and postprocessual archeology.
Traditional Archeology
Before 1960, archeology was based in antiquarianism: it started with the collection and thorough description of large amounts of artifacts from a site. After collecting and describing, archeologists explored the implications for the cultures that occupied the area based upon these artifacts. They created chronological maps of the cultural changes in one area or diffusion maps that emphasized how one cultural change moved from place to place. The focus was mainly on the description and mapping of artifacts and a discussion of the culture [15].
This emphasis on antiquarianism was natural, as those who funded archeological expeditions desired the collection of these artifacts. As a side benefit to the collection, archeologists conjectured about culture and diffusion. However, archeologists began to feel that there was little advancement in the field. While they were gathering and describing artifacts and culture, they were not advancing archeological theory that supported how people lived [2].
Traditional Archeology and Digital Library Evaluation
This scenario is similar to the current state of digital library evaluation. Most reports of digital library usage are aggregations of user groups and materials accessed. The usage data are collected, grouped, and described much in the same way that archeological artifacts were collected, tagged, and grouped. Some discussion of the users involved might be explored, and then the results are presented or published, just as artifacts are presented for others to view and enjoy. Many contemporary digital library evaluations published today fall into this “gather and describe” cycle [16,17] although some researchers, such as [9,18] have developed more complex research explorations.
New Archeology
Traditional archeologists collected and cataloged artifact after artifact, creating more of the same types of descriptive culture maps. The addition of more data, however, was not advancing the knowledge base as little high-level analysis was occurring. Throughout the 1960s and 1970s, a shift occurred in archeological thought. The Binfords [2] are attributed with one of the first significant writings on the topic of new archeology; in reality, the movement was a methodological adaptation that spread over time through the body of archeological thought inspired by a common dissatisfaction with the traditional methods [8].
Traditional archeology was focused on the material artifacts and the concept of a “culture” and not on the individuals behind those artifacts and those cultures. The call came for archeologists to be both more scientific and more anthropological in their studies. New archeologists no longer used “culture shift” as the only reason why anything changed; instead, archeologists were pushed to consider the individuals involved and the culture as a system made up of individual (and mutable) components [2].
There were several new types of archeological thinking that came out of this new archeology. The first was generalization through systems theory – by looking at culture as an interrelated system of sub-components, it became possible to generalize findings from one culture system to other culture systems that shared those components. Along with this new way of thinking about the internal components of a culture, archeologists began to seriously consider the external forces other than culture and other cultures [15].
Another shift was in the nature of archeological science. Instead of describing what was found at a site, archeologists began to ask why these items were found at the site. The goal became to understand the underlying process that caused these artifacts to end up where they did rather than focus solely on the artifact. These types of questions led archeologists to begin employing the scientific method; that is, presenting hypotheses about a phenomenon and testing those hypotheses through gathered data. Employing this method also required archeologists to be explicit about their definitions and biases [15].
Hypothesis-based archeology led to a new type of artifact collection. Instead of the goal of traveling to sites to collect large numbers of artifacts, archeologists chose sites and excavation strategies with specific research questions in mind. Sampling methods were employed in site selection and efforts were made to be representative in selections. Artifacts were collected to aid understanding of an issue and not to fill the storehouses of a museum [15].
David
Clark, in his classic work Analytical Archaeology,
focused on an archeological model with three portions – data recovery,
systematic description, and model/hypothesis development. In order to achieve
these goals,
According to Kaplan [19], the discovery of patterns is one way of creating new scientific laws. Some might argue that researchers are more interested in what makes people different rather than what makes them the same. However, it is only possible to understand differences between people by contrasting them through their similarities. Kaplan states that “how we conceive of an individual is the product of generalizations . . . To understand a person or a particular configuration of an event is to know something of what kind of person or happening it is; if we have no generalizations to draw on, no kinds are available to use for knowledge of individuals.” [19,118] Therefore, discovering these patterns and using them as the inspiration for hypotheses (and eventually laws) are essential to understanding differences.
Binford [2] presented three types of theory developed from archeological studies. The first, Low Range Theory, is applicable only to a single setting and explains one aspect in that setting. A Middle Range Theory can be applied beyond the setting in question, and works to explain an aspect across settings. These theories are “generalizations that attempt to account for the regularities that occur between two or more sets of variables in multiple instances” [20,21]. Finally, Upper Range Theories apply to all settings; Binford felt that archeology could not produce Upper Range Theory.
New Archeology and Digital Library Evaluation
This evolution in archeological theory can aid researchers in thinking about ways of evaluating digital library services and understanding the use of those services. New archeology encourages researchers to move beyond the collection and description of artifacts; therefore, library evaluators need to move beyond presenting basic summaries of their data-based artifacts as the final product of an examination. The next step is to look for patterns, or “systematically correlated attributes that give recognizable group identity” [1,21] in order to seek out subgroups of users and behaviors. Digital library archeologists create a framework of understanding from these generalizations, and researchers can then develop testable models and hypotheses based on that framework to better inform the library decision-making process.
It is important to think about the library as a system of components in order to aid generalization. By focusing an examination on one component, then researchers may be able to create research questions that will lead to building theoretical knowledge for library use. Systems theory suggests that care must be taken when generalizing results; if the results from measuring one aspect of one library subsystem are affected by another subsystem in the library, then results may not be easily generalizable [21]. One way to resolve this conflict is to test the same subsystem in different library settings; if the results are consistent in different library subsystems, then researchers can be more confident in the generalization of results.
In
addition, researchers should seek ways of comparing system use across different
settings. One track of existing research in this vein is Evidence-Based
Librarianship, which uses meta-analysis to combine studies to produce more
generalizable results [13]. Through these additional
steps, researchers can advance the theoretical understanding of library use and
therefore advance the science of librarianship through the development of
Finally, library evaluators can benefit from hypothesis-based research as they can answer questions beyond what is available from the artifacts left in the system. The artifacts provide insight only to how the user manipulated the system; they do not allow understanding of other aspects of use, e.g., what the user was thinking during the process, why the user made those choices, what else the user wanted to do, if the user was satisfied, or if the information need was met. Since a researcher cannot determine many user issues from only the artifacts left behind, then a scientific approach involving an educated guess (a hypothesis) and a research question followed by a study where users are involved may shed light on the issue.
This need to move from a practical evaluation toward a hypothesis-based exploration to improve the science of librarianship has been voiced explicitly and implicitly by other library scientists. As McClure states, “library and information science fosters little research that is intended to produce ‘knowledge for the sake of knowledge’”[22,17] and focuses on the gap between the generalizable research of library scientists and the applied action research desired by librarians. He argues for the need for ways to increase the impact of research on libraries. Tenopir [17] examines many significant large-scale library evaluations and find that most of them draw conclusions only about individuals or specific groups of users. She also develops numerous generalizations based on the collection of studies; this is a perfect example of the next step in generalizing results from traditional library evaluations and could serve as the inspiration for many testable hypotheses.
Combining Traditional and New Frameworks
Up to this point, two different archeological frameworks have been presented along with some discussion of how each could be applied to bibliomining. Just as archeological research based on both frameworks has added to the understanding of the past, evaluators looking to understand more about library users can employ a combination of these frameworks in analyzing the data-based artifacts of use. There is such a combined framework for scientific archeology that is appropriate (which is also used in many other branches of science) - the hypothetico-deductive-inductive scientific cycle.
Hypothetico-Deductive-Inductive (HDI) Scientific Cycle
This cycle, first outlined by Kemeny[23] , was applied by archeology by South [24] and provided the bridge between traditional and new archeology. Traditional archeology focused on describing and finding patterns within the data, and new archeology started with a problem and sought data to support or refute hypotheses. Scientific archeology employs this HDI cycle to connect these two frameworks.
In an archeological site, just as in the logs of a digital library, there is a large amount of data available. The first step is to collect samples of data from around a site and explore those data for patterns. Inspired by these patterns, the researcher creates basic generalizations about the data. These generalizations may be based upon cultural variables or other demographic/geographic factors. These generalizations are examined for gaps and commonalities (along with data and findings from other studies) in order to develop hypotheses about the people who make up these groupings.
Research questions are created to explore the hypotheses, and then additional data are gathered to test those hypotheses. These data may come from the same source or may require different sources. This method may support or refute the hypotheses, which has the effect of building the knowledge base for the field.
New archeology focuses on the second half of this cycle and functions well when based on the artifacts from traditional archeological methods. Both methodological frameworks are useful in building knowledge from the successful exploration site – the original data power preliminary findings, and researchers go beyond the data to create and test hypotheses. Upon moving to a new site, the process begins again, and scientists can combine knowledge from previous explorations when deducing new research questions and hypotheses to explore [24].
South presented this hypothetico-deductive-inductive cycle in a graphical chart, called the Dolphin chart, which used a dolphin hopping out of the “The Particularistic Sea of Observed Facts” into “The Nomothetic Atmosphere” and back into the sea. During its trip in the air, the dolphin goes through the stages of “Induction (Pattern Recognition), Theory (Lawlike Generalty), Deduction (Logical Analysis), Prediction (Hypothesis), and
Verification (Testing)” [24,19]
This methodology is applicable to the evaluation of digital library services. Currently, many researchers are in the first few phases, akin to traditional archeology. Most evaluators of digital library services gather data and describe the data through graphical and basic statistical analysis. Many of these evaluations stop at this point, although some go further and discover more descriptive patterns and generalizations beyond raw frequencies and averages. The challenge, therefore, is to move on to the types of tasks inspired by new archeology. These patterns and basic generalizations can be analyzed to create problem statements and hypotheses about digital library use. Research studies can be performed to explore these hypotheses and further our scientific knowledge about digital libraries.
Postprocessual Archeology
Along with postmodern movements in other fields, some archeologists claim that the rigid scientific method presented cannot capture the nuances of the individuals involved with the culture system. As Wylie said, “archeologists would not know (could not determine) whether, or in what respects, past contexts diverge from hypothetical reconstructions of them given the nature of their evidence” [25,21]. Another concern is that if the artifact data can only be interpreted in certain contexts, as the new archeology stated, then a set of data could produce different results under different contextual assumptions [25].
The term postprocessual does not refer to a single method; rather, many different frameworks of archeology take on the postprocessual label. Up until this point, there was a reasonable amount of consistency in the way archeological science