Search Capabilities for Users: A Research Agenda Paper

UCLA, National Science Foundation Workshop on Social Aspects of Digital Libraries February 16-17, 1996

The immanent availability of direct access by many millions of casual end-users to large volumes of electronically stored and delivered information resources raises three important problems that demand a new emphasis in research and development on search methods.

1. Current search methods have been developed primarily for well-trained intermediaries with both search and domain expertise. These methods need re-examination and, in many cases, replacement.
2. Digital technology will provide access not only to bibliographic information and document-surrogates, but to complete multimedia resources. Presenting such materials through the computer medium is a prospect full of promising new applications for search facilities and new, hardly explored, difficulties.
3. The networked digital medium also makes it possible to move instantly from one resource to another. Important issues in making such facilities useful and usable for search and consumption of information are in need of research.

1. More direct use by amateur end-users against a more heterogeneous information space. The end-users of information search technology will no longer be primarily library information specialists or even students working at OPACS in libraries where they have the assistance of such specialists. It is hoped that the new end user population will, instead, be everyone from everywhere working independently. Thus the difficulty posed by a plethora of different access, login, authorization, query language types and syntaxes is in great need of resolution. Current developments in Internet search methods have both simplified and exacerbated this problem. As market-driven evolution gradually resolves into defacto standards it is to be hoped that research will help to guide the winning inventions toward techniques that are optimal for as many users as possible. Consider these facts. A typical user initiated search on large bibliographic systems starts with a single term subject search that returns an average of 400 document pointers, and is followed by a two-term Boolean AND search that returns nothing. There are at least three reasons (see Landauer et al., 1993 for review).
   1. The average college educated person cannot form a correct Boolean expression for even a simple case.
   2. There are typically 30 different words that users will think of to describe an information object they know well, but it will usually be indexed by less than five, so Booleans would seldom suffice even if understood.
   3. Casual searchers often know little about the topic and its vocabulary, and will know even less as they gain access not only to collections suitable to their level of expertise (e.g. highschool library collections) but to the union of all collections at all levels, plus, potentially, ad hoc materials and junk contributed by any and all.

   Thus the new population of users will need not just a system that is formally sufficient and computationally efficient, but one that will be very helpful to them in their state of relative ignorance and ineptitude. Professional searchers have been provided with classification systems, controlled vocabularies and training in search methods, collection characteristics, and much more. Since such training for all is unlikely and unattractive, we need to press known techniques such as natural language interfaces, Latent Semantic Indexing (LSI), adaptive indexing, and, especially, relevance feedback into wider service, and we need research to find new and better methods and combinations of methods. Such research must be guided by empirical studies of what works for ordinary people doing occasional interactive search.

   The organizers' background paper suggests that users be given a variety of different specialized search tools. I agree with their identification of the problem, and with the notion that the new environment calls for new search techniques, but I worry about requiring users to know how to choose and operate many different tools. Accuracy and speed of choice among options decreases as their number increases, multiple ways of doing the same thing confuses people and makes it harder for them to learn, and given several methods, users do not always make optimal choices. An alternative solution is to find ways in which the system can use multiple search strategies and let the user act as a supervisory guide by some form of relevance feedback. For example, objects might be identified by meaning or topic (as in LSI or thesaurus-based systems) and also by exact term matches, and the user asked to choose the best among a sample of returns. On the other hand, it would seem a manageable burden to let users make some prior decisions, e.g. whether they want books by Newt Gingrich or about Newt Gingrich. The best mixture needs resolution.

   It is clear that the new user population will need more help. Help might be provided by traditional expert intermediaries working by email and using a greatly expanded manual effort to categorize and catalogue the burgeoning universal collection, or by "intelligent agents" designed to translate protocols and query languages into a common form and to find and list suitable collections, or by "virtual community" recommenders that find things of interest to user A by first finding users B through Z whose choices predict what A will like, by some facilitated form of broadcast request for references from e-peers, or by other techniques waiting to be invented. I think what is needed now is trial of many such techniques and a method of empirical evaluation of their value that will beat the market in speed and accuracy. A particular dimension on which much more help than has been available before, at least by automated means, is determining what found resources are at a conceptual level and in a vocabulary that will be appropriate to the background of the searcher.

   Another important topic for research effort is cross-language retrieval; the e-world is international, and there are promising techniques (e.g. we've found that LSI works well, Landauer and Littman, 1990), but more support, effort, and, especially availability of parallel test corpora is needed to make them a reality.

2. Searching and filtering in the context of electronic delivery. We will want the (potentially multimedia) resources that are found to be delivered by computer. Some such resources will be of significant size, for example whole textbooks. Reading from a computer screen is significantly slower than from print on paper and the majority of interfaces for reading, largely because of limited space, lead to frequent disorientation and loss of contextual embedding. On the other hand, the ability to provide search within a document, and from a document to accessory sources such as dictionaries, maps, and cited sources can be a powerful aid. However, we need to know much more about how to apply these techniques to make them highly useful and usable. With only a few exceptions, evaluations of online text and graphics browsers have found them inferior to corresponding paper and print presentation for finding information and learning (see Landauer et al. 1993; Landauer, 1995). The successful cases have used some method for preserving orientation and context, such as SuperBook's structured search feedback and dynamic table-of-contents. They have also greatly downplayed arbitrary hypertext links, which seems a great shame given their apparent potential utility. The problem again appears to be the maintenance of context and discourse coherence. Kintsch and collaborators have shown that comprehension and learning depend critically on these factors (see Kintsch, 1994). Explorations of how to effectively exploit freedom from bound pages without incurring such disadvantages are greatly needed. Among the possible avenues are techniques for labeling and explaining links, for assessing the conceptual coherence of link targets and of maintaining context and organization despite disruption. It think we also may need to face the fact that link following may need to be discouraged in many instances, just as in current print practice expert judgment dictates that footnotes be suppressed or minimized in some kinds of materials for some audiences. We need research to better understand these tradeoffs, especially so, I believe, in educational applications.

3. Search and filtering aspects of hypertext. I have already addressed some of the issues and research needs here. Context and coherence problems are especially serious in the common hypertext usage situation in which users are tempted to jump from one paragraph, document, or collection of documents to any of many quite different ones. Because the new source can be radically different in its assumed knowledge and vocabulary, the user will also need to be properly informed or adequately steered in order to be able to understand or learn from the next segment, unless the hypertext author has taken pains that every possible path maintains optimal conceptual transitions, a daunting task even when possible. Again, the work of Kintsch et al. (Kintsch, 1994) make it clear that comprehension and learning are dramatically dependent on matching the conceptual sophistication of materials to the prior knowledge of its consumer.

   Another research issue for hypertext is finding non-text sources. Here I think we need research to see if people more effectively describe the pictures and music they seek in drawings and hummed tunes, in features-list choices, or by descriptive phrases about the semantics of the object. Until we know that, which we might learn from simple Wizard-of-Oz experiments, implementation of sophisticated search schemes seems premature.

   The issue of search by hypertext links is an interesting footnote here. Database query began with link following, a technique that was almost immediately abandoned, at least in part because of the unsatisfactory search results people obtain. Many designers of hypertext have come to the realization that better search methods are needed, and that awareness has also recently reached the Internet facility implementing community, but there is still much left to do in discovering optimal methods and applying them consistently.

Formative evaluation and iterative design.

References

Egan, D. E., Lesk, M. E., Ketchum, R. D., Lochbaum, C. C., Remde, J. R., Littman, M., & Landauer, T. K. (1991). Hypertext For the Electronic Library? CORE Sample Results. In Hypertext '91 Proceedings, (pp. 1-14). Bellcore.

Kintsch, W. (1994). Text comprehension, memory, and learning. American Psychologist, 49(4), 294-303.

Landauer, T. K., Egan, D. E., Remde, J. R., Lesk, M. E., Lochbaum, C. C., & Ketchum, D. (1993). Enhancing the usability of text through computer delivery and formative evaluation: the SuperBook project. In C. McKnight, A. Dillon, & J. Richardson (Eds.), Hypertext: A Psychological Perspective (pp. 71-136). New York: Ellis Horwood.

Landauer, T. K. (1995) The trouble with computers: Usefulness, usability and productivity. Cambridge, MA. MIT Press.

Landauer, T. K. and Littman, M. L. (1990) Fully automatic cross-language information retrieval using latent semantic indexing. Proceedings of the 6th Annual Conference on Electronic Text Research, University of Waterloo Center for the New Oxford Dictionary

[ Return to Digital Libraries Workshop ]