Reading notes for Evaluation

The link to Arms' chapter didn't work, so I looked it up elsewhere and found a 1999 version at the following web address: http://www.cs.cornell.edu/wya/DigLib/MS1999/Chapter8.html
I assume it's not too outdated!

It gives a good basic introduction to the principles of a user interface, and the ways that an interface should change along with technology over time. The conceptual model that this chapter proposed was pretty straightforward, as was its review of browser technology. Most of the chapter's points are ones I was already familiar with, e.g. that a web designer must balance effective use of advanced or sophisticated features with the ability to offer simplicity and speed for less well-equipped users. I was also already familiar with mirroring and caching -- and was interested to see that when this article was written (presumably in 1999) video skimming was mostly merely an idea for future development. What I found most interesting were the chapter's brief references to the writer's own experiences, such as the fact that his online magazine redesigned its interface yearly.

Kling and Elliott's article brings a focus to usability concerns in designing an interface; they recognize that ease of use improves users' performance. They break ease of use down into four components:

Learnability - which also concerns the speed with which a user can begin using the software

Efficiency - how productively a user can make use of the system

Memorability - whether the user can easily return to using the system after an absence

Low error rate - no catastrophic errors and easy recovery from the minor ones

Clearly an intuitive system organization that works well on a server will lead to the best results. Given that users of digital libraries will have all sorts of different goals and intentions, it's probably best for system developers to survey users frequently to determine the areas in need of improvement.

For organizations, the authors break down concerns as follows:

Accessibility - the ease with which people can locate specific systems and content, both physically and administratively

Compatibility - of file transfers between systems

Integrability into work practices - how smoothly the system fits existing practices

Social-organizational expertise - how well people can obtain training and consulting to learn to use systems and troubleshoot

Unsurprisingly, many of the authors' recommendations for digital libraries involve testing systems, surveying users, exploring multiple design alternatives, etc. They implore us to pay attention to cultural models of user bases, reminding us that a system appropriate for elementary schoolchildren will not be as appropriate for graduate-level science laboratories.

Finally, Tefko Saracevic's article evaluates evaluation: analyzing the methods and contexts of the (relatively rare) evaluation of several different digital libraries. The article goes into details about the evaluative methods that were used, and highlights the variety of approaches possible: usability-centered (as with the article above), ethnographic, anthropological, sociological, and economic. The article highlights many distinct matrices of assessment, and briefly acknowledges that despite many factual criteria there is also the role of human judgment in certain evaluations. Digital libraries are fairly new, so it is understandable that not much evaluation has been done on them, but one of the take-home messages of this article is that despite apparent lack of interest and definite lack of funding, evaluation is important and should become a bigger part of digital library culture.