Appears in Proceedings of the Symposium on Designing Interactive Systems, Ann Arbor, MI, August, ACM Press, New York, 1995, pp. 43-53.

A Framework For Developing
Experience-Based Usability Guidelines

Scott Henninger, Kyle Haynes, Michael W. Reith
Department of Computer Science & Engineering
University of Nebraska-Lincoln
scotth@cse.unl.edu

Abstract


Reflecting the growing consensus that principles and methods for developing effective interfaces are beginning to mature, usability design guidelines have begun to proliferate. But current approaches to guidelines tend to either be technology-centric, focusing on platform-specific interface widgets, or abstract and general-purpose. At best, these general guidelines provide weak support that is insufficient to support developers faced with specific interface design problems targeted for specific user populations.

If the potential of usability guidelines as an interface design technique is to be fully realized, they need to be augmented with context-specific guidelines and examples that synthesize isolated guidelines into domain-specific solutions to design problems. In this paper, we present a method in which software development organizations can develop and evolve domain-specific guidelines based on the kinds of applications they develop. The method facilitates the process of determining when and how guidelines should be applied by tying guidelines to specific design cases and providing the means to match customer requirements to specific interface techniques that have proven effective for similar users and application domains. The concrete cases help designers interpret the guidelines, making them easier to comprehend and apply to the current design problem. We demonstrate these issues through Mimir, a case-based system that supports the refinement and location of relevant guidelines and cases.

Keywords: Design methodology, organizational memory, usability guidelines, case-based reasoning.

Table of Contents


Introduction
Existing Approaches to Usability Guidelines
The Limited Scope of Interface Design Guidelines
Finding and Understanding Relevant Guidelines
Guideline Examples as a Development Aid
Tailoring Guidelines for Specific Applicability
An Organizational Memory Approach to Developing Design Guidelines
A Case-Based Approach to Organizational Learning
Using Experience-Based Design Guidelines
Migrating Data to a Client/Server Database
Using Appropriate User Terminology in Interface Designs
Extending and Specializing Guidelines
Capturing Guideline Cases
Assessment and Future Directions
Related Methods
Conclusions
References

Introduction


Developing effective, useful and usable interfaces is increasingly becoming a primary issue for assessing the quality of software applications [Myers 94]. To many developers, the answer lies simply in presenting the user with a Graphical User Interface (GUI, or "gooey" in developer vernacular). GUI and application builders have proliferated, further deepening the misconception that user interface issues are restricted to widget layout and look-and-feel principles provided by these development environments. While GUI builders give developers easy ways to implement interface widgets, they provide no guidelines for how they should be used to develop a useful, usable interface. They do not magically transform developers into good interface builders, and in many respects merely expose the developer's ignorance faster, allowing them to proliferate bad interfaces.

The issues involved in creating an effective, useful, and usable system reaches far beyond the superficial treatment of interfaces as a facade to functionality [Gould, Boies, Lewis 91]. The "interface" is the window to the services provided by the system, how it helps people accomplish their tasks, reflecting "not only the interface design but also the conception of the application itself" [Laurel 93]. Far from being solely a matter of screen layout or widget sets, the interface defines what the system is, what it does. Interface design is not so much about menus, icons and widgets as it is about people, tasks, organizations and how the system fits into the web of commitments and tasks of users. Developers must strive to understand the needs of their users and the context in which applications are placed.

Interface design guidelines are becoming a popular way to convey known principles of human-computer interface design. Guidelines have been developed for both platform-independent design guidelines [Smith, Mosier 86; Brown 88] as well as platform-specific style guides [Apple 93; Microsoft 94]. The goal of these efforts is to provide reference and guidance for designers during the process of design [de Souza, Bevan 90]. Ultimately, the use of guidelines in the development process could shorten and reduce (but not eliminate) the number of iterations involved in the design-evaluate-redesign cycle of HCI development [Strong 94]. But these guidelines are notoriously thick and heady material that have had little impact to date [de Souza, Bevan 90; Gould, Lewis 85; Tetzlaff, Schwartz 91]. Guidelines tend to be stated either at a high level of abstraction, making them vague and difficult to interpret in terms of a specific design context [Tetzlaff, Schwartz 91], or at the level of interface widgets, making it difficult to design interaction strategies for different kinds of users.

There have been some attempts to make interface guidelines more accessible through hypertext-based search systems [Iannella 94; Alben, Faris, Saddler 94; Perlman 89]. Providing hypertext and retrieval interfaces can provide access to guidelines, but formidable problems remain with respect to defining when guidelines are applicable or how guidelines can be refined to meet user task requirements for a specific set of users and a specific type of application. Work on usability issues has become voluminous to the point that it is difficult to determine which principles are applicable to a given design problem, and the continuing proliferation of computer technologies make it difficult to find a good fit [Poltrock, Grudin 94]. But little research to date has been done to accumulate knowledge about interface design in a form that can be proactively applied to the development of interactive systems [Strong 94].

This paper presents a framework designed to take interactive guidelines such as Making It Macintosh [Alben, Faris, Saddler 94] and HyperSAM [Iannella 94] a step further, turning them into an "organizational memory", a repository of usability experiences or "living design memory" [Terveen, Selfridge, Long 93] for interface design. Our approach begins with a hypertext representation of interface guidelines that are adopted by a development organization. The guidelines are then augmented by case-based representations of usability requirements that help find, explain, specialize, and extend the guidelines. The result is an interactive case-based system supporting a continuous improvement process [Paulk et al. 93], that helps developers design effective, useful and usable applications, and can contribute to the systematic understanding of factors that contribute to the success or failure of an interface design [Strong 94].

Existing Approaches to Usability Guidelines


Techniques for improving the quality of user interfaces have been researched in the CHI community for a number of years, and are as diverse as the many ways in which computers are used. While some would argue that the validity of guidelines are questionable, having been derived from small studies in artificial settings [Thimbleby 90], or that the field is too immature to offer accurate recommendations for design [Rosson, Maass, Kellogg 88], interface design guidelines have are becoming increasingly popular. In many respects, this represents the maturation of the HCI field. But the combination of diverse user backgrounds, diverse application domains, and proliferating technology solutions makes it difficult to have a single, coherent, concise, and universally applicable theory as the basis for interface design. The problem is not that there is insufficient information to guide HCI design, but that the diversity of concerns creates an overabundance and it is difficult to determine what information is applicable and when.

The Limited Scope of Interface Design Guidelines


Currently guidelines generally fall into one of two groups: technology centric and general. Technology-centric guidelines tend to be platform dependent and focus on interface widgets [Apple 93; Microsoft 92], such as dialogue boxes, pull-down menus, screen layouts, and naming conventions. For example, about 88% (essentially all but the first three chapters) of Apple's Macintosh Human Interface Guidelines are devoted to the discussion of "Interface Elements." Microsoft's The Windows Interface: An Application Design guide devotes only 3% to interface principles [Microsoft 92]. User issues are treated in the abstract, with little to no discussion on how interface objects and interactive techniques can be used to solve specific tasks. The focus is instead on the minutiae, such as the difference between using an ellipse character and using a diamond mark character in a menu item [Apple 93]. Questions such as when a particular widget should be used or how the interface elements integrate together are left unanswered.

General guidelines attempt to create universally applicable, platform-independent, guidelines [Mosier, Smith 86; Brown 88; Heckel 91] that provide advice along the lines of "Allow the user to control the dialogue." At some level, this is sound advice, but it lacks the contextual information to assess how the guideline can be applied to a given set of circumstances. For instance, stating that an interface should 'provide feedback' is a useful and well-accepted guideline, but it is unclear how this could be applied to, say, a system for planning railroad schedules. Questions such as how to apply the guidelines to a specific set of circumstances are left unanswered.

The tendency for guidelines to be abstract and technology-centric is understandable because they must be applied to a wide variety of applications to maximize their utility. To achieve this general applicability, a least common denominator must be found. This naturally settles at levels such as widgets and high level abstract guidelines, which inherently have universal applicability. Therefore, the focus is on attributes of the interface, which misses important information on what users need to do to solve their problems with the application.

Both technology-centric and general guidelines provide some useful information along complementary dimensions, but fail to guide interface developers on how the guidelines match to different kinds of users, tasks, and application domains. As opposed to seeing interfaces as an interactive medium between people and computers, these guideline techniques focus on usability as attributes of the user interface at the expense of understanding what motivates users [Curtis, Hefley 94, p. 28]. If the potential of guidelines as an interface design technique is to be fully realized, they need to be augmented with guidelines and examples that synthesize both kinds into task or domain specific guidelines based upon the needs of specific kinds of users.

Finding and Understanding Relevant Guidelines


While guidelines are often considered useful, problems with finding relevant guidelines and choosing among alternatives have limited their effectiveness [Mosier, Smith 86]. This means that systems supporting guidelines must support the process of locating relevant guidelines and determining which guidelines are applicable under a given set of circumstances. Studies have also revealed difficulties in interpreting guidelines [Tetzlaff, Schwartz 91], as well as translating the general wording endemic to guidelines into specific design rules that are immediately applicable [Mosier, Smith 86]. This can be done by linking the guidelines with examples stored in the organizational memory. By either directly reusing these examples or using them as metaphors that concretely guide the design process, developers can more readily interpret and use appropriate guidelines.

Guideline Examples as a Development Aid


A study of developers applying usability guidelines to design paper and pencil interfaces revealed that examples of guidelines were referenced more frequently than the guidelines: "Examples were inspected carefully, were frequently referenced, and were copied directly in constructing the design. Participants all felt there should be more of them" [Tetzlaff, Schwartz 91, p. 331]. The examples were used not just to illustrate the text, but were also copied directly into the design.

Participants in the study with some experience in design found general human factors guidelines uninformative, perhaps being seen as preaching to the choir. But "strategies for approaching the design were welcome, and were consistently and carefully considered" [Tetzlaff, Schwartz 91, p. 331]. Participants also found it frustrating that guidelines were treated in isolation with no real attempt to describe the effects of combining multiple and potentially conflicting guidelines together. Examples performed this integration to some point, although participants wanted more.

These two factors point to the use of design guideline examples as "reusable components". To the extent that guidelines have proven to be effective and are packaged in a form that that can be directly reused, developers will readily do so, improving not only productivity but the overall quality of the application. Software reuse has proven to be a tricky issue, loaded with potential that has yet to be realized [Biggerstaff 92]. Current research in reusability has observed that the most successful efforts have focused on the reuse of domain-specific components, not on universally applicable techniques [Arango, Prieto-Diaz 91].

Tailoring Guidelines for Specific Applicability


Nielson states that "it may be one of the defining characteristics of next-generation is that they abandon the principle of conforming to a canonical interface style and instead become more radically tailored to the requirements of the individual tasks" [Nielson, 93]. To achieve this level of specialization we need to create organization specific guidelines that reflect the needs and tasks of the users and not the developer or managers. This organizational memory cannot be static, but must evolve along with the organization.

An Organizational Memory Approach to Developing Design Guidelines


The discussion of guidelines has shown that it is often difficult to understand how specific guidelines apply to a designer's current situation, or which guidelines are most appropriate. There is ample evidence that designers in many domains actively seek concrete examples of existing designs, especially in the early stages of design where many approaches to the problem are being considered [Lewis, Olson 87; Domeshek, Kolodner 92]. Thus, an effective strategy for developing a user interface is to find an existing interface that has proven effective for users with similar backgrounds and needs and copy the ideas and methods into the new interface.

The organizational memory approach to design guidelines builds on this method by collecting and disseminating the accumulated wisdom of developers in a development organization. The general notion is that an organization begins with a core set of general design guidelines, such as Smith and Mosier's 944 guidelines [Smith, Mosier 86], or others, combined with specific widget guidelines for the application platforms used in the organization. As interfaces are developed, developers record which guidelines were used and other criteria that significantly impacted interface design. These experiences are stored in cases that are attached to the general guidelines to create a repository capable of distributing the accumulated experience of the organization. The cases demonstrate how different guidelines have been applied to interface designs, and can serve as examples of interface designs (both good and bad) that can be used by subsequent developers.

Using the guidelines becomes a matter of describing user needs and matching them to existing cases. The assumption is that an organization as a whole will be fairly homogeneous in the types of users and design problems that are encountered. By tying guidelines to specific cases and providing the means to locate interface problems with similar characteristics, the method provides the means to determine when the guidelines should be applied. The concrete cases help designers interpret the guidelines, making them easier to comprehend and apply to the current design problem.

As the cases accumulate, the knowledge contained in the repository becomes increasingly tailored to the kinds of design problems that frequently occur in the organization. The repository therefore serves not only as a means to disseminate interface design knowledge, but also helps an organization learn what does and does not work for their base of customers. The method naturally incorporates an evolutionary, continuous, improvement process [Paulk et al 93] that evolves with the ever-changing development context. As the guidelines are refined through the process of adding new cases, developers can draw on and improve techniques that have been used in the past. As cases accumulate the need for new guidelines will emerge. These guidelines draw on and generalize the cases to create new and improved knowledge of interface design for specific instances that recur in the organization. As opposed to attempting to derive guidelines from abstract first principles, the organizational learning approach seeks to accumulate concrete information from which new principles can be derived. The method incorporates the means to reify the theoretical to the concrete [Bellotti 93] by providing examples that make guidelines more ecologically valid, more understandable, less open to interpretation, and more easily located [Tetzlaff, Schwartz 91].

But while some of the problems of using design guidelines are solved with an organizational learning approach, other problems are exacerbated. Guidelines are faced with an inherent conflict: to be useful, guidelines need to be as specific to the design needs as possible, but this creates a proliferation of guidelines, making it even harder to find appropriate guidelines. The complexity and size of guidelines are immense to begin with (witness Smith and Mosier's 944 general-purpose guidelines [Smith, Mosier 86]), making it difficult to find guidelines that are applicable to a given kind of problem. Providing usability examples and extending guidelines to meet specific user needs provides important information, but increases the problem of locating relevant guidelines.

An effective method for retrieving guidelines and cases is needed. Creating a classification structure involves a great deal of a priori knowledge about the domain under investigation. Our premise is that much of this knowledge does not currently exist and must be derived by the organization as relevant issues arise in the development of interactive systems. We have therefore chosen to use case-based technology. Case-based methods do not require extensive classification to find information, and are often touted as a technique that works best in the kinds of ill-defined problem solving situations we are interested in [Kolodner 93; Slade 91].

A Case-Based Approach to Organizational Learning


Case-based reasoning is an artificial intelligence method based on cognitive models postulating that much of human problem solving involves applying past experiences to analogically related situations. While early case-based systems attempted to provide autonomous problem solving by adapting existing solutions to new situations, recent systems have emphasized providing an external memory for users through an interactive process of decision support [Kolodner 91; Pearce et al. 92; Barber et al. 92]. A case-based repository for decision support can suggest how new problems can be approached, suggest the means for adapting a solution that does not quite fit, warn of possible failures, and help designers interpret and understand a situation [Kolodner 93].

Case-based decision aid technology is a perfect fit with an organizational memory approach to design guidelines because we are interested in situations in which there is no formalized or algorithmic solution available, but problem solving examples exist. Human interpretation and analysis of the situation is necessary, but people need help finding relevant cases because they either forgot or did not know about existing solutions and approaches to a given problem. Case-based methods can also support the abstraction process that is so important to domain analysis by detecting patterns, such as when several cases suggest the same solution and/or are indexed with similar terms.

Using Experience-Based Design Guidelines


Our framework for an organizational learning approach to design guidelines has two elements: Guidelines are interface principles or rules of thumb that have been adopted or otherwise "blessed" by the organization's interface experts. Usability Examples (also referred to as cases) are examples of specific interfaces that have been developed by the organization. We are in the process of validating our approach with an in-house software development organization for a major railroad corporation in the US, Union Pacific Railroad (UPRR). This organization employs about 300 people developing in-house information systems that support the business operations of UPRR. The organization is in the midst of a transition from mainframe data delivery systems to PC-based client-server decision support systems. User interface issues and working closely with customers have always been part of the corporate culture, but management is looking to streamline their development process and begin collecting project experiences [Henninger et al. 95]. Guidelines for designing interfaces is one of the areas receiving additional attention.

A second-generation experimental prototype, the Mimir Design Guidelines System, is being used to demonstrate how an organizational learning approach is utilized to develop interactive software systems. Developing user interfaces with Mimir can be accomplished with two integrated strategies. The first is to describe the current problem and user requirements, retrieving cases (usability examples) with similar characteristics. The designer can then peruse the cases and reuse any of the solutions that closely match the problem. If no directly matching cases are found, the designer can follow hypertext links to the more generally applicable guidelines, which will also point to specific cases that may prove useful. The second strategy is the more traditional approach of perusing a hierarchically arranged guideline structure, using any applicable cases or guidelines as they are found.

In this section we present two scenarios that have been derived from actual protocols we have collected at UPRR. The first scenario depicts a developer searching for a solution to a problem by reviewing similar cases, while the second one involves searching through guidelines.

Figure 1: Searching for Usability Cases.


Migrating Data to a Client/Server Database


There are many development efforts at UPRR currently focused on migrating mainframe applications to a client/server architecture that offers superior graphical interface facilities. One of these projects involves access to TSO (Time Sharing Option) data. Currently, this data is only available through a program running on the mainframe (the UPRR-built car flow system). Moving this data to Lotus Notes, which is used for widespread dissemination of information in the organization, would make the data more accessible to UPRR employees and support more applications for viewing and graphing the data. The objective is therefore to move this data, which is periodically generated, from the mainframe to Lotus Notes. A number of issues ranging from client/server infrastructure to formatting the data must be resolved to accomplish this task. To address these issues, we will show how the Mimir system can be used to determine if other projects have encountered similar problems.

Mimir is a prototype system that provides access to a structured set of design guidelines and design cases. We begin by demonstrating how the case-based system underlying Mimir can be used to query for design problems with similar characteristics. The basic process is to construct a query using a brief description or a set of keywords. Mimir responds with a list of cases in rank order of relevance to the query. From there, users can explore the cases in full detail or use a relevance feedback mechanism to refine their query.

For example, suppose a developer working on the TSO to Lotus Notes problem enters the term 'information storage' in Mimir's Search for Cases window (Figure 1). A number of cases are returned that can be displayed in detail. The developer can also choose to modify the query to find additional information based on the information retrieved. This is a form of relevance feedback [Salton, Buckley 90] that is accomplished by choosing a subset of the items retrieved and clicking on the 'relevance feedback' button (see Figure 1).


Figure 2: Displaying a Usability Case.

Only a few cases are retrieved this time, and the developer reviews the details of the case with the highest rank (Figure 2). The selected case discusses how a related project stored graphical information within a Lotus Notes database. Of particular interest to the developer in this case is the description of how Excel graphs were embedded within Lotus Notes. This is a key piece of information with broad applicability that has become buried deep within an isolated document. The developer can easily turn this information into a guideline, as described in the "Extending and Specializing Guidelines" section.

Clicking on the Related Cases/Guidelines reveals a set of design guidelines and cases that are related to the case (Figure 3). The developer can further explore cases or guidelines by clicking on these labels. Cases can provide information on related issues that may or may not be relevant to the developer's current situation. Guidelines can show how the cases relate to general issues. For example, the guideline on "Place information in an easily accessible location" can show the developer how a well-chosen location for the information increases usability. This may give the developer some insight into how users will access the Notes version of the TSO dataset.

Through this process, Mimir helps designers find specific cases that are related to the user's current design problem. By providing links to design guidelines, users can explore the larger issues involved and discover some of the rationale and implications of specific design cases. Mimir can also work in the opposite direction, helping users develop an understanding of design guidelines through concrete cases associated with them, as demonstrated in the following section.

Using Appropriate User Terminology in Interface Designs


Designers and users typically have very different models of how a system should operate [Gentner, Grudin 90]. Understanding user terminology well enough to provide proper cues and command names [Barnard, Grudin 88] is just one way in which developers must understand how users operate. Creating an effective interface using proper terminology can be a difficult task for a developer in a large corporation since there may be many different groups, each with different professional backgrounds with their own dialects and acronyms.


Figure 3: Displaying Retrieved Guidelines and Usability Cases.

Suppose a development project is working on an interface that will be used by the track engineering group. Eventually the subject of terminology used in the interface arises. The developers can use Mimir to search for guidelines on terminology usage. The designer constructs a query with the term 'terminology' in the Design Guidelines system (Figure 4a). Retrieval results include, among others, "Speak the User's Language", and "Avoid Frustrating the User." These are examples of general or abstract guidelines which can be applied to nearly any situation. Yet there are no suggestions on how these two guidelines could be implemented in this, or any, specific situation.

Mimir represents guidelines hierarchically from least to most specific. Although the search shown in Figure 4a transcends this hierarchy, the 'Use a Terminology Study' and 'Make a Terminology Study' are sub-guidelines of 'Speak the User's Language' (the guideline hierarchy can be traversed with an indented outline that is not shown here). The 'Use a Terminology Study' topic in Figure 4a points to the guideline shown in Figure 4b. This provides a brief description of the guideline along with hypertext links to related guidelines and cases. Cases point to terminology studies performed for specific user groups, for example, track engineers, which is a good match for the designers in our scenario. If none of the cases matched the target user audience, the developer would choose the Make a Terminology Study', which provides advice on how to study user vocabulary and provide pointers to specific cases containing survey and other material used to perform a user terminology study.


Figure 4: Searching For Guidelines (a), and Displaying a Guideline (b).

Cases also allow users to provide comments that explain the issues and tradeoffs involved in the case. Comments on the 'Terminology Survey for Track Engineers' case indicate that users had expressed some frustrations with the interface that was developed in that project. The exact causes of the problems are not clear, but the study is rather old, so the developers decide to re-do the study. Using some of the same materials used in the previous study, the developers perform a "confirming" study and add a new case to the system (see Figure 5). The case is added through a copy/edit process in which all the links from the previous study are copied to the new case. From there, the developers are free to add and delete hypertext links and edit text to create the new case.


Figure 5: Adding a New Usability Case.

Extending and Specializing Guidelines

A crucial element of our approach is to dynamically maintain and capture emerging guidelines to meet the changing needs of an organization. In the scenario involving the TSO data (see Figure 3), the developer realizes that their solution has lead to a specialization of the two guidelines: "Place information in an easily accessible location" and "Use visual aids (graphs) to show information." Mimir allows the developer to add the new guideline "To display statistical information, embed Excel 'graphs' in Lotus Notes," with a description and comments into the organizational memory.

For situations where a current guideline is sufficient in most areas, the developer can annotate the guideline with comments. Trends in the guidelines can act as a prompt for the creation of a more specialized guideline or the changing of organizational policies.

Capturing Guideline Cases


The process of capturing usability cases needs to strike a delicate balance between gathering enough knowledge to support usability issues, while not becoming overly disruptive to the development process. Our goal is to understand organizational practice so that new practices can fit the organizational framework in a natural manner, yet transform the activity into a knowledge collecting process that can provide the means for improving the development process.

Achieving these goals means that the system must become part of the organization's normal design process [Terveen, Selfridge, Long 93]. The system must quickly reach a critical mass of information to become an invaluable part of the design process and maintain that usefulness by evolving with the changing needs of the organization. This can be accomplished by initially "seeding" [Fischer et al. 94] the repository with standard or organization-wide usability guidelines. Providing this information on-line would prove useful to the interface design process in and of itself [Alben, Faris, Saddler 94]. But we wish to extend this valuable knowledge with case information. This must occur through the more problematic process of in-situ knowledge capture.

We have identified two places in which this process can be deployed in our development organization. The first comes from a standard practice in the organization to have a post-implementation survey for each project. This survey can provide information about the effectiveness of different resources for a given problem type as well as capturing caveats that can be stored in a "Caution" field such as the one in Figure 5. The second is to formalize the process of status reports and project tracking to monitor the issues and subproblems that arise in the course of the project. Centering the information around a single repository through an ongoing process of capturing project experiences can prevent the duplication of efforts, avoid repeating common mistakes, and help streamline the development process. Similar projects have shown that such a system will be used by development personnel, provided it contains relevant and useful information [Terveen, Selfridge, Long 93].

Assessment and Future Directions


An organizational learning approach to domain analysis and software development is best suited to development contexts in which common customer needs are being addressed in similar application domains by multiple projects. This is closest in scope to in-house development organizations, but can also be adopted in organizations that do contract or commercial off-the-shelf development [Grudin 91]. The scope of our approach naturally incorporates all of the stakeholders involved in software development. Management, developers, marketing, customers and others can share project experiences. The nature of these experiences will differ across the various disciplines, but the general infrastructure of organizational memory systems remains intact.

Our long-term goal is to use the repository as an empirical testbed to show which techniques work best for a given domain. Although data may be spotty, it will be real, and we feel confident that clear trends will emerge that can add an empirical basis to vendor, methodology, and researcher claims [Fenton, Pfleeger, Glass 94]. Our joint charter thus far has been to explore and characterize the organization to understand what kind of infrastructure is needed to more effectively develop software at this organization. Incorporating the technique into the complex fabric of any large development organization is a lengthy and tenuous process.

We are currently working with UPRR to integrate these methods into their development methodology and begin the process of setting up the case-based repository [Henninger et al. 95]. The prototype has thus far been used as a communication medium to disseminate intermediate results and conclusions. We are currently working closely with a project at UPRR that will serve as a pilot project for this approach. We are "shadowing" the development project, using videotape, project-related electronic mail, and frequent site visits to "seed" our prototype with data relevant to the project [Fischer et al . 94]. Toward the end of the year, we plan to have project personnel take over the repository we have seeded. We will then analyze the effort and lessons learned from adopting the technology to design an organization-wide set of tools supporting the development of effective design guidelines.

Related Methods


Design guidelines that attempt to distill the accumulated wisdom of the HCI field [Smith, Mosier 86; Brown 88], document specific interface design methodologies [IBM 92] or enforce compliance with the look-and-feel of products [Apple 93; Microsoft 92] have proliferated. Hypertext systems that help designers wade through the guidelines have been designed [Alben, Faris, Saddler 94; Iannella 94], some including checklists to help users apply the reference material in a step-by-step manner [Perlman 89]. While many books and manuals have been published on design guidelines, to date little work has been done on the impact of these guidelines in specific organizations. In particular, the subject of adapting guidelines to specific development organizations, classes of users (not the novice/expert distinction, but different professional fields and the like), or different kinds of tasks that users wish to accomplish has not been addressed. There are, however a number of other research efforts that have significant impact on our approach.

Design rationale is the process of capturing the decisions made during systems development such that re-hashing old decisions can be avoided, and future efforts can use the reasoning and experience of previous projects. In contrast with guidelines and standards, rationale is used to explore the pros and cons of specific issues. Issue structures such as issues, answers, and arguments [Conklin, Begeman 88; Fischer et al 91a] and questions, options, and criteria [Maclean et al. 91] and similar structures incorporating artificial intelligence techniques [Lee 93; Ramesh, Dahr 94] have been developed to help designers explore alternatives and leave traces of their reasoning process. This technique is especially pertinent to ill-defined domains such as interface design where definitive answers are not readily available and designs are open to interpretation.

In the same way that standards and guidelines suffer from abstractness, design rationale suffers from too much concreteness. Looking at the detail without any attempts to synthesize the information can confuse the designer in the myriad of detail that may or may not be relevant. Design rationale systems can also become victims of their own success. Every project has a myriad of issues needing capturing, causing an information overload problem [Hiltz, Turoff 85; Fischer, Stevens 91] that demands effective information retrieval and filtering [Belkin, Croft 92]. In addition to problems with capturing and finding relevant rationale [Conklin, Yakemovic 91], it is generally difficult to apply specific cases to new and different interfaces. There are also important questions, such as which development methods and tools should be used, that design rationale has not been concerned with. This kind of information can be a part of design rationale in principle, but none of the techniques to date have explicitly separated different kinds of concerns, choosing instead to focus on the structure of arguments on design alternatives. The focus has remained exclusively on rationale for design decisions, leaving no room to collect crucial information on the effectiveness of development methods and tools.

Perhaps the ultimate in design guideline technology is to embody the guidelines within a development environment. Fischer and colleagues have been developing design environments that provide guidelines to support designers in creating quality designs [Fischer, Lemke 88]. The guidelines are embodied in knowledge-based "critics" that identify suboptimal designs based on criteria derived from experts [Fischer et al. 91a; Fischer et al. 91b]. By contextualizing the guidelines on the artifact under design, this method solves some of the problems with finding relevant guidelines. But it also relies on computationally detecting the relationship between design elements and can only provide advice once a mis-step is taken. This means the guidelines must be formally definable, and design alternatives outside the scope of the current design cannot be supported. Our approach is to contextualize the advice given by the system by creating an evolving organizational memory of usability experiences that can support computational critics once an appropriate level of formality is achieved, but that also supplies information on less formal, but equally relevant, design guidelines.

We have also been influenced by the various process improvement efforts [Paulk et al 93]. Our approach shares the common goal of enhancing productivity and quality of software development as a continuous improvement process. These approaches, however, suffer from an over-emphasis on process aspects of development. We extend this notion to the continuous improvement of design guidelines that serve as an organizational asset that can be re-used, refined, and accumulated as an organization matures in different application domains.

Our approach is most closely related to some approaches to constructing organizational memory systems [Walsh, Rivera 91]. While the organizational learning approach outlined in this paper emphasizes the process of learning from and improving on previous efforts, these efforts focus on the first step in our domain lifecycle, collecting and disseminating design information. TeamInfo focused primarily on querying and browsing issues for an organizational memory of loosely organized e-mail messages [Berlin et al. 93]. Answer Garden was built to turn knowledge into an organizational asset in a network of multiple-choice questions and answers [Ackerman, Malone 91]. Their bottom-up process evolves the repository in response to user questions, and would be most useful for collecting experiences about development tools. Our framework goes further to support the process of analyzing domains and turning the individual cases into assets that can streamline the development process.

Our approach is closest in scope to the Designer Assistant effort at AT&T which provides access to a repository of issues such as real-time performance constraints, local programming conventions, properties of an implementation, and others [Terveen, Selfridge, Long 93]. Their repository approach uses traditional knowledge-based technology, but accomplishes many of the goals we have set out to address. We are using a similar approach to collect and disseminate design guidelines and are investigating design methodologies that incorporate the use and refinement of guidelines.

Conclusions


Creating effective interfaces is less a matter of getting graphical widgets right than understanding customer needs and knowing how these needs can be met with specific interface techniques. While many interface development methods demand user involvement in the design process, there are no means for capturing this information for future use. Our approach uses organizational memory to capture information on the effectiveness of tools and techniques at satisfying user requirements. This information can be used to match customer requirements to specific interface techniques that have successfully been employed before, thus supporting the process of designing effective interfaces.

Our approach to design guidelines changes the form and scope of traditional usability guidelines. By attaching to guidelines some of the contextual knowledge that is necessary to ensure good design, our technique allows designers to actually reuse the results of successful efforts. This allows usability issues to be incorporated into design at the very outset of application development and helps development organizations build a culture based on success, avoid duplicate efforts, and avoid repeating mistakes.
The premise of this work is that organizations are the best place to begin collecting the kind of information that will be useful for augmenting guidelines with specific cases. Development organizations accumulate a great deal of knowledge about the needs and work practices of their clients. But the deployment of such systems can have wider implications that can contribute to the systematic understanding of factors that contribute to the success or failure of an interface design [Strong 94]. Such a contribution will go far to improve methods for designing interfaces.

Acknowledgments. We gratefully acknowledge the efforts of Lynden Tennison and Jim Montequin at Union Pacific Railroad. This research was funded by Union Pacific Railroad, the Applied Management Institute in Omaha, NE, and the Center for Communication and Information Science at UNL.

References

[Ackerman, Malone 90] M.S. Ackerman, T.W. Malone, "Answer Garden: A tool for growing organizational memory", Proceedings of the Conference on Office Information Systems, ACM, New York, 1990, pp. 31-39.

[Alben, Faris, Saddler 94] L. Alben, J. Faris, H. Saddler, "Making it Macintosh: Designing the Message When the Message is Design," interactions, 1(1), pp. 10-20, January 1994.

[Apple 93] Apple Computer, Inc., Macintosh Human Interface Guidelines, Addison-Wesley, Reading, MA, 1992.

[Arango, Prieto-Diaz 91] G. Arango, R Prieto-Diaz, Domain Analysis and software system modeling, IEEE Computer Society Press, Los Alamos, CA, 1991.

[Barber et al. 92] J. Barber et al., "AskJef: Integration of Case-Based and Multimedia Technologies for Interface Design Support," Artificial Intelligence in Design '92, J.S. Gero (Ed.), Kluwer Academic, pp. 457-474, 1992.

[Barnard, Grudin 88] P. Barnard, J. Grudin, "Command Names," in Handbook of Human-Computer Interaction, M. Helander (Ed.), Elsevier Science Publishers, Amsterdam, 1988.

[Belkin, Croft 92] N.J. Belkin, W.B. Croft, "Information Filtering and Information Retrieval: Two Sides of the Same Coin?", Communications of the ACM, 35(12), pp. 29-38, Dec., 1992.

[Bellotti 93] V. Bellotti, "Integrating Theoreticians' and Practitioners' Perspectives with Design Rationale", Proceeding of the Conference on Computer-Human Interaction (InterCHI '93), ACM, pp. 101-114, 1993.

[Berlin et al. 93] L.M. Berlin, R. Jeffries, V.L. O'day, A. Paepcke, C. Wharton, "Where Did You Put It? Issues in the Design And Use of a Group Memory," Proceeding of the Conference on Computer-Human Interaction (InterCHI '93), ACM, 1993, pp. 23-30.

[Biggerstaff 92] T.J. Biggerstaff, "An Assessment and Analysis of Software Reuse", Advances in Computers, 34, pp. 1-57, 1992.

[Brown 88] C.M. Brown, Human-Computer Interface Design Guidelines, Ablex, New Jersey, 1988.

[Conklin, Begeman 88] J. Conklin, M. Begeman, "gIBIS: A Hypertext Tool for Exploratory Policy Discussion", Transactions on Office Information Systems, 6(4), 1988, pp. 303-331.

[Conklin, Yakemovic 91] E.J. Conklin, KC Yakemovic, "A Process-Oriented Approach to Design Rationale," Human-Computer Interaction, 6, pp. 357-391, 1991.

[Curtis, Hefley 94] B. Curtis, B. Hefley "A WIMP No More: The Maturing of User Interface Engineering," interactions, 1(1), pp. 23-34, January 1994.

[de Souza, Bevan 90] F. de Souza, N. Bevan, "The Use of Guidelines in Menu Interface Design: Evaluation of a Draft Standard," Human-Computer Interaction - INTERACT '90, Elsevier, North-Holland, pp. 435-440, 1990.

[Domeshek, Kolodner 92] E.A. Domeshek, J.L. Kolodner, "A Case-Based Design Aid for Architecture," Artificial Intelligence in Design '92, J.S. Gero (Ed.), Kluwer Academic, pp. 497-516, 1992.

[Fenton, Pfleeger, Glass 94] N. Fenton, S.L. Pfleeger, R.L. Glass, "Science and Substance: A Challenge to Software Engineers," IEEE Software, 11(4), July 1994, pp. 86-95.

[Fischer et al. 94] G. Fischer, R. McCall, J. Ostwald, B. Reeves, F. Shipman, "Seeding, Evolutionary Growth and Reseeding: Supporting the Incremental Development of Design Environments", Proceeding of the Conference on Computer-Human Interaction (CHI '94), ACM, pp. 292-298, 1994.

[Fischer et al. 91a] G. Fischer, A.C. Lemke, T. Mastaglio, A.I. Morch, "Critics: An Emerging Approach to Knowledge-Based Human Computer Interaction," International Journal of Man-Machine Studies, 35(5), pp. 695-721, 1991.

[Fischer et al. 91b] G. Fischer, A.C. Lemke, R. McCall, A. Morch, "Making Argumentation Serve Design", Human-Computer Interaction, 6(3-4), 1991, pp. 393-419.

[Fischer, Lemke 88] G. Fischer, A.C. Lemke, "Construction Kits and Design Environments: Steps Toward Human Problem-Domain Communication", Human-Computer Interaction, 3(3), 1988, pp. 179-222.

[Fischer, Stevens 91] G. Fischer, C. Stevens, "Information Access in Complex, Poorly Structured Information Spaces", Human Factors in Computing Systems, CHI'91 Conference Proceedings (New Orleans, LA), ACM, pp. 63-70, 1991.

[Gentner, Grudin 90] D.R. Gentner, J. Grudin, "Why Good Engineers (Sometimes) Create Bad Interfaces," CHI'90 Conference Proceedings (Seattle, WA), ACM, 1990, pp. 277-282.

[Gould, Boies, Lewis 91] J.D. Gould, S.J. Boies, C.H. Lewis, Making Usable, Useful, Productivity-Enhancing Computer Applications, Communications of the ACM, 34(1), pp. 75-85, Jan. 1991.

[Gould, Lewis 85] J.D. Gould, C.H. Lewis, "Designing for Usability - Key Principles and What Designers Think," Communications of the ACM, 28, pp. 300-311, 1985.

[Grady 93] R.B. Grady, "Practical Results From Measuring Software Quality," Communications of the ACM, 36(11), pp. 62-68, November 1993.

[Grudin 91] J. Grudin, "Interactive Systems: Bridging the Gaps Between Developers and Users," Computer, 24(4), April 1991, pp. 59-69.

[Grudin 90] J. Grudin, "The Computer Reaches Out: The Historical Continuity of Interface Design," Human Factors in Computing Systems, CHI'90 Conference Proceedings (Seattle, WA), ACM, New York, pp. 261-268, 1990.

[Heckel 91] P. Heckel, The Elements of Friendly Software Design, Sybex, San Francisco, 1991.

[Henninger et al. 95] S. Henninger, K. Lappala, A. Raghavendran, "An Organizational Learning Approach to Domain Analysis," Seventeenth International Conference on Software Engineering (Seattle, WA), ACM, IEEE, Los Alamitos, CA, pp. 95-104.

[Hiltz, Turoff 85] S.R. Hiltz, M. Turoff, Structuring Computer-Mediated Communication Systems to Avoid Information Overload, Communications of the ACM, 28(7), pp. 680-689, July, 1985.

[IBM 92] IBM Corporation, Object-Oriented Interface Design: IBM Common User Access Guidelines, Que, Carmel, Indiana, 1992.

[Iannella 94] R. Iannella, "HyperSAM: A Practical User Interface Guidelines Management System," Proceedings of the Second Annual SIGCHI (Queensland) Symposium - QCHI '94, Bond Univ., Australia, 1994.

[Kolodner 91] J.L. Kolodner, "Improving Human Decision Making through Case-Based Decision Aiding," AI Magazine, 12(2), Summer, pp. 52-68, 1991.

[Kolodner 93] J.L. Kolodner, Case-Based Reasoning, Morgan-Kaufman, San Mateo, CA, 1993.
[Laurel 93] B. Laurel, Computers as Theatre, Addison-Wesley, Reading, MA, 1993.

[Lee 93] J. Lee, "Design Rationale Capture and Use," AI Magazine, 14(2), Summer 1993, pp. 24-26.

[Lewis, Olson 87] C.H. Lewis, G.M. Olson, "Can the Principles of Cognition Lower the Barriers of Programming?," Empirical Studies of Programmers (Vol. 2), G.M. Olson, E. Soloway, S. Sheppard, Ablex, Lawrence Erlbaum Associates, Norwood, NJ, pp. 248-263, 1987.

[Maclean et al. 91] Maclean et al., "Questions, Options, and Criteria: Elements of Design Space Analysis", Human-Computer Interaction, 6(3-4), 1991, pp. 201-251.

[Microsoft 92] Microsoft Corporation, The Windows Interface: An Application Design Guide, Microsoft Press, Redmond, WA, 1992.

[Myers 94] B.A. Myers, "Challenges of HCI Design and Implementation," interactions, 1(1), pp. 73-83, January 1994.

[Nielson 92] J. Nielson, "The Usability Engineering Life Cycle," Computer, 25(3), pp. 12-22, March, 1992.

[Paulk et al 93] M.C. Paulk, B. Curtis, M. Chrissis, C.V. Weber, "Capability Maturity Model, Version 1.1," IEEE Software, 10(4), pp. 18-27, July 1993.

[Pearce et al. 92] M. Pearce, A.K. Goel, J.L. Kolodner, C. Zimring, L. Sentosa, R. Billington, "Case-Based Design Support: A Case Study in Architectural Design," IEEE Expert, 7(5), October 1992, pp. 14-20.

[Perlman 89] G. Perlman, "Asynchronous Design/Evaluation Methods for Hypertext Development," Hypertext '89 Proceedings, pp. 61-68, 1989.

[Poltrock, Grudin 94] S.E. Poltrock, J. Grudin, "Organizational Obstacles to Interface Design and Development: Two Participant Observer Studies", ACM Transactions on Computer-Human Interaction, 1(1), March 1994, pp. 52-80.

[Ramesh, Dahr 94] B. Ramesh, V. Dahr, "Representing and Maintaining Process Knowledge for Large-Scale Systems Development," IEEE Expert, 9(2), April 1994, pp. 54-59.

[Rosson, Maass, Kellogg 88] M.B. Rosson, S. Maass, W.A. Kellogg, "The Designer as User: Building Requirements for Design Tools from Design Practice," Communications of the ACM, 31(11), November 1988, pp. 1288-1298.

[Salton, Buckley 90] G. Salton, C. Buckley, "Improving Retrieval Performance by Relevance Feedback, Journal of the American Society for Information Science, 41(4), pp. 288-297, 1990.

[Slade 91] S. Slade, "Case-Based Reasoning: A Research Paradigm," AI Magazine, 12(1), Spring, pp. 42-55 1991.

[Smith 86] S.L. Smith, "Standards versus Guidelines for Designing User Interface Software", Behaviour and Information Technology, 5(1), 1986, pp. 47-61.

[Smith, Mosier 86] S.L. Smith, J.N. Mosier, Guidelines for Designing User Interface Software, Technical Report, The MITRE Corporation, ESD-TR-86-278, 1986.

[Strong et al. 94] G.W. Strong et al., New Directions in Human-Computer Interaction Education, Research, and Practice, report sponsored by NSF and ARPA, November 3, 1994, available on WWW at http://129.25.40.211.

[Terveen, Selfridge, Long 93] L.G. Terveen, P.G. Selfridge, M.D. Long, "From 'Folklore' To 'Living Design Memory'," Proceedings InterCHI '93, ACM, 1993, pp. 15-22.

[Tetzlaff, Schwartz 91] L. Tetzlaff, D.R. Schwartz, "The Use of Guidelines in Interface Design", Proceeding of ACM CHI '91 Conference on Human factors in Computing Systems, 1991, pp. 329-333.

[Thimbleby 90] H. Thimbleby, User Interface Design, Addison-Wesley, New York, 1990.

[Walsh, Rivera 91] J.P. Walsh, G. Rivera, "Organizational Memory", Academy of Management Review, 16(1), 1991, pp. 57-91