Engineering models of human performance for human-computer interface evaluation

Bonnie E. John

Human-Computer Interaction Institute
Carnegie Mellon University

CONTACT INFORMATION

5000 Forbes Ave
Pittsburgh, PA 15213
Office: 412-268-7182
Fax: 412-268-1266
Email: bej@cs.cmu.edu

WWW PAGE

http://www.cs.cmu.edu/~bej/

PROGRAM AREA

Usability and User-Centered Design

KEYWORDS

usability evaluation methods, cognitive modeling, GOMS.

PROJECT SUMMARY

Current "engineering models of human performance" (i.e., GOMS models) can predict the processes experts use to accomplish tasks with a computer system and the time it takes them to complete these tasks. In the course of this contract, I plan to extend such models to predict the problem-solving behavior exhibited by novice computer users, learning in realistic situations, and/or the commission and detection of errors in real world use. As with their more limited predecessors, the predictive power of these improved engineering models of behavior can then be used by computer developers to evaluate interface ideas early in the specification stage of system design, even before running systems or prototypes are available.

PROJECT REFERENCES

Bhavnani, S. K, & John, B. E. (1996) Exploring the unrealized potential of computer-aided drafting. Proceedings of CHI, 1996 (Vancouver, BC, April 14-18, 1996) ACM, New York.

Bhavnani, S. K, & John, B. E. (1997) From sufficient to efficient usage: An analysis of strategic knowledge. Proceedings of CHI, 1997(Atlanta Georgia, March 22-27, 1997) ACM, New York.

Flemming, U., Bhavnani, S. K., & John, B. E. (in press) Mismatched metaphor: User vs. System Model in Computer Aided Drafting. To appear in Design Studies Journal.

John, B. E., & Marks, S. J. (1997). Tracking the effectiveness of usability evaluation methods. Behaviour and Information Technology, 16.

AREA BACKGROUND

Since the seminal Card, Moran, & Newell (1983) book, The Psychology of Human-Computer Interaction, the GOMS model has been one of the few widely known theoretical concepts in human-computer interaction. The overall motivation for GOMS and other HCI cognitive modeling efforts is to provide engineering models of human performance. In the ideal, such models produce a priori quantitative predictions of performance at an earlier stage in the development process than prototyping and user testing. That is, they predict execution time, learning time, errors, and identify those parts of an interface that lead to these predictions, thereby focusing the designer on what to fix. They allow analysis at different levels of approximation so predictions appropriate to the design situation can be obtained with minimum effort. They are straight-forward enough for computer designers to use without extensive training in psychology, and these models are integrated enough to cover total tasks. Although HCI research has not yet reached this ideal, GOMS is currently the most mature of engineering models and can be truly useful in real-world system development.

The evaluation results produced by GOMS engineering models are both quite useful and quite limited. Thus, rather than replacing user testing, the current engineering models are best viewed as reducing the amount of user testing required to develop a highly usable system. The iterative design process should then use engineering models, and other non-user testing techniques (e.g., see Nielsen & Mack, 1994), where applicable early in the design process, to evaluate candidate designs and resolve design issues as much as possible before investing in actual user testing. Such a multiple-technique approach will make the best use of available scientific and practical knowledge about human-computer interaction (see Olson & Moran, 1996, for a discussion of coordinated use of methods).

AREA REFERENCES

Cognitive Modeling

For information about the GOMS formalism for cognitive modeling, see the original work by Card, Moran and Newell, recent review articles by John & Kieras, and an extensive evaluation of GOMS predictions against real-world performance data by Gray, John & Atwood.

Card, S. K., Moran, T. P., & Newell, A. (1983).The psychology of human-computer interaction. Lawrence Erlbaum, Associates, Hillsdale, NJ.

Gray, W. D., John, B. E., & Atwood, M. E. (1993) Project Ernestine: Validating a GOMS analysis for predicting and explaining real-world task performance. Human-Computer Interaction, 8, pp. 237-309.

John, B. E. & Kieras, D. E. (1996) The GOMS family of user interface analysis techniques: Comparison and Contrast. ACM Transactions on Computer-Human Interaction., 3 (4), pp. 320-351.

John, B. E. & Kieras, D. E. (1996) Using GOMS for user interface design and evaluation: Which technique? ACM Transactions on Computer-Human Interaction, 3 (4), pp. 287-319.

Usability Evaluation Methods

For information about other UEMs see Nielsen and Mack (1994) and Olson & Moran (1996). For some approaches to empirically understanding their effectiveness, see these forthcoming papers from our laboratory (John & Marks, 1997; John & Mashyna, 1997).

Nielsen, J. & Mack, R. L. (Eds.) (1994). Usability Inspection Methods. New York: John Wiley & Sons, Inc.

Olson, J. S. & Moran, T. P. (1996) Mapping the method muddle: Guidance in using methods for user interface design. In M. Rudisill, C. Lewis, P. B., Polson, & T. D. McKay (eds.) Human-Computer Interface Design: Success Stories, Emerging Methods and Real-World Context. San Francisco: Morgan Kaufmann Publishers.

John, B. E., & Marks, S. J. (1997). Tracking the effectiveness of usability evaluation methods. Behaviour and Information Technology, 16.

John, B. E., & Mashyna, M. M. (1997) Evaluating a Multimedia Authoring Tool with Cognitive Walkthrough and Think-Aloud User Studies. Journal of the American Society of Information Science, 48 (9).

RELATED PROGRAM AREAS

Since UEMs, including cognitive modeling, are tools to evaluate user interfaces in general, our work could, in principle, relate to almost any of the other Interactive Systems project areas. However, UEMs are currently much more suited to WIMP interfaces, as opposed to speech or immersive virtual reality interfaces. Therefore, this work might best relate to Adaptive Human Interfaces, and possibly Intelligent Interactive Systems for Persons with Disabilities or Other Communication Modalities (depending on the modality).

POTENTIAL RELATED PROJECTS

We have successfully collaborated with another NSF-supported work in the past: the ACSE project (see John & Marks, 1997 and John & Mashyna, 1997, referenced above, and for a description of ACSE, see, Pane & Miller, 1993, "The ACSE multimedia science learning environment". Proceedings of the 1993 International Conference on Computers in Education, Taipei, Taiwan.).

In that collaboration, the ASCE project had a prose specification from which they built their educational system. They gave us their spec and we used different UEMs, including cognitive modeling, to predict usability problems with their system. We then ran usability tests on the system they built from the spec to assess the predictive accuracy of the UEMs. In addition, we built different versions of the system and ran more usability tests to see if the solutions suggested by the UEMs actually improved the usability of the system. The information we discovered about usability was fed back to the ACSE project.

If other projects in the Interactive Systems Program have written specifications, or story-boards, or mock-ups, from which they are building their systems, we could do a similar collaboration.