Dept. of Electrical Engineering and Computer Science
Tufts University
Our goal is to develop a model and abstraction that captures the formal structure of next-generation dialogues in the way that existing techniques have captured command-based, textual, and event-based dialogues. Most current user interface description languages and software systems are based on serial, discrete, event-based interaction. Most of today's examples of non-WIMP interfaces have, of necessity, been designed and implemented with event-based models more suited to previous interface styles. Because those models fail to capture continuous, parallel interaction explicitly, the interfaces have required considerable ad-hoc, low-level programming approaches. While some of these are very inventive, they have made such systems difficult to develop, reuse, and maintain.
R.J.K. Jacob, "A Visual Language for Non-WIMP User Interfaces," Proc. IEEE Symposium on Visual Languages pp. 231-238, IEEE Computer Society Press (1996). [ASCII]; [Postscript].
R.J.K. Jacob, "Input Devices and Techniques," pp. 1494-1511 in The Computer Science and Engineering Handbook, ed. Allen B. Tucker, CRC Press (1996). [HTML]; [ASCII]; [Postscript].
R.J.K. Jacob, "Human-Computer Interaction: Input Devices," ACM Computing Surveys 28(1) pp. 177-179 (March 1996). [HTML]; [Postscript].
A user interface management system (UIMS) is a separate software component that conducts all interactions with the user; it is separate from the application program that performs the underlying task. It removes the programming of the user interface from each individual application and permits some of the effort of designing tools for human-computer interaction to be amortized over many applications and shared by them. A UIMS partitions interactive computer applications into a user interface portion and an application portion.
To be useful, a UIMS needs a convenient and understandable way for the user interface designer to describe the desired interface. The choice of specification language or model is thus the key to UIMS design. Whether such methods are literally linguistic or interactive, they generally fall under the rubric of user interface description languages (UIDL). A UIDL describes the user-visible behavior of the interface but not its implementation. It can allow a designer to explore alternative user interface designs without having to code them in the conventional way. A variety of specification languages for describing WIMP and other previous generations of user interfaces has been developed, and user interface management systems have been built based up on them, using approaches such as BNF, state transition diagrams, event handlers, declarative specifications, and frames.
However, few UIDL's to date address the problems introduced by non-WIMP interaction styles. The emerging generation of "non-WIMP" user interfaces provides parallel, continuous, multi-mode, "non-command", interaction--in contrast to current GUI or WIMP style interfaces. The non-WIMP interaction style can be seen most clearly in virtual reality (VR) interfaces, but its fundamental characteristics are common to a more general class of emerging user-computer environments, including multimedia, new types of games, musical accompaniment systems, intelligent agent interfaces, interactive entertainment media, pen-based interfaces, eye movement-based interfaces, and ubiquitous computing. They share a higher degree of interactivity than previous interfaces: continuous input/output exchanges occurring in parallel, rather than one single-thread, discrete event dialogue. Non-WIMP interfaces are characterized by continuous interaction between user and computer via several parallel, asynchronous channels or devices.
M. Green and R.J.K. Jacob, "Software Architectures and Metaphors for Non-WIMP User Interfaces," Computer Graphics 25(3) pp. 229-235 (July 1991).
J.B. Lewis, L. Koved, and D.T. Ling, "Dialogue Structures for Virtual Worlds," Proc. ACM CHI'91 Human Factors in Computing Systems Conference pp. 131-136, Addison-Wesley/ACM Press (1991).
B.A. Myers, "User Interface Software Tools," ACM Transactions on Computer-Human Interaction 2(1) pp. 64-103 (March 1995).
D.R. Olsen, User Interface Management Systems: Models and Algorithms, Morgan Kaufmann, San Mateo, Calif. (1992).
C. Shaw, M. Green, J. Liang, and Y. Sun, "Decoupled Simulation in Virtual Reality with the MR Toolkit," ACM Transactions on Information Systems 11(3) pp. 287-317 (1993).
B. Shneiderman, Designing the User Interface: Strategies for Effective Human-Computer Interaction, Second Edition, Addison-Wesley, Reading, Mass. (1992).
5. Usability and User-Centered Design.
6. Intelligent Interactive Systems for Persons with Disabilities.
3. Other Communication Modalities. We are particularly interested in unconventional interaction techniques and devices, because they help stretch our model and make sure it is general enough to include all cases, not just the obvious ones. We claim our model supports a wide range of unconventional interaction techniques and devices quite nicely. For example, we are currently integrating an eye tracker and are interested in including other modalities.
5. Usability and User-Centered Design. Our work will lead to languages and design tools for developing non-WIMP user interfaces. We are interested in how such tools might fit into and be used in methods for user-centered design and might support the building of more usable systems.
6. Intelligent Interactive Systems for Persons with Disabilities. This area often requires non-standard interaction techniques and devices. As in (3), we are particularly interested in including these in our model and software, and have begun with the eye tracker.
Computer-supported Cooperative Work. Our basic model and language should support multi-person interfaces with little modification. We are interested in testing this assertion and then extending the model as needed for collaborative multi-user virtual environments.