BACK OF AN ENVELOPE: AN ARCHITECTURE FOR KNOWLEDGE BASED DESIGN ENVIRONMENTS

Mark D. Gross

College of Architecture and Planning
University of Colorado

CONTACT INFORMATION

Sundance Laboratory for Computing in Design and Planning
College of Architecture and Planning University of Colorado
Boulder, CO 80309-0314
Phone: (303) 492-2807
Fax : (303) 492-6163
Email: mdg@cs.colorado.edu

WWW PAGE

http://wallstreet.colorado.edu/

PROGRAM AREA

3. Other Communication Modalities.

KEYWORDS

freehand drawing; pen based interfaces; design

PROJECT SUMMARY

People draw diagrams and sketch to explore ideas, understand problems, generate alternatives, find solutions, and to communicate with each other. In many domains hand drawn diagrams play a key role in supporting these tasks. Nowhere is this more evident than in design, where pencil and paper (and whiteboard) are still the tools of the trade, especially in early stage and in conceptual work. Today's computer based drawing programs fail to support this activity. Structured draw programs fail because sketching directly with a pencil is easier than operating a widget driven interface, and because their discrete palettes of shapes, pen sizes, and other drawing options force designers into false commitments. On the other hand, freehand sketching programs that emulate physical media overcome this limit, but they do not produce a useful internal representation for further processing by knowledge based back end tools. The proposed 'Back of an Envelope' program will address these two requirements: unstructured freehand input, and a representation suitable for processing by knowledge based tools. In short, the project aims to develop an architecture for intelligent paper.

The overall goal of the Back of an Envelope project is to support freehand drawing, diagramming, and sketching as an interface to knowledge based design assistants. Specifically, the project has three objectives. (1) It will develop an architecture for freehand interfaces to communicate with back end design tools, including case libraries and visual databases, interactive simulation programs, and expert advisors. The architecture will support both input from designers to the knowledge based assistants, and display results from the assistants back into the drawing environment. (2) The project will construct a working version of the Back of an Envelope, linked to at least three design assistants (in the domain of architectural design). (3) The project will assess the performance of the Back of an Envelope in controlled use evaluations and by placing it in design situations in architectural design studios and classes in our College.

The project will contribute to theory, in particular to an understanding of how (and under what circumstances) diagrams help people think, especially the function of drawing in design. The construction of freehand interfaces will provide a computational platform for exploring theoretical issues (for example, how visual analogies function in creative design). The project will also apply techniques developed in the visual language community for parsing visual programming languages, to the more open-ended task of supporting diagrams and sketches for problem solving. For design, the project will provide a kind of interface that can support conceptual thinking, linked with knowledge based tools that provide strategic advice and support when it is most needed-in the early stages of work. The freehand drawing interface architecture can be applied to other domains as well, for example, to commercial applications like word processors for text mark up, or for constructing new kinds of interactive learning environments. In short, the time has come for human-computer interfaces that can recognize and interpret drawings, diagrams, and sketches.

PROJECT REFERENCES

1997 Kuczun, K. and M. Gross. "Network Design Tools and Tasks", Proc. ACM Designing Interactive Systems (DIS '97). Amsterdam, the Netherlands.

1997 Gross, M. , E. Do, R. McCall, W. Citrin, P. Hamill, A. Warmack, and K. Kuczun. "Collaboration and Coordination in Architectural Design: approaches to computer mediated work", TeamCAD symposium on collaborative CAD, Graphics, Visualization, and Usability Center, Georgia Tech, May 12-13, 1997. pp 17-24.

1997 Do, E. and M.D.Gross, "Inferring Design Intentions from Sketches", Proceedings of Computer Aided Architectural Design Research in Asia '97.

1996 Gross, M.D. and E. Do. "Ambiguous Intentions: A paper-like interface for creative design", Proceedings ACM Conference on User Interface Software Technology (UIST) '96 Seattle, WA. 183-192

1996 Gross, M. D. "The Electronic Cocktail Napkin - computer support for working with diagrams," Design Studies. 17(1), 53-70.

1996 Gross, M. D. and E. Do, "Demonstrating the Electronic Cocktail Napkin," Conference Companion, ACM Conference on Human Factors in Computing (CHI '96) , Vancouver, 5-6.

1995 Gross, M. and E. Do, "Drawing Analogies - Supporting Creative Architectural Design with Visual References," in 3d International Conference on Computational Models of Creative Design, M-L Maher and J. Gero (eds), Sydney: University of Sydney, 37-58.

1994 "Stretch-A-Sketch, a dynamic diagrammer," In A. Ambler ed., IEEE Symposium on Visual Languages 1994, 232-238.

1994 Gross, M. "Recognizing and Interpreting Diagrams in Design," in T. Catarci. M. Costabile, S. Levialdi, G. Santucci eds., Advanced Visual Interfaces '94, ACM Press, 89-94.

AREA BACKGROUND

Although freehand drawing interfaces were being explored in the 'sixties with light pens on CRTs, the widespread adoption of the mouse as a primary input modality influenced the kinds of interfaces used in commercial and scientific software. With the development of inexpensive digitizing tablets and LCD display hardware, we have seen a return to pen-based systems and renewed interest in developing interfaces that exploit this modality. People often draw when they think; thus freehand drawing interfaces offer an opportunity to capture some of this thinking. Recognition of drawing marks is an important component technology; but beyond recognition of individual marks, a visual grammar can be used to parse the drawing structure and interpret it in the context of a specific domain. As freehand drawing is typically done in early and conceptual stages of thinking, precision is not important, nor is strict adherence to a formal graphical system of rules.

AREA REFERENCES

Greenberg, S., Hayne, S., & Rada, R. (1995). Groupware for Real-Time Drawing. London: McGraw Hill.

Futrelle, R., Kakadiaris, I. A., Alexander, J., Carriero, C. M., Nikolakis, N., & Futrelle, J. M. (1992). "Understanding Diagrams in Technical Documents." IEEE Computer, July, 75-78.

Lakin, F., Wambaugh, J., Leifer, L., Cannon, D., & Steward, C. (1989). "The electronic notebook: performing medium and processing medium." Visual Computer, 5, 214-226.

Landay, J. A., & Myers, B. A. (1995). "Interactive Sketching for the Early Stages of Interface Design, CHI '95 - Human Factors in Computing Systems", (pp. 43-50). Denver, Colorado: ACM Press.

MacKay, W., Verlay, G., Carter, K., Ma, C., & Pagani, D. (1993). "Augmenting Reality: Adding Computational Dimensions to Paper. ", Communications of the ACM, 36(7), 96-97.

Moran, T. P., Chiu, P., Melle, W. v., & Kurtenbach, G. (1995). "Implicit Structures for Pen-Based Systems within a Freeform Interaction Paradigm", CHI '95 - Human Factors in Computing Systems, (pp. 487-494). Denver, Colorado: ACM Press.

Negroponte, N. (1975). Soft Architecture Machines. Cambridge, Mass.: MIT Press.

Robbins, E. (1994). Why Architects Draw. Cambridge, MA: MIT Press.

RELATED PROGRAM AREAS

Speech and Natural Language Understanding.
Usability and User-Centered Design.

POTENTIAL RELATED PROJECTS

Multi-modal (speech + drawing) interfaces for collaborative design

This project would combine freehand drawing interfaces with speech recognition and natural language. Most design work is collaborative. A synchronous collaborative design session often involves a conversation, argument, or discussion about a drawing beingmade together by two or more participants. Whether collaborators are co-located or whether the conversation happens in a distributed fashion, capturing, processing the verbal spoken argument and associating it with the freehand drawing as the designers develop and debug designs could provide highly effective multi-modal interface for collaborative design.