Language Learning and its Applications

Jerome Feldman
George Lakoff
International Computer Science Institute

CONTACT INFORMATION

1947 Center St.
Berkeley, CA 94704
Phone: (510) 643-9153
Fax : (510) 643-7684
Email: jfeldman@icsi.berkeley.edu

WWW PAGE

NTL Web Page - http://www.icsi.berkeley.edu/LZERO

PROGRAM AREA

Speech and Natural Language Understanding.

KEYWORDS

Learning, semantics, structured, connectionist, neural, metaphor, language.

PROJECT SUMMARY

The question of how human beings learn natural languages is one of abiding scientific interest, and computational models have begun to play a fruitful role in understanding how language might be acquired. Computational studies of grammar and language learning have also resulted in a wide range of applications. Both of the PIs (JF and GL) have worked on the problem for some time. The collaboration arises from two articulating insights. GL has been involved for 20 years in the study of how human conceptual systems are structured in terms of the details of the functioning of our bodies and brains and how language reflects that bodily structure (Feldman 89, Lakoff 87). Over roughly the same period, JF came to believe that structured connectionism provides the only known computational formalism adequate for the fine-grained modeling of human intelligence (Shastri 93). The joint effort began in 1988 when JF came to UC Berkeley and ICSI and with GL founded the L-zero project. The group seeks to develop structured connectionist models that can learn and use both natural conceptual systems and the languages that express them. A current overview of our efforts found on the NTL web page.

After some preliminary explorations, we were able to formulate a version of the language acquisition problem that was small enough to be tractable, but seemed to address most of the important issues. We repeat the initial manifesto below. Pursuing this path, our early efforts were quite productive and well received, but there are a number of limitations that we found no way to surmount. Over the last year, we have developed an extended version of the task specification and made significant changes in our representation and learning methods. For comparison, the 1990 challenge (Feldman et al 1990) was:

The system is given examples of pictures paired with true statements about those pictures in an arbitrary natural language.

The system is to learn the relevant portion of the language well enough so that given a novel sentence of that language, it can determine whether or not the sentence is true of the accompanying picture.

The task was extraordinarily difficult, since the conceptual systems for spatial relations concepts differ markedly from language to language. The problem was solved by making use of results from cognitive linguistics, neuroscience, and psychophysics in the design of a structured connectionist acquisition model that worked for a wide range of languages. As cognitive science, the result is a theory of how spatial relations concepts arise from the structure of the visual system. The result from the perspective of connectionist computation is that structured connectionism can be employed in learning a very important segment of human conceptual systems and the language that expresses them. A detailed description of the system can be found in Terry Regier's recently published book entitled "The Human Semantic Potential" from MIT Press, 1996.

Although both the grammar learning and concept learning projects were quite successful in their own terms, each exhibits limitations and this has led us to reformulate the original problem (Feldman et al 1996). In requiring only recognition, the original task fell far short of the human capabilities, particularly in concept acquisition and use. Much of our proposed work involves reformulating the language learning task and developing representations adequate for the extended problem.

Of the many shortcomings of our concept learning system we believe that the most important are three: invertibility, inference, and abstract concepts. The connectionst networks used in Regier's work were feedforward networks trained by back-propagation. This is fine for most applied tasks, but is inadequate as a model of human concept or word learning or inference. For example, feedforward networks have no structure that could produce an example of a concept that it recognizes perfectly. Its inferential inadequacies arise in similar ways. The network has no way to infer that A above B and B above C suggest that A is above C. After exploring a wide range of modifications to the back-prop structure, we conclude that a radically different representation was called for.

The problem of learning words for higher level and abstract concepts is not, like invertibility and inference, primarily computational. The very idea of learning labels for direct bodily experience breaks down for concepts like ``sell'', and even more so for ``inflation'', etc. GL (along with many others) has been working for decades on how such abstract target concepts map to more concrete source domains (Lakoff 1987). Such ``metaphoric'' mappings allow the inference structures of concrete source domains to be used to reason about abstract target domains. In fact, one of the main attractions of spatial relations as the subject of our first effort was that space is known to be the source domain for a wide range of metaphorical mappings. A critical part of our next phase of research is to explicitly model how the mapping of abstract to more concrete source domains plays a key role in language understanding.

The three requirements of invertibility, inference, and abstract concepts have dominated our investigations of task domains and problem formulation for the next round of research. In the rest of this section we describe our plans and their current state of development. As in the first phase, the overall task has been restricted and then divided into projects of approximately doctoral thesis size.

One new project extends our results to the motor system by modeling the acquistion of verbs of hand action in a wide variety of languages. This requires the introduction of what we call ``executing schemas'' - or x-schemas - that can not only represent the structure of the action in terms of the body-model, but can actually be used to control the performance of such an action. The executing schemas that control actions are interfaced with linking feature-structures that correlate distinctive primitives features of hand-action systems with (1) the schemas that execute those features and (2) the linguistic system that encodes those features in natural languages.

A further project extends the paradigm to abstract concepts. It will study short discourses from the domain of international economics, where metaphors based on bodily actions are commonly used in discussing economics. An example would be ``France fell into a recession. Germany pulled it out.'' In the domain of bodily actions, executable schemas for body action are used to conceptualize, reason about, and understand sentences about bodies. The theory of conceptual metaphor, studied in great detail by GL and others (Lakoff and Johnson 1980), shows how abstract domains such as economics are both thought about and talked about in terms of concrete domains such as bodily action. By employing the same computational structures in both the concrete and the abstract task we hope to gain further insights into the complex relationship among the body, concrete and abstract concepts, and the learning of natural languages.

Research Plan

The focus of our research over the next two years will be on refining and testing the x-schema/f-struct representation and its use in modeling lexical acquisition and metaphor. As always, this will combine the construction of performance systems with theoretical work in connectionist modeling and cognitive linguistics. For the initial period, we will concentrate on two ambitious demonstrations, each of which requires the completion of a number of subtasks.

Acquisition and use of motion terms

The general outline of this project and its goals were presented above, we now present a more detailed plan of how we are proceeding.

Problem Formulation: We have produced a classification of English verbs of hand action into ten categories of increasing difficulty. Our initial effort will focus on the first three of these: verbs of possession (get, place, ...), verbs of translation(slide, throw, ...) and rigid-object manipulation (tap, rotate, ...). The next two categories involve deformable objects and simple tools and we hope to tackle these as well.
Cross Linguistic Sampling: As in other semantic domains, there is considerable variation among the world's languages in the representation of hand actions. This is a continuing effort used both in initial design and in later testing of the system. Besides the European languages, we have gathered data on Arabic, Farsi, Japanese, Korean, Mandarin, and Tamil.
Representation and Model Design: This is also an ongoing effort and the current state has been described in detail above. We already know of problems that will require extensions to our current formalism and will work these out in parallel with the continued testing of the current design on the current tasks. As mentioned, it is important that this subtask is common to the various applications.
Environment: There is a significant technical challenge in constructing an environment for studying the acquisition of verbs of motion. We need a robot or simulated robot to carry out the motions so they can be labelled by native speakers of various languages. We have decided to use the Jack simulator from U. Penn (Badler et al 1993) and have made considerable progress on getting it to run in our environment and to communicate with the learning code.
Program Construction and Testing: Even after all the ideas are understood, it is a difficult task to build a system that will appropriately interface with the robot simulation, the user and the learning code. Because x-schemas are asynchronous active representations, there are additional technical problems in their implementation. We are using Sather for the implementation.
Training, Test and Evaluation: When a version of the system is complete, we need to run it through the test cycle. Native speakers of various languages label the robot action sequence. We replay the labelled robot motions and have the learning system develop word meanings. The system is then tested and, usually, changes are made to correct errors in design or coding.

Metaphoric mapping in story interpretation

The general problem of interpreting news stories or other real language is clearly beyond the scope of our efforts. We believe we have isolated an important but tractable sub-problem and are working on that in the broader context of our work on conceptual metaphor, connectionist modeling and language learning.

Problem Formulation: The central idea is to demonstrate how inferences from concrete source domains aid in the interpretation of news stories about abstract target domains, here involving international economics and politics. The performance criterion will be that the system deal correctly with novel stories within its domain. Given a pre-parsed story, it should be able to infer the bindings for appropriate unspecified features, using a combination of target and source domain inference.
Data Gathering: We have examined hundreds of stories from the wire services and on-line periodicals in order to better understand what constructions to model. The story interpretation system is currently not adaptive so it is important to have sound intuitions.
Representation and Model Design: As mentioned above, the same basic x-schema/f-struct formalism is used in all current work. In addition, we are employing a form of belief nets (Pearl 1988) for target domain inference.
Aspect: It turns out that linguistic aspect plays a key role in this domain and we have been developing an active model of aspect, both for its own sake and to aid in the inference project.
Program Building: Again for this task domain, we will need to construct and test a running program. Besides the theoretical problems, the main technical issues involve combining active x-schemas, source-target mappings and belief propagation.
Test and Evaluation: Since there is a continuous stream of news stories, we anticipate no difficulty finding novel inputs for testing the system. There will be the standard problem of evaluating a system that only solves part of a larger problem, but we believe that the criterion of inferring bindings is sharp enough for the quality of the results to be determined.

References

Badler, N., Phillips, C. and Webber, B.: Simulating Humans , Oxford University Press, NY, 1993.

Feldman, J.A.: ``Neural Representation Of Conceptual Knowledge,'' Neural Connections, Mental Computation, eds. Lynn Nadel and others, MIT Press, Cambridge, MA, pgs. 68-103, 1989.

Feldman, J., et. al. Miniature Language Acquisition: A Touchstone for Cognitive Science. Proceedings of the 12th Annual Conference of the Cognitive Science Society, 686-693. Cambridge, Mass. MIT Press.

Lakoff, G. and Johnson, M.: Metaphors We Live By, University of Chicago Press, 1980.

Lakoff, G. Women, Fire, And Dangerous Things, U. Chicago Press. 1987

Pearl, J.: Probabilistic Reasoning in Intelligent Systems, Morgan Kaufmann, CA, 1988.

Regier, Terry (1996). The Human Semantic Potential: Spatial Language and Constrained Connectionism, Cambridge, MA: MIT Press.

Shastri, L. and V. Ajjanagadde. From simple associations to systematic reasoning, Behavioral and Brain Sciences Vol. 16, No. 3, 417--494, 1993.

PROGRESS

Since this project officially began in June 1997, there are no new results to report. The project web page , NTL , will always be the best source of the current status.

PROJECT REFERENCES

For an overview of the NTL project, see "Lzero: The First Five Years" Artificial Intelligence Review, v10 pp103-129, April 1996.

Regier, Terry (1996). The Human Semantic Potential: Spatial Language and Constrained Connectionism, Cambridge, MA: MIT Press.

A short paper describing our model of acquisition of hand-action verbs postscript version from Proceedings of the Nineteenth Annual Meeting of the Cognitive Science Society COGSCI-97

A short paper describing our motor control model of Verbal Aspect postscript version from Proceedings of the Nineteenth Annual Meeting of the Cognitive Science Society COGSCI-97

AREA BACKGROUND

Our project is inherently interdisciplinary, involving structured connectionist modeling and cognitive lingiustics. Cognitive linguistics has a bienneial conference of the International Cognitive Linguistics Association and the main journal is Cognitive Linguistics . The structured approach to connectionist modeling (aka neural networks) is not organized as a separate discipline with papers appearing in a wide range of conferences and journals; the annual Cognitive Science Conference always contains several contributions.

AREA REFERENCES

Ballard, Dana. Natural Computation . MIT Press. 1997

Lakoff, George. Women, Fire, And Dangerous Things. U. Chicago Press. 1987

Fauconnier, Gilles. Mappings in Thought & Language. Cambridge University Press. 1997.

Shastri , L. and V. Ajjanagadde. From simple associations to systematic reasoning, Behavioral and Brain Sciences Vol. 16, No. 3, 417--494, 1993.

RELATED PROGRAM AREAS

3 .Other Communication Modalities

4. Adaptive Human Interfaces

5. Usability and User-Centered Design

POTENTIAL RELATED PROJECTS

We have ongoing collaboration with the other ICSI groups including Fillmore's Framenet project and a speech project ICSI Speech Research soon to join ISP. We are also working with Dan Jurafsky at Boulder. The work of Lynn Stein and the Cog project at MIT is similar in spirit and the two groups have maintained good contacts. We believe that some of the more applied efforts could benefit from our work on active semantics, metaphor, etc. and look forward to exploring these possibilities.