Comparing Spatial Information in Speech/Gesture and Signed Language
SARAH TAUB, PILAR PIÑAR & DENNIS GALVAN
Gallaudet University

1. Introduction

Linguists have always wanted to compare how languages express information. Do users of different languages communicate essentially the same information, or does language play a critical role in shaping what is included and excluded from a narrative? The "null hypothesis" or simplest assumption might be that humans in general attend to and express the same conceptual elements universally. This paper presents a research methodology that should allow us to evaluate that hypothesis. Earlier work has suggested that the hypothesis fails; that spoken languages differ to a large degree in what is expressed, and that signed languages draw on different resources and may express more information (in certain domains) than spoken languages. Taking spatial motion events as a test case, this paper argues that it is not appropriate to compare languages based on speech alone, or to compare speech to signed language; rather, the speech/gesture complex produced in discourse and narrative is analogous to signed language, and constitutes the appropriate level for cross-linguistic comparison. If speech/gesture is taken as the level of comparison, we predict more universal trends in expression of information. The technique presented in this paper should allow collection of quantitative data over a large range of languages.

Talmy (1985) provided an analysis of spatial events upon which much subsequent typological research has been based. He divided motion events into such categories as Figure, Ground, Motion, Path, and Manner, then classified languages as to how they typically expressed each class of information. In particular, verb-framed languages such as Spanish express Path information through verbs, and satellite-framed languages such as English express Path through other members of the verb complex ("satellites"). As examples, consider (1), a typical English sentence in which a bird's Path through a window is described with a verb particle and prepositional phrase, and (2), a typical Spanish sentence where the same Path is described with a verb.
(1) It flew out of the window.
(2) Salió por la ventana.
went-out-3rd through the window
Talmy noted that verb-framed languages are less likely to provide complex specifications of Path, as they would require a long series of verbs and thus many clauses (making the discourse feel "over-detailed"); as Manner requires an extra verb or gerund in these languages, it is often left out as well. On the other hand, satellite-framed manner-type languages such as English incorporate Manner easily in the main verb, then typically present long series of Path elements via several satellites. (3) demonstrates the use of a second verb in Spanish to specify Manner, and (4) demonstrates the use of multiple verb particles and prepositions in English to specify complex Paths.
(3) Salió volando por la ventana.
went-out-3rd flying through the window
(4) It flew right back out the window, across the street, and into the forest.
Slobin's (1996) studies of Spanish and English written narratives indicate that differences between the two languages (exemplars of the verb-framed and satellite-framed types) persist over a longer time-period than the individual clause. That is, Spanish narratives consistently present fewer explicit Manner and Path elements than English narratives; they do not "catch up" in the course of the story. This analysis, however, considers only written texts and not oral narrative or discourse. Studies show (e.g., Kendon 1986, McNeill 1992) that the gesture that accompanies speech also contains conceptual elements (though it is a matter of hot debate whether such gesture is intended to communicate those elements).

Galvan & Taub (forthcoming) used a similar method of counting conceptual elements to compare the information expressed in American Sign Language and English narratives. Narratives were elicited by the same stimulus (the wordless storybook Frog, Where Are You?) and so could be compared more easily. They found that in nearly every category (Ground elements being the only exception), ASL storytellers included more conceptual information; in general, the signed narratives relied on ASL's highly iconic grammatical resources to compress a huge amount of spatial information into each sentence. But again in this study, analysis of the English narratives looked only at the speech produced by the subject; there was no record of gesture or prosody.
Many people have noted similarities between signed languages and speakers' gestures; many others have noted differences. (The debate has a political edge, since when ASL was thought to be "only gesture," deaf signers were thought to be without language and mentally impaired.) Liddell (2000) has created a precise explanation for this phenomenon based on conceptual blending. According to Liddell, both signers and speaking gesturers create a "blend" (Fauconnier & Turner 1996) between an imagined mental space and Real space (i.e., their conceptual model of the space in which they stand). In the blend, imagined entities are conceived of as located in the space around the person. Iconic and deictic gestures may be aimed toward those entities, or may track their progress through space.

For spoken languages, these gestures are loosely integrated with the speech signal – the emphatic stroke phase of the gesture occurs precisely with a spoken word or words that relates to its meaning (McNeill 1992), but the preparatory and refractory phases may vary in timing. (Indeed, McNeill notes that the precise timing of these other phases may reveal details of the speaker's thought processes). For signed languages, on the other hand, these gestures are tightly constrained to unify with lexical and grammatical elements. Indeed, certain signs (e.g., pronouns, some verbs, classifier signs) are ungrammatical if they do not contain a gestural element. For one example, ASL pronouns must be placed in signing space so as to indicate the entity to which they refer; for another, verbs such as GIVE must indicate their subject or Source argument and object or Goal argument by moving from a space associated with the Source to a space associated with the Goal. We might describe these gestural elements of signed languages as "grammaticalized gesture."

If Liddell's proposal is correct, then signed language is more fully analogous to the speech/gesture complex than to speech alone. If co-speech gesture presents additional conceptual information, then comparison of speech/gesture across spoken languages may also prove more fruitful than comparison of speech alone. Elaborating on the null hypothesis above, if gesture is taken into consideration, cross-linguistically and cross-modally, in comparable narratives we may expect to see the following:
A. Similar amounts of conceptual information
B. i. Similar types of information in speech and lexical sign elements
ii. Similar types of information in gesture and gestural sign elements
Given the research of Talmy and Slobin, we may hypothesize more specifically the following:
C. Spanish uses gesture to add more Path and Manner information.


2. Methods

Subjects were 12 native users of English, 12 of Spanish, and 11 of ASL. Each subject was paired with another native user of his/her language. Subjects watched animated cartoons involving the adventures of a cat and a bird; they were instructed to tell what happened in the cartoon to their partner clearly enough so that the partner could then tell the story to third person. All narratives were videotaped. Partners' narratives were recorded but not analyzed in this study.
In the particular scene analyzed here, the cat and the bird are in high-rise apartments across the street from each other. The cat has been studying the bird through their windows. The cat swings from his window to the bird's window on a rope, but misses the window, crashes into the wall, and falls to the street below. The researchers developed a list of conceptual elements present in the cartoon, including potential Figure, Ground, Path, Manner, and Instrument elements. The analysis compiled 1) total number of conceptual elements expressed; 2) number of elements expressed through speech or lexical sign elements; and 3) number of elements expressed through gesture or gestural sign elements. Lexical and gestural sign elements were distinguished on the following criterion: if the element (e.g., handshape, movement, location, complete sign) is conventionalized in the lexicon of ASL, than it was counted as lexical; if it is not, then it was counted as gestural. A number of signs contained both lexical and gestural elements.

3. Results

Tables 1-4 summarize the results. In each table, the first row shows items expressed through lexical means, and the second row shows items expressed through gestural means; note that some elements are represented in both rows. The third row shows all elements in both modes, while the fourth row excludes elements that were expressed only through gesture. The purpose of the fourth row is to simulate the results of previous methodologies which did not include gesture. An ANOVA was performed on the data; means that are statistically different are printed in different colors for easy identification.

Table 1: Total Conceptual Items: Statistics
Mean Spanish Mean English Mean ASL F Sig
lexical 12.67 14.33 6.82 23.08 0
gestural 9.33 9.58 13.45 7.979 0.002
all 15.33 16.75 15.36 1.11 0.342
all w/o gesture 12.67 14.33 15.36 2.774 0.077

Table 2: Figure/Ground Conceptual Items: Statistics
Mean Spanish Mean English Mean ASL F Sig
lexical 4.42 4.5 3.36 2.769 0.078
gestural 1.67 1.5 2.55 3.531 0.041
all 4.42 4.5 3.36 2.769 0.078
all w/o gesture 4.42 4.5 3.36 2.769 0.078Table 3: Path Conceptual Items: Statistics
Mean Spanish Mean English Mean ASL F Sig
lexical 4.17 5.17 1.36 37.3 0
gestural 4.92 4.92 5.82 1.983 0.154
all 5.83 6.33 6.27 0.699 0.504
all w/o gesture 4.17 5.25 6.27 11.3 0Table 4: Manner Conceptual Items: Statistics
Mean Spanish Mean English Mean ASL F Sig
lexical 3.17 3.58 1.27 16.79 0
gesture 2.33 2.58 4.27 8.812 0.001
all 4 4.5 4.45 0.861 0.432
all w/o gesture 3.17 3.67 4.45 3.272 0.051

Figures 1-4 illustrate the same data graphically; in these figures, the yellow bar segments represent items presented through gesture only, the blue segments represent items presented through lexical means only, and the green segments represent items that were presented through both means (though presentations in the two modes were rarely simultaneous). Thus, the total height of the bars represents the total number of items for each language.


4. Conclusions

The data suggest a significant role for gesture in equalizing the expression of information across languages and modalities.

A. Similar amounts of conceptual information
This hypothesis is supported. There were no significant differences between languages in total amounts of information. On the other hand, English and Spanish clearly convey more information through lexical means, and ASL conveys more information through gestural means. Thus, gesture may be said to play a role in equalizing the amount of information expressed.

B. i. Similar types of information in speech and lexical sign elements
ii. Similar types of information in gesture and gestural sign elements
These hypotheses are supported. Certain types of conceptual item are preferentially expressed through one mode or the other: Figure/Ground information always shows up in lexical elements, and Path information always shows up in gestural elements. But spoken and signed languages vary as to how much those types show up in their non-preferred mode. Thus, all subjects support Figure/Ground information with some gestural elements but this is significantly more prevalent for ASL than for English and Spanish. On the other hand, ASL only rarely supports Path information with lexical elements, while English and Spanish do this to a considerable degree, adding non-gestured Path items in their speech. These two tendencies make sense, in that identification of things (Figure/Ground elements) is most easily accomplished through memorized lexical elements or category names, while the details of Paths are more easily shown through free gesture than through fixed lexical categories.
Manner, however, shows no overall mode preference. Instead, users of the spoken languages express it largely lexically, adding little new information through gesture; while users of the signed language show the opposite pattern, expressing most Manner information through mapped elements.

C. Spanish uses gesture to add more Path and Manner information.
This hypothesis is not supported. The results show no significant difference between English and Spanish for any measure, lexical or gestural. On the other hand, trends are visible that support Slobin and Talmy’s results; notably, the difference between English and Spanish in Path information expressed through lexical means approaches significance (sig. <.059), while if gesture is included the difference disappears. It is possible that a larger sample size would make the trends significant.

5. Summary

We find that in both of the spoken languages, subjects express a significant amount of additional information through gesture; this leads to a rough equalization of amount of information expressed.
Particular categories of conceptual items appear universally in speech or universally in gesture, optionally supported by the other mode. This pattern would not have been apparent had we not separated gestural and lexical elements within signed language, and brought in gesture as well as speech for cross-modal comparisons.
We conclude that the methodology of comparing speech/gesture complexes with each other and with signed languages produces interesting results, and we recommend it as the standard for future typological investigations.

References

Galvan, Dennis, & Sarah F. Taub. Forthcoming. The encoding of motion information in American Sign Language. In R. Berman, D. Slobin, S. Strömqvist, & L. Verhoeven (eds.) The Frog Story Revisited. Hillsdale, NJ: Lawrence Erlbaum Associates.
Fauconnier, Gilles, & Mark Turner. 1996. Blending as a central process of grammar. In Adele Goldberg (ed.) Conceptual Structure, Discourse, and Language, pp. 113-130. Stanford, CA: CSLI.
Slobin, Dan I. 1996. Two ways to travel: verbs of motion in English and Spanish. In M. Shibatani & S. A. Thompson (eds.) Essays in semantics. Oxford: Oxford University Press.
Kendon, Adam. 1986. Some Reasons for Studying Gesture. Amsterdam: Mouton de Gruyter.
Liddell, Scott. 2000. Blended spaces and deixis in sign language discourse. In D. McNeill (ed.) Language and Gesture. Cambridge: Cambridge University Press.
McNeill, David. 1992. Hand and Mind: What Gestures Reveal About Thought. Chicago: University of Chicago Press.
Talmy, Leonard. 1985. Lexicalization Patterns: Semantic Structure in Lexical Forms. In T. Shopen (ed.) Language Typology and Syntactic Description, Vol.3: Grammatical Categories and the Lexicon. Cambridge: Cambridge University Press.