A Study on Coordination of Speech, Gesture and Breathing Movements

Abstract
This paper describes an experiment in which the subjects were each instructed to synchronize
articulations of /ba/ and movements of the right-hand wrist at increasing frequencies
in two different modes of coordination. The means and the standard deviations
of relative phase among different movements were used as indices of the degree of
equilibrium of the coordination to discover what control parameters contribute to the
dynamics of the coordination characterized by the relative phase as order parameters.

Introduction

People often gesture while articulating meaningfully related speech segments.
To explain such coordination between speech and gesture produced at different parts of
the body, it is often either argued or merely assumed that the parameters determining
how articulation and gesticulation are coordinated are planned and specified in the
command system usually associated with the central nervous system (CNS). Coordination
between speech and gesture can be observed not only within single individuals,
but also between individuals, however (Furuyama 2000). This simple observation
poses a fundamental question to the aforementioned explanation of speech-gesture coordination
because inter-personal coordination of speech and gesture is achieved not by
a system in which the sub-systems are entirely connected mechanically, but by one in
which the sub-systems are connected only functionally at best.

The present study attempts, from the viewpoint of a dynamical systems approach,
to test the hypothesis that coordination of speech and gesture, regardless of
whether it is done intra-personally or inter-personally, is based on the hierarchized coordination
of movements and/or actions among articulatory movements, limb movements,
and breathing movements rather than only on a set of motor commands to specify
every single degrees of freedom involved in the coordination.

The physiologist Nicolas Bernstein (1967) proposed the idea of coordination to
solve the “Degree-of-Freedom Problem” (Turvey 1990). This is a problem with a system
such as a human body of having too many degrees of freedom to be each specified
by the brain with a set of motor plans. As Bernstein proposes, by coordinating different
parts of our body, we can decrease the degrees of freedom to a significantly smaller
number. The coordinated structure consisting of body parts, however, presents additional
implications. Kelso conducted an experiment in which the subject was asked to
synchronize the oscillation of her/his fingers at a frequency specified by a metronome
pulse in either the in-phase mode (two limbs are moving in synch such that they are in
the same position of a cycle simultaneously) or the anti-phase mode (two limbs are
moving out of synch such that they are in the opposite position of a cycle simultaneously)
(Figure 1). Kelso’s results show that a breakdown of the anti-phase mode, but
not the in-phase mode is found at higher frequencies of oscillation. The breakdowns
are sometimes so devastating such that the anti-phase mode undergoes a “phase transition”
to the in-phase mode. Similar results are obtained in coordination between
Nobuhiro Furuyama
National Institute of Informatics
(JAPAN)
Hiroki Takase
Shinshu University
(JAPAN)
Koji Hayashi
The University of Tokyo
(JAPAN)
breathing movements where abdomen movements and chest movements constitute one coordinated system, and in coordination between breathing movements and limb movements
(Takase et al. 1999). The finding of such a coordinated structureof breathing movements and
limb movements is extremely relevant to our concern here with both intra- and inter-
personal coordination of speech and gesture. This is because the articulation of
speech is a special kind of breathing and the gesticulations are a special kind of limb
and/or body movements that assume a certain semiotic function. These studies suggest
that the frequency of oscillation conditions the way in which different parts of the body
are coordinated, and that different frequencies have different consequences for the
in-phase and anti-phase modes of coordination. Moreover, the finding that the coordinated
structure can and in fact does break down or shift entirely to the opposite pattern
suggests that coordinated limb movements (and breathing movements) can and in fact
do assume their own spontaneous and self-organizational regularity, and that they
sometimes even override the ‘intended motor plan’ made and sent from the central
nervous system that supposedly controls the entire movement.

One might argue that if this kind of phenomenon is only found in intra-personal
coordination of movements of different body parts of a single individual, it must still be
under the complete control of the motor plan made by the central nervous system.
Crucial to the context of the present study, however, some of the phase transition phenomena
mentioned above are even replicated by inter-personal coordination experiments
in which two individuals sitting side by side are asked to visually coordinate the
oscillation of their lower legs in the sagittal plane in either the in-phase mode or
anti-phase mode (e.g., Schmidt et al. 1990). Furthermore, similar phenomena are observed
in inter-personal coordination between breathing movements and limb movements
(Takase et al., in press) If we consider the already attested synchronous relationship
between speech and gesture as the coordinated oscillating movements of different
body parts (Nobe 1996), a dynamical systems approach to coordination of
movements and actions opens up wide possibilities for explaining inter-personal as well
as intra-personal coordination of speech and gesture without recourse to the system
consisting entirely of mechanically connected components, but by invoking the system
consisting at least in part of only functionally connected components.

Experiment

Purpose of the Study;
The present experiment attempts to examine the stability and variability among
articulatory movements, wrist movements and breathing movements at the ribcage as
well as the abdomen. The experimental parameters manipulated were relative phase
mode and oscillatory frequency of the coordinated movements.

Method of Data Collection;
Subjects: Ten undergraduate/graduate students (5 male and 5 female) participated in
this experiment. They were all right-handed. None of them had any problems with
breathing or motor control at any part of the body relevant to the present study.
Procedure: The subjects were each asked to synchronize the articulation of /ba/ and extending/
bending wrist movements either in in-phase mode or anti-phase mode of oscillation
at the rate specified by cyclic sound pulses emitted from a Macintosh computer.
The rate of the sound pulses was set to increase in increments of 0.2 Hz, covering eight
frequencies in the range 0.6 Hz to 2.4 Hz, and there were ten pulses for each frequency.
In addition, the subjects were asked to maintain constant amplitude and pitch of articulation
while synchronizing with their wrist movements, to make their limb movements
as smooth as possible and not accented, and to elongate the vowel of /ba/ as long as the
corresponding phase lasted. The subjects each performed five sets of two different
tasks (i.e., two relative phase modes). Thus, they each performed ten trials in total.
The order of the trials was randomized within each set. There was a minimum interval
of two minutes between each trial.

Data Acquisition: The articulated speech sound was acquired with a condenser microphone
(SONY ECM-360) connected to an amplifier (SONY CFD-700). The wrist
movements were measured with an electro goniometer (Penny & Giles, Inc.) attached to
the right wrist of the subject. The chest and abdomen breathing movements were
measured with Respitrace (Ambulatory Monitoring, Inc.). All of these data were recorded
by a PC (EPSON) with a DAQ board (National Instruments, AT-MIO-16). The
entire experimental scene was videotaped throughout.

Data Analysis: The onset of articulation of /ba/ was manually plotted on the waveform
displayed in the analysis tool, which was designed and made to be used in the present
study with LabVIEW (National Instruments). The onset of each articulation was defined
to be a point from which the amplitude increases abruptly, compared with the
segment before, where no sound can be heard. Since the target sound is a combination
of the consonant /b/ and the vowel /a/, the increment in amplitude occurs in two stages
in many cases. In such cases we used the first increment in amplitude as an index of
the onset of articulation, because that must be closer to the onset of articulation of /ba/
intended by the subject her/himself.

The time series data of wrist movements was smoothed by the triangular
moving average method (the average of five points; i.e., the point in question and two
points before and after it). The time series øn of the relative phase between wrist
movement and articulated speech sound was computed by the following formula:
1 n Wr n Wr
n Sp n Wr
extension peak of extension peak of
on articulati of start of extension peak of
2
+ • •
• •
-
-
• =
time time
time time
n þ • (1)
ba..., ba..., ba...
ba..., ba..., ba...
In-phase Mode Anti-phase Mode
Figure 9. In-phase mode and anti-phase mode of coordination of articulation of /ba/
and bending and extending movements of the right-hand wrist.
where Wr is the peak of wrist movements (i.e., the turning point from the extending
movement to the bending movement when in-phase mode is intended and that from the
bending movement to the extending movement when anti-phase mode is intended) and
Sp is the onset of articulated speech sound.

The pattern of the coordination of movements was evaluated by computing the
means of phase difference between the wrist movements and the articulations (mean ø)
in each oscillatory frequency. The stability of the coordination was evaluated by
computing standard deviation of the phase difference between the wrist movements and
the articulations (SD ø) in each oscillatory frequency. The first data points after each
change of metronome frequency were not included in the data analyses so as to eliminate
the instability introduced by each frequency change.

Results;
The data of one of the ten subjects were entirely excluded from the quantitative analyses
below (i.e., (2)-(4)) because of a technical problem during the data acquisition. Also
excluded were all of the data points obtained at frequencies beyond 2.4 Hz and all of the
data points at which any of the cases (1a) through (1c) below were observed.
(1a) The targeted articulation is missing. This was observed at least once with four
subjects when the oscillatory frequency was 1.6 Hz or above. Three of these four
subjects failed to articulate almost every time the oscillatory frequency became
higher than a certain point.
(1b) The targeted articulation and the wrist movements are both missing. This was
observed twice within the same trial by one subject when the oscillatory frequencies
were 2.0 Hz and 2.4 Hz.
(1c) The wrist movements become gradually delayed with respect to the articulations to
the extent that the delayed cycle reaches approximately one cycle or above. This
was observed with only one subject in four out of the five trials of anti-phase mode.
Having excluded all these data points from the entire data set, we conducted statistical
analyses. The results of the analyses show the following:
(2) ANOVA (relative phase mode x oscillatory frequency) conducted on mean ø of articulations
and the wrist movements shows no significant main effect of either
phase mode or oscillatory frequency.
(3) ANOVA (relative phase mode x oscillatory frequency) conducted on SD ø of speech
and the wrist movements shows no significant main effect of phase mode
(F(1,8)=0.99, ns), but a significant main effect of frequency (F(8,64)=9.88, p<.01).
Further tests show that the higher the oscillatory frequency, the significantly higher
the SD ø becomes.
(4) ANOVA (relative phase mode) conducted on the ribcage movements and the wrist
movements shows that SD ø is higher when the intended relative phase is
anti-phase mode than when it is in-phase mode (F(1,7)=6.35, p<.05). (Data of one
of the subjects were not taken into consideration as there was a problem with an
algorithm that was meant to capture the data points.)

Discussion and Conclusion

(1a) and (1b) show that the continuous articulation becomes difficult for some
subjects as the oscillatory frequency increases. (1c), though it was only observed with
a single subject, may possibly be what is a so-called prolonged phase transition and it
calls for further examination. (2) may have resulted from the fact that in the present
study the control of population was minimum and it calls for further study with more
population control. (3) is consistent with (1a) and (1b). (4) shows that the relative
phase between breathing movements and wrist movements is less stable when the intended
phase mode is anti-phase than when it is in-phase. This implies that breathing
movements are a crucial part of the coordination between articulations and wrist movements/
gesticulations.

One of the main results of the present study is that the relative phase between
breathing movements and wrist movements is less stable when the intended relative
phase mode between articulations and wrist movements is anti-phase than when it is
in-phase. This suggests that breathing movements are a crucial part of the coordination
between articulations and wrist movements. To fully understand the mechanism
underlying speech-gesture coordination, then, we should further consider the coordination
by taking into consideration breathing movements as a sub-system of the coordination
between articulation and gesticulation. The other results call for further examination.
Particularly interesting is the case of prolonged phase transition that we observed
with one of our subjects.

Acknowledgements

The present study is supported by the Grant-in-Aid for Scientific Research from the Japan
Society for the Promotion of Science (Contract number: 13224095).

References

Bernstein, N. (1967). The coordination and regulation of movements. London: Pergamon.
Furuyama, N. (2000b). Gestural interaction between the instructor and the learner in
origami instruction: In McNeill, D. (Ed.), Language and Gesture. Cambridge:
Cambridge University Press.
Kelso, J.A.S. (1984). Phase transitions and critical behavior in human bimanual coordination.
American Journal of Physiology: Regulatory, Integrative and Comparative,
246, R1000-R1004.
Schmidt, R.C., Carello, C., & Turvey, M.T. 1990. Phase transitions and critical fluctuations
in the visual coordination of rhythmic movements between people. Journal
of Experimental Psychology: Human Perception and Performance, 16, 227-247.
Takase, H., Furuyama, N., Mishima, H., & Haruki, Y. (in press). Interpersonal coordination
between breathing and limb movements.
Takase, H., Mishima, H. & Haruki, Y. (1999) Coordination between breathing and
limb movement, Poster presented at ICPA-X at Edinburgh University.
Turvey, M.T. (1990). Coordination. American Psychologist, 45, 938-953.