Multimodality and Cross-modality in Art and Science

[Info Point]

*_Multimodality and Cross-modality in Art and Science_*

June 28, 2011 | h. 17.30 | room 128-129 //

*SISSA - *Via Bonomea, 265 -- Trieste, Italy

/_Chair: Raffaella Rumiati_/

/Angelika Lingnau/ - Center for <http://www.unitn.it/cimec> Mind/Brain 
Sciences <http://www.unitn.it/cimec> -- University of Trento (I)

*"The representation of actions in the human brain"*

Humans have very high skills in understanding actions performed by 
others such as waving goodbye or passing over a bottle of water, 
irrespective of the precise way these actions are performed. It has been 
argued that we understand actions by simulating them in our own motor 
system as if we were to execute these actions ourselves, but this view 
is being challenged. Here I present a series of neuroimaging experiments 
that investigate how actions are represented in the human brain, to 
which degree these representations rely on visual input, and how we 
understand actions even if low-level visual features such as the visual 
appearance of body parts are not available.

/Karmen Franinovic(/ - IAD Interaction Design <http://iad.zhdk.ch/en> -- 
ZHdK-Zurich University of the Arts (CH)

*"Amplifying actions: The experience of self-produced sound"*

Each our movement generates sound. Sometimes louder, sometimes quieter, 
sometimes inaudible even to ourselves. As we step on the speed pedal in 
the car, the roaring machine responds to our pressure; as we walk down 
the street, the rhythm of our steps resonates in our body; as we 
chop potatoes, the knife cuts through the vegetable to hit the cutting 
board. In each of these everyday situations, we are the cause of the 
sound, the reason for its sounding. We are not only listening, but 
immersed in and guided by sonic responses of the world, we are ``doing 
with sound".  Yet, this essential part of our everyday life has been 
sparsely considered within scientific and design communities who have 
preferred to study the listening subject. I introduce the enactive sound 
design in order to shift the perspective from such reception-based focus 
towards  the experience of  ``doing with sound". The goal is to design 
sound for action by creating sonic feedback which can affect, guide and 
support physical movement of the user who willingly generates that same 
sound. The enactive sound design approach is directed toward challenging 
the technical understanding of interaction as input triggering some form 
of output and toward supporting a much more situated, direct, less 
abstract and hence, highly experiential framing of the world.

/Pietro Polotti/ - Department of New Musical Technologies and 
Languages-- Conservatory "G. Tartini" of Trieste (I)

*"Visual sonic enaction: Multimodality and embodiment"*

In this seminar, we focus on artistic research about sound and body, 
and, more specifically, about auditory and proprioceptive aspects of our 
interaction with the external world contextualized in interactive 
performances and public art installations. Computationally generated 
sound is employed as representation (sonification) of gesture and its 
expressive contents. Sound, thus, organized according to something 
external (the human gesture) and not constrained by inner musical 
relationships. /Embodied/ and /enactive/ sound, where the former term 
points to the (re-)gaining of body within the human-computer interaction 
domain and the latter to the consideration of the intimate and 
inextricable relationship between action and perception: the sound is 
produced by the machine coherently controlled by the human gesture, and 
it acquires function of representation and guide of the gesture itself 
so to become part of the body joint mechanism.

/Federico Fontan/a - DIMI / Human-Computer Interaction Laboratory -- 
University of Udine

*"Enactive sound design: Movement, touch, audition"*

The use of sound as an information carrier traditionally distinguishes 
the musical from the environmental component. While musical sounds are 
mainly characterized in artistic and emotional sense, conversely the 
environmental sounds are usually associated to activities involving 
objects and contexts of the everyday experience.

At the early perception level, that is, prior to the high-level 
cognitive processes of sound recognition, such two components are indeed 
the same since they are physically generated by the same phenomena. To 
prove this, successful simulation examples will be shown capable of 
virtually reproducing musical as well as environmental sounds starting 
from a unique phenomenon. Furthermore, since the same phenomena are also 
at the base of the generation of vibrotactile cues that are integrated 
with sound in ways to form a unitary multimodal percept, some 
cross-modal experiments will be illustrated which show how sound 
manipulation determines vibrotactile illusions and vice-versa.

/ ***/

/The term multimodality refers to the use of more than one single 
perceptual modality for the definition of our mental representation of 
the world. The modes, by means of which we communicate with the external 
world, depend on our senses and, in most cases, on the information 
coming from more than one mode at a time. For example, when we open a 
window during a sunny day, the "image" that our cognitive system 
processes is not just the result of what we see or what we hear 
separately, but of what we see, what we hear, the fresh air we smell and 
the heat of the sun that we perceive through the skin. The "image" in a 
broad sense that we organize in our mind is the amalgam of all of these 
sensations forming something organic that is more than the mere sum of 
its parts. /

/By trans-modality or cross-modality, we mean those characteristics of 
our perceptual and cognitive system that do not depend on a specific 
modality, and regulate the functioning of two or more sensory channels.
Multimodality and trans-modality are research subjects that require a 
multidisciplinary approach involving disciplines such as experimental 
psychology, neurosciences, computer sciences, virtual reality 
applications, human computer interaction, industrial design, cinema 
theory and practice, audiovisual translation and, last but not least, 
artistic research. In these two days, we will discuss these subjects 
from many points of view in a multidisciplinary fashion, with a 
particular focus on the relationship between the perspectives of 
scientific and artistic investigation.///