Home » Research » Publications » COSMOROE: A Cross-Media Relations Framework for Modelling Multimedia Dialectics
COSMOROE: A Cross-Media Relations Framework for Modelling Multimedia Dialectics
Title
COSMOROE: A Cross-Media Relations Framework for Modelling Multimedia Dialectics
Publication Type
Journal Papers
Year of Publication
2008
Authors
Pastra, K
Journal
Multimedia Systems
Volume
14
Pagination
299–323
ISSN
1432-1882
Abstract
Though everyday interaction is predominantly multimodal, a purpose-developed framework for describing the semantic interplay between verbal and non-verbal communication is still lacking. This indicates not only one's poor understanding of multimodal human behaviour, but also weakens any attempt to model such behaviour computationally. In this paper, we present COSMOROE, a corpus-based framework for describing semantic interrelations between images, language and body movements. We argue that in viewing such relations from a message-formation perspective rather than a communicative goals one, one may develop a framework with descriptive power and computational applicability. We test COSMOROE for compliance to these criteria, by using it for annotating a corpus of TV travel programmes; we present all particulars of the annotation process and conclude with a discussion on the usability and scope of such annotated corpora.
Though everyday interaction is predominantly multimodal, a purpose-developed framework for describing the semantic interplay between verbal and non-verbal communication is still lacking. This indicates not only one's poor understanding of multimodal human behaviour, but also weakens any attempt to model such behaviour computationally. In this paper, we present COSMOROE, a corpus-based framework for describing semantic interrelations between images, language and body movements. We argue that in viewing such relations from a message-formation perspective rather than a communicative goals one, one may develop a framework with descriptive power and computational applicability. We test COSMOROE for compliance to these criteria, by using it for annotating a corpus of TV travel programmes; we present all particulars of the annotation process and conclude with a discussion on the usability and scope of such annotated corpora.