COSMOROE Annotation Guide

ΤίτλοςCOSMOROE Annotation Guide
Publication TypeTechnical Reports
Year of Publication2015
AuthorsPastra, K
PublisherCognitive Systems Research Institute
Place PublishedAthens, Greece
ISSN Number2407-9952

This technical report provides guidance on annotating semantic relations mainly between language, images and sounds as they occur in naturalistic contexts, such as audiovisual material and follow the COSMOROE cross-media semantics framework.

COSMOROE (CMR) is a descriptive framework for modeling the semantic interplay between different means of expression, when formulating multimodal messages. It identifies a number of semantic association types through which integration of modalities is served in multimodal message formation processes (Pastra 2008). In this document, we present an annotation scheme for COSMOROE-based analysis of multimedia documents of any kind. The annotation required is multi-faceted and with a number of by-products. Thus, this technical report aims at providing guidelines for use by any annotator regardless scientific background and expertise. To this end, examples have been included, as well as tips and lists of ‘interesting’ cases as we have indicated them in the last 5 years, through the annotation of TV travel series, newspaper caricatures and Hollywood movies by several teams of annotators with diverse backgrounds. In what follows, one focuses on annotation of audiovisual (video) documents; however, annotation of static images (captioned or surrounded by accompanying text) or other files with any combination of language, image, and sound follows along the same lines.

The first release of COSMOROE annotated multimodal data repackaged for different research needs is available at:

Online navigation to the annotated data can be carried out through the use of the COSMOROE Search Engine at:

Citation KeyPastra2015