Dynamic Probabilistic CCA for Analysis of Affective Behaviour and Fusion of Continuous Annotations

Nicolaou, Mihalis; Pavlovic, Vladimir and Pantic, Maja. 2014. Dynamic Probabilistic CCA for Analysis of Affective Behaviour and Fusion of Continuous Annotations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7), pp. 1299-1311. ISSN 0162-8828 [Article]


Download (1MB) | Preview

Abstract or Description

Fusing multiple continuous expert annotations is a crucial problem in machine learning and computer vision, particularly when dealing with uncertain and subjective tasks related to affective behavior. Inspired by the concept of inferring shared and individual latent spaces in Probabilistic Canonical Correlation Analysis (PCCA), we propose a novel, generative model that discovers temporal dependencies on the shared/individual spaces (Dynamic Probabilistic CCA, DPCCA). In order to accommodate for temporal lags, which are prominent amongst continuous annotations, we further introduce a latent warping process, leading to the DPCCA with Time Warpings (DPCTW) model. Finally, we propose two supervised variants of DPCCA/DPCTW which incorporate inputs (i.e., visual or audio features), both in a generative (SG-DPCCA) and discriminative manner (SD-DPCCA). We show that the resulting family of models (i) can be used as a unifying framework for solving the problems of temporal alignment and fusion of multiple annotations in time, (ii) can automatically rank and filter annotations based on latent posteriors or other model statistics, and (iii) that by incorporating dynamics, modeling annotation-specific biases, noise estimation, time warping and supervision, DPCTW outperforms state-of-the-art methods for both the aggregation of multiple, yet imperfect expert annotations as well as the alignment of affective behavior.

Item Type:


Identification Number (DOI):


Additional Information:

(c) 2014, IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Departments, Centres and Research Units:



July 2014Published
9 January 2014Published Online
27 November 2013Accepted

Item ID:


Date Deposited:

21 Mar 2016 17:08

Last Modified:

29 Apr 2020 16:15

Peer Reviewed:

Yes, this version has been peer-reviewed.



View statistics for this item...

Edit Record Edit Record (login required)