Generalized Tensor-based Multiple Feature Fusion Network using Block Decomposition

Abstract

Feature fusion based on neural networks has drawn great attention of researchers in the enhancement of system performance. Recent studies show the advantage of the tensor-based feature fusion methods, e.g., Tensor Fusion Network, over the traditional early fusion and late fusion approaches given the superior power in the representation of complex dynamics. However, the computational complexity of the tensor-based feature fusion increases exponentially due to the tensor products. Some tensor decomposition methods are applied to solve the problem in previous work, such as Tucker decomposition and CP decomposition. In this paper, we propose a multiple feature fusion method using tensor Block decomposition. Theoretically, it is a general form of two previous tensor decomposition approaches and holds strengths from both of them. Based on the Block decomposition, we construct the corresponding Multiple feature Block Fusion Network (MBFN) practically. Our MBFN is applicable to both unimodal features and multimodal features and it can fuse multiple features at the same time. Experimental studies on a unimodal dataset (i.e., Reuters) prove the effectiveness of multiple feature fusion. Experiments on two multimodal datasets (i.e., MOSI and IEMOCAP) validate the generality and high-performance of the proposed MBFN.