Материал: искусственный интеллект

Information Retrieval

Investigating Non-classical Correlations

Between Decision Fused Multi-modal

Documents

Dimitris Gkoumas1(B), Sagar Uprety1, and Dawei Song1,2

1 The Open University, Milton Keynes, UK

{dimitris.gkoumas,sagar.uprety,dawei.song}@open.ac.uk

2 Beijing Institute of Technology, Beijing, China

Abstract. Correlation has been widely used to facilitate various information retrieval methods such as query expansion, relevance feedback, document clustering, and multi-modal fusion. Especially, correlation and independence are important issues when fusing di erent modalities that influence a multi-modal information retrieval process. The basic idea of correlation is that an observable can help predict or enhance another observable. In quantum mechanics, quantum correlation, called entanglement, is a sort of correlation between the observables measured in atomic-size particles when these particles are not necessarily collected in ensembles. In this paper, we examine a multimodal fusion scenario that might be similar to that encountered in physics by firstly measuring two observables (i.e., text-based relevance and image-based relevance) of a multi-modal document without counting on an ensemble of multi-modal documents already labeled in terms of these two variables. Then, we investigate the existence of non-classical correlations between pairs of multi-modal documents. Despite there are some basic di erences between entanglement and classical correlation encountered in the macroscopic world, we investigate the existence of this kind of non-classical correlation through the Bell inequality violation. Here, we experimentally test several novel association methods in a small-scale experiment. However, in the current experiment we did not find any violation of the Bell inequality. Finally, we present a series of interesting discussions, which may provide theoretical and empirical insights and inspirations for future development of this direction.

Keywords: Multi-modal information retrieval · Non-classical correlations · Decision fused multi-modal documents · CHSH inequality

1 Introduction

Nowadays, the Web surrounding us often involves multiple modalities - we read texts, watch images and videos, and listen to sounds. In general terms, modality refers to a certain type of information and/or the representation format in which information is stored. A research problem is characterized as multi-modal

c Springer Nature Switzerland AG 2019

B. Coecke and A. Lambert-Mogiliansky (Eds.): QI 2018, LNCS 11690, pp. 163–176, 2019. https://doi.org/10.1007/978-3-030-35895-2_11

164 D. Gkoumas et al.

when it includes multiple such modalities. Integrating unimodal representations from various input modalities and combining them into a compact multi-modal representation, called multi-modal fusion, o ers a possibility of understanding in-depth real world problems. For instance, in information retrieval, suppose a user types in a text query to retrieve multi-modal documents consisting of an image and a caption as shown in Fig. 1. One can notice that the query term “plane” can be matched in both textual and visual modalities of the given multimodal document. However, the query term “London” can be matched only in its textual modality, while the term “sunset” only in its visual modality. This implies that only when the text and image modalities are fused, we get the benefit of complementary information, in turn increasing the precision of information retrieval.

Fig. 1. Example of multi-modal information retrieval

The main challenge of multi-modal fusion is to capture inter-dependencies and complementary presence in heterogeneous data originating from multiple modalities. In the literature, two main approaches to the fusion process have been proposed: (a) feature level or early fusion and (b) decision level or late fusion [4]. Early fusion involves the integration of multiple sources of raw or preprocessed data to be fed into a model, which finally makes an inference as illustrated in Fig. 2. In contrast, late fusion refers to the aggregation of decisions from multiple classifiers, each trained on separate modalities as shown in Fig. 3.

There are distinctive issues that influence the multi-modal fusion process. Correlation between di erent modalities is one of them. Correlation can be perceived either in low-level features, e.g., raw data, or high-level features that are obtained on di erent classifiers, e.g., semantic concepts [4]. In both cases, correlation informs us how to fuse di erent modalities. In the early fusion, we fuse multi-modal information either by projecting all of the modalities to the same space (Fig. 4(c)), called joint representations, or by learning separate representations for each modality but coordinate them through a similarity measure (Fig. 4(b)) [5]. In both approaches, the construction of the multi-modal spaces is based on correlations among di erent modalities. The late fusion process can be rule-based, e.g., by linear weighted fusion and majority voting rules, or based on classification-based methods, e.g., support vector machines [4]. In many cases, the correlation among di erent modalities provides additional cues that are very useful for aggregating decisions either by following a rule-based approach or a classification-based approach. In addition, the absence of correlation may equally provide valuable insight with respect to a particular scenario or context.

Fig. 4. Construction of multi-modal spaces

There are various statistical and probabilistic forms of correlation that have been utilized by researchers, being causal or not. Since our experiment focuses on late fusion only, we briefly report the most important methods for computing correlations between decisions from multiple modalities. Specifically, decision level correlation has been exploited in the form of causal link analysis, causal strength, and agreement coe cient [4]. In all cases, the basic idea of correlation is that a modality can help predict or enhance another modality.

In quantum mechanics, correlation has been also an important topic. In quantum mechanical framework, uncertainty may occur not only when the elements are collected in ensemble but also when each of them is in a superposed state. In quantum theory, making an observation on one part of a system instantaneously could a ect the state in another part of a system, even if the respective systems are separated by space-like distances. Such a quantum correlation presents some peculiarities which led to the notion of entanglement. Entanglement is a

166 D. Gkoumas et al.

sort of correlation between observables measured in atomic-size particles, such as photons, when these particles are not necessarily collected in ensembles.

Despite entanglement being a kind of correlation, there are some basic differences between entanglement and the classical correlation encountered in the macroscopic world. A classical correlation is a statistical relationship, causal or not, between two random variables. In entanglement, besides correlation, cause exists as well since the correlation does not depend on an underlying value attached to the particles. Instead, it depends on what is measured on either side. This non-classical property of quantum entanglement motivates us to investigate non-classical correlations between multi-modal decisions as shown in Fig. 5. At first, we calculate the probability of relevance for each document, with respect to both text-based and image-based modality concerning a multimodal query as shown in Fig. 5. Then, we check for any violation of Bell’s inequalities based on the estimated relevance probabilities for each possible pair of decision fused multimodal documents in a dataset. Our assumption is that if a pair of decision fused multi-modal documents is entangled, then knowing that a document is relevant concerning the text-based representation for a query, then we can simultaneously predict with certainty the relevance of the other document concerning the text-based and image-based representation for the same query.

Fig. 5. Investigation of non-classical correlations between decision fused multi-modal documents

The rest of the paper is organized as follows. Section 2 presents a brief review of related work. In Sect. 3 we provide a foundation in quantum entanglement and Bell inequality, while in Sect. 4 we explain some basic concepts of geometry in information retrieval and then formalize the proposed model. Section 5 reports all the experiment settings. In Sect. 6 we report and discuss the results. Finally, Sect. 7 concludes the paper.