Learning to Answer Questions From Image Using Convolutional Neural Network

Abstract

In this paper, we propose to employ the convolutional neural network (CNN) for the image question answering (QA). Our proposed CNN provides an end-to-end framework with convolutional architectures for learning not only the image and question representations, but also their inter-modal interactions to produce the answer. More specifically, our model consists of three CNNs: one image CNN to encode the image content, one sentence CNN to compose the words of the question, and one multimodal convolution layer to learn their joint representation for the classification in the space of candidate answer words. We demonstrate the efficacy of our proposed model on the DAQUAR and COCO-QA datasets, which are two benchmark datasets for the image QA, with the performances significantly outperforming the state-of-the-art..

The contributions of this work:

• An end-to-end CNN model for learning to answer questions about the image. Experimental results on public image QA datasets show that our proposed CNN model surpasses the state-of-the-art.
• Convolutional architectures to encode the image content, represent the question, and learn the interactions between the image and question representations, which are jointly learned to produce the answer conditioning on the image and question.

Convolutional Neural Network for Image Question Answering (ConvIQA)

The image QA task, resembling the visual Turing test, differs with the other multimodal learning tasks between image and sentence, such as the automatic image captioning. The answer produced by the image QA needs to be conditioned on both the image and question. As such, the image QA involves more interactions between image and language. The questions about the images are very specific, which requires a detailed understanding of the image content.

ConvIQA is to predict the answer a given the question q and the related image I.

In order to make a reliable prediction of the answer a, the question q and image I need to be adequately represented. Based on their representations, the relations between the two multimodal inputs are further learned to produce the answer.

ConvIQA consists of three individual CNNs:

• Image CNN is used to encode the image content for the image question answering task.
• Sentence CNN employed to model the question. The convolution and max-pooling prorcesses exploit the the local relation signals between consecutive words. More layers of convolution and max-pooling can help to summarize the local interactions between words at larger scales and finally reach the whole representation of the question.
• Multimodal Convolution Layer takes the image representation and question representation as input and fuse the multimodal inputs together to generate their joint representation for further answer prediction.

Experimental Results

Image question answering performances on DAQUAR-All

Image question answering performances on DAQUAR-Reduced 

Image question answering performances on COCO-QA

Reference

Contact Me

Back to top