Материал: искусственный интеллект

Fig. 10 Hierarchical recurrent neural network [64]

Text Representation

Text Representation

Text representation

Fig. 11 The Þgure shows that the main difference between a sentence-pair task and a sentencebased task is that there is one extra interaction for the matching task

Fig. 12 A demo of SQuAD dataset [85]

Question Answering Differently from expert systems with structured knowledge, question answering in IR is more about retrieval and ranking tasks in limited unstructured document candidates. In some literature, reading comprehension is also considered a question answering task like SQuAD QA. Generally speaking, reading comprehension is a question answering task in a speciÞc context like a long document with some internal phrases or sentences as answers, as shown in Fig. 12. Table 1 reports current popular QA datasets.

In order to compare the neural matching model and non-neural models, we focus on TREC (answer selection), which has limited answer candidates, instead of an unstructured document as context in reading comprehension. Some matching methods are shown in Table 2, which mainly refers to the ACL wiki page.7

3.4 Seq2seq Application

Seq2seq is a kind of task with both input and output as sequential objects, like a machine translation task. It mainly uses an encoderÐdecoder architecture [19, 100] and further attention mechanisms [3], as shown in Fig. 13. Both the encoder and decoder can be implemented as RNN [19], CNN [34], or only attention mechanisms (i.e., Transformer [111]).

7https://aclweb.org/aclwiki/Question_Answering_(State_of_the_art).

The frequent publishing of QA datasets demonstrates that the academic community is paying more and more attention to this task. Almost all the researchers in this community tend to use word embedding-based neural networks for this task

ahttp://cs.stanford.edu/people/mengqiu/data/qg-emnlp07-data.tgz

bhttp://quac.ai/

chttps://stanfordnlp.github.io/coqa/

3.5 Evaluation

The basic evaluations of word embedding techniques are based on the above applications [94], e.g., word-level evaluation and downstream NLP tasks like those mentioned in the last section, as shown in [58]. Especially for a downstream task, there are two common ways to use word embedding, namely as Þxed features or by treating it only as initial weights and Þne-tuning it. We mainly divide it into two part of evaluations, i.e., context-free word properties and embedding-based downstream NLP tasks, while the latter may involve the context and the embedding can be Þnetuned.

Word Property Examples of the context-free word properties include word polarity classiÞcation, word similarity, word analogy, and recognition of synonyms and antonyms. In particular, one of the typical tasks is called an analogy task [70], which

Table 2 State-of-the-art methods for sentence selection, where the evaluation relies on the TREC QA dataset

Recent papers about TREC QA used embedding-based neural network approaches, while previous ones were based on some traditional methods like IR approaches and edit distance

mainly targets both the syntactic and semantic analogies. For instance, Òman is to womanÓ is semantically similar to Òking is to queen,Ó while Òpredict is to predictingÓ is syntactically similar to Òdance is to dancing.Ó Word Embedding methods achieve good performance in the above word-level tasks, which demonstrates that the word embedding can capture the basic semantic and syntactic properties of the word.

Downstream Task If word embedding is used in a context, which means we consider each word in a phrase or sentence for a speciÞc target, we can train the word embedding by using the labels of the speciÞc task, e.g., sequential labeling, text classiÞcation, text matching, and machine translation. These tasks are divided by the pattern of input and output, shown in Table 3.

Fig. 13 An illustration of the proposed Seq2seq (RNN EncoderÐDecoder)

Table 3 The difference of the downstream tasks

Generally speaking, the tasks for the word properties can partially reßect the quality of the word embedding. However, the Þnal performance in the downstream tasks may vary. It is more reasonable to directly assess it in the real-world downstream tasks as shown in Table 3.

4 Reconsidering Word Embedding

Some limitations and trends of word embedding are introduced in Sects. 4.1 and 4.2. We also try to discuss the connections between word embedding and topic models in Sect. 4.3. In Sect. 4.4, the dynamic properties of word embedding are discussed in detail.