WebQuestions-Movies Property Selection QA Dataset
=================================================

This task tries to find property path which leads from question entity to answer entity.
For example, in the question "Who directed Harry Potter" the correct property (relation) is "Directed by".

The dataset consists of pairs question - property path. For each question, there are such paths of maximal length 2
which are connected to each enitity linked during entity linking. If the question has two or more entities, and they can be connected
by path with the same first property as the path from question entity to answer entity, the second ("witness") property is concatenated to path
as third property. The properties on the path are separated by # symbol.

The dataset was created from the WebQuestions + moviesE QA datasets, as contained in https://github.com/brmson/dataset-factoid-webquestions
branch movies (commit c2b3c4b).  The data source for answering is the Freebase knowledge graph dump.
The paths were generated based on entity linking etc. as generated by YodaQA v1.5.

There are two types of data sets:

  * trainmodel.csv, test.csv, val.csv contain original question representation (val.csv also contains devtest split of the source QA dataset).
  * Files with prefix enttok- have named entities replaced by ENT_TOK. Only entities, which participated on given path creation were replaced.

These files can be generated using scipt propsel-dataset-refresh.sh contained in https://github.com/brmson/dataset-factoid-webquestions.

Model Comparison
----------------

For randomized models, 95% confidence intervals (t-distribution) are reported.

Original (w/o enttok- substitutions):

| Model                    | trainAllMRR | devMRR   | testMAP  | testMRR  | settings
|--------------------------|-------------|----------|----------|----------|---------
| termfreq BM25 #w         | 0.184683    | 0.177923 | 0.216100 | 0.193827 | (defaults)
| attn1511                 | 0.894100    | 0.732434 | 0.701140 | 0.728824 | ``inp_e_dropout=0`` ``dropout=0``
|                          |±0.024455    |±0.010177 |±0.007767 |±0.004660 |
| cnn                      | 0.834471    | 0.689337 | 0.664300 | 0.694464 | ``inp_e_dropout=0`` ``dropout=0`` ``cnnsiamese=False``
|                          |±0.027208    |±0.017536 |±0.021354 |±0.018800 |
| rnn                      | 0.790574    | 0.676416 | 0.653450 | 0.682013 | ``inp_e_dropout=0`` ``dropout=0``
|                          |±0.121752    |±0.065309 |±0.067856 |±0.065095 |
| cnn                      | 0.656457    | 0.597415 | 0.562771 | 0.599110 | ``inp_e_dropout=0`` ``dropout=0``
|                          |±0.057810    |±0.030885 |±0.034142 |±0.032238 |
| rnncnn                   | 0.584183    | 0.546255 | 0.517417 | 0.555994 | ``inp_e_dropout=0`` ``dropout=0``
|                          |±0.077842    |±0.048224 |±0.051735 |±0.048143 |
| avg                      | 0.497539    | 0.495806 | 0.456750 | 0.503249 | ``inp_e_dropout=0`` ``dropout=0`` ``inp_w_dropout=1/3`` ``deep=2`` ``pact='relu'``
|                          |±0.008361    |±0.006968 |±0.006505 |±0.007881 |
| avg                      | 0.508851    | 0.493943 | 0.462075 | 0.506396 | ``inp_e_dropout=0`` ``dropout=0``
|                          |±0.021817    |±0.018339 |±0.013255 |±0.015074 |

