Extractive or document question answering aims at finding a span (document_index, start, end) representing the answer
to a given question. It assumes that such documents are given apriori. We call supporting documents simply the "support"
in our framework.
We list trained models with scores, relative speeds and number of parameters.
| Model | F1 | Exact | Speed | Params in M | Download |
|---|---|---|---|---|---|
| FastQA | 77.4 | 67.4 | 1.00 | 0.95 | fastqa |
| BiDAF | 77.8 | 68.5 | 0.45 | 2.02 | bidaf |
| JackQA | 79.6 | 69.9 | 0.92 | 1.18 | jackqa |
| Model | F1 | Exact | Speed | Params in M | Download |
|---|---|---|---|---|---|
| FastQA | 63.7 | 58.7 | 1.00 | 0.95 | fastqa |
| JackQA | 65.4 | 60.3 | 1.11 | 1.18 | jackqa |
| Model | F1 | Exact | Speed | Params in M | Download |
|---|---|---|---|---|---|
| FastQA | 64.8 | 59.6 | 1.00 | 0.95 | fastqa |
| BiDAF | 58.3 | 53.0 | 0.45 | 2.02 | bidaf |
| JackQA | 66.1 | 60.6 | 0.87 | 1.18 | jackqa |
Please make sure that you roughly understand Jack before going on with this document.
New models can be stuck together using only the configuration file. All our model implementations except FastQA are
defined using our ModularQAModel. Possible modules are explained in more detail here.
Example configs can be found in conf/qa. However, if you think your fancy model cannot be defined like this please
keep on reading.
In preparation it is advised to at least go through the Implementing_a_new_model.ipynb to understand the basics behind Jacks modular design and how to use it to implement new models.
Jack contains implementations for extractive question answering in the extractive_qa
package. They include reusable modules defined in extractive_qa/shared.py,
e.g., XQAInputModule, XQAOutputModule and an AbstractXQAModelModule. This makes it very easy to implement new
models, because all you need to implement is a single function, see for instance FastQA.
After implementing your model you merely need to register it in
implementations.py. After that you are ready to train your model using the function
name (new_xqa_reader) you specified in the implementations.py file as value for the reader flag in you
configuration. You can of course reuse existing configs (e.g., fastqa.yaml) and overwrite the reader flag on the
command line, e.g., with the following command:
$ python3 bin/jack-train.py with config='./conf/fastqa.yaml' reader=new_xqa_readerOur existing input module serves a bunch of potentially useful TensorPorts, but of course not all of them need to be
used. Take a look at its defined output_ports and the existing model implementations for reference.
It also supports multi-paragraph/-document QA (we call supporting paragraphs/documents simply "support").
That means, that there might be multiple supports for a single question. The alignment between question and corresponding
supports is represented by the support2question tensor. If there are too many supporting documents for a single
question, that is, the max_num_support configuration is set to something lower, then the input module will sub-sample
paragraphs using TF-IDF similarity between question and support and only retain the max_num_support highest ranked.
If the XQAInputModule doesn't provide a crucial "tensor" for your new model, you can of course simply implement your own InputModule.
By the time of writing, Jack supports the following implementations: FastQA, BiDAF and DCN+,
as well ass our own creation JackQA, which aims at being resource friendly while retaining good performance.
Ready-to-use configurations for training such models can be found in the conf/qa directory.
There are download and conversion scripts for SQuAD, TriviaQA and NewsQA in data/. NewsQA can be converted to either
jack or squad format after download using the converters in jack/io. Setting up TriviaQA needs a couple of hours
but is fully automatized by running the download script resulting in jack formatted datasets. See the
data/triviaqa/README for more details. Supported loaders include 'jack' and 'squad', so no need to convert your
dataset if it comes in squad format.