Global Relation Embedding for Relation Extraction (GloRE)

GloRE is a relation embedding model that can be used to augment existing relation extraction models and improve their performance. Most remarkably, for the top 1,000 relational facts discovered by the best existing model (PCNN+ATT), the precision can be improved from 83.9% to 89.3%.

Prerequisite

• Python 2.7
• Tensorflow 0.11

Install Tensorflow 0.11

(GPU support)

export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.11.0-cp27-none-linux_x86_64.whl
pip install --ignore-installed --upgrade $TF_BINARY_URL

(CPU only)

export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.11.0-cp27-none-linux_x86_64.whl
pip install --ignore-installed --upgrade $TF_BINARY_URL

Results

The result files from held-out and manual evaluations are included in results. To reproduce the figures and tables in the paper, simply follow the IPython notebook:

plot.ipynb

We've also provided a pretrained model in runs/pretrained_model. You can use it to re-generate the Precision-Recall files with the following command:

python steps.py --steps 2,4 --model_dir runs/pretrained_model/

Data

We use the NYT dataset as an example to show how to use our model to improve existing relation extraction models. The original NYT dataset can be downloaded from http://iesl.cs.umass.edu/riedel/ecml/ or https://github.com/thunlp/NRE.

We provide the following pre-processed files in data:

• data.train.gz / data.valid.gz : training and validation files that are in the format of

  textual relation [tab] KB relation [tab] weight
  

  As an example,

  <-nmod:for>##center##<-dobj>##created##<nsubj>##seminary##<compound>    /people/person/religion 0.666666666667
  

• kb_relation2id.txt : the set of target KB relations and their ids.
• left.20000.vocab / right.-1.vocab : vocabulary files. For the encoder input, we keep the most frequent 20,000 tokens in the vocabulary.
• left.20000.word2vec.vocab.npy : 300-dimensional word2vec vectors pre-trained on the Google News corpus.
• train_textual_relation.gz / test_textual_relation.gz : textual relations extracted from the training / testing corpus.
• train_pcnn_att_scores.gz / test_pcnn_att_scores.gz : relation extraction scores from PCNN+ATT (previously best performing model). The format is as follows:

  #
  entity1 [tab] entity2
  textual relaiton id (line number)
  binary encoding for the KB relation labels
  scores of each KB relation generated from the existing relation extraction model
  

  As an example,

  #
  m.010016        m.0492jkz
  542694
  1      0       0       0       0       0       0       0       0       0       0       0       0       0       0
         0       0       0       0       0       0       0       0       0       0       0       0       0       0
         0       0       0       0       0       0       0       0       0       0       0       0       0       0
         0       0       0       0       0       0       0       0       0       0
  0.999106        0.000001        0.000000        0.000000        0.000000        0.000000        0.000054        0.000000                        0.000001        0.000002        0.000000        0.000000        0.000001        0.000003        0.000000        0.000000                  0.000000        0.000007        0.000002        0.000000        0.000000        0.000000
         0.000000        0.000004        0.000001        0.000000        0.000000        0.000000        0.000000
         0.000003        0.000000        0.000000        0.000000        0.000003        0.000000        0.000001
         0.000001        0.000003        0.000000        0.000000        0.000000        0.000000        0.000000
         0.000001        0.000000        0.000001        0.000000        0.000001        0.000443        0.000002
         0.000351        0.000000        0.000000
  #
  

Step-by-Step

To run the training and testing all together, it can be as simple as

python steps.py

By default, this command will run through the following 4 steps:

1. Train a GloRE model

   You can run this step alone with

   python steps.py --steps 1
   

   OR

   python scripts/train_rel2vec.py your_model_directory

2. Extract GloRE scores

   You can run this step alone with

   python steps.py --steps 2 --model_dir your_model_directory
   

   OR

   python scripts/test_rel2vec.py your_model_directory

3. Train a merging model to combine GloRE scores with external scores

   You can run this step alone with

   python steps.py --steps 3 --model_dir your_model_directory
   

   OR

   python scripts/train_mlp.py your_model_directory

4. Generate final scores and Precision-Recall files

   You can run this step alone with

   python steps.py --steps 4 --model_dir your_model_directory
   

   OR

   python scripts/test_mlp.py your_model_directory

   The Precision-Recall files will be located in your_model_directory/output

We also provide some scripts for interactively exploring the trained GloRE model. Given a textual relation, you can check GloRE's predictions of KB relations with

python scripts/interactive_rel2vec.py your_model_directory

Configuration

All the hyperparameters can be changed by modifying files in scripts.

Cite

If you use the code in your research, please cite the following paper:

Global Relation Embedding for Relation Extraction
Yu Su, Honglei Liu, Semih Yavuz, Izzeddin Gur, Huan Sun, and Xifeng Yan
arXiv preprint arXiv:1704.05958 (2017)