performance comparison

[YanLiang1102]

test performance using topic as multi-task learning. only enhance the encoder, not using the topic as input for event extraction though

epoch 1
('lose Mention', u'cfcec8e30722564d5bf43fcb5f739cd8')
('lose Mention', u'04cdc0c45303f7024417e4d94f9a7c13')
('lose Mention', u'66b0e5014943dde6c74c048086b7a0a3')
('Micro_F1:', 67.776451242312973)
('Micro_Precision:', 66.690309424594375)
('Micro_Recall:', 68.898557362033429)
('Macro_F1:', 50.955298281699037)
('Macro_Precision:', 55.190862598324387)
('Macro_Recall:', 50.812833395810941)

epoch 2
('lose Mention', u'395e263d21484d8998d853b0b1b6ec5a')
('lose Mention', u'063e9a5b7265bc3ae0ea731c5f06545b')
('lose Mention', u'362cbeafe8c757195425faa40a88ff61')
('Micro_F1:', 67.062822261786408)
('Micro_Precision:', 67.72205099467638)
('Micro_Recall:', 66.416304098923746)
('Macro_F1:', 58.714489780943694)
('Macro_Precision:', 63.478851556871454)
('Macro_Recall:', 58.704903536659984)

epoch 3
('lose Mention', u'f0ae798aa5b9014e3c8e47aefb281330')
('lose Mention', u'd3f3a2e2f335d8c50f7612c29aa0cb2a')
('lose Mention', u'1418f6cb3c97797a7542e7ec6ac04427')
('lose Mention', u'913c2d91f695ccb637386f211cd8d94d')
('lose Mention', u'f697f918553fc11d73d709bf3029603d')
('Micro_F1:', 68.213119095965396)
('Micro_Precision:', 65.833084820858005)
('Micro_Recall:', 70.771696817036869)
('Macro_F1:', 60.237205257354191)
('Macro_Precision:', 61.145512218719368)
('Macro_Recall:', 62.252039674965374)

epoch 10
('Micro_F1:', 64.949966644429608)
('Micro_Precision:', 63.125135076723581)
('Micro_Recall:', 66.883444011907486)
('Macro_F1:', 59.76302032588179)
('Macro_Precision:', 58.998255594751669)
('Macro_Recall:', 61.678292061307531)

[YanLiang1102]

test performance bert-crf:

epoch 1
('Micro_F1:', 67.462058468012714)
('Micro_Precision:', 65.979246026533559)
('Micro_Recall:', 69.013052438745135)
('Macro_F1:', 54.734193211038985)
('Macro_Precision:', 58.043941483652027)
('Macro_Recall:', 55.793730660778905)

epoch 3
('lose Mention', u'df0a542b1da02933d7ec99db1d242f88')
('lose Mention', u'35cc76d3b1b331cf496dc13451da70db')
('lose Mention', u'aca274ee083d94192aa9d58832f1a258')
('lose Mention', u'49d6dc789d08db61efd0fbc46a55570a')
('lose Mention', u'b30bf4393b0cc55add7ae3e7380aaac3')
('lose Mention', u'24345d37f8e66bd7fcb3362bdc861942')
('Micro_F1:', 68.219692783733009)
('Micro_Precision:', 65.924818453652293)
('Micro_Recall:', 70.68010075566751)
('Macro_F1:', 60.441497708246828)
('Macro_Precision:', 61.680135350505118)
('Macro_Recall:', 61.821271767813826)

[YanLiang1102]

The above two performance shows that only using topic_id as multi-task learning won't help the performance. next thing to try make the event extraction conditioned on topic. and also at the same time using multi-task learning.

[YanLiang1102]

conditional on the topic, by adding the topic embedding directly to the sentence embedding.
epoch3
('Micro_F1:', 67.491943549283263)
('Micro_Precision:', 65.561312607944728)
('Micro_Recall:', 69.539729791618967)
('Macro_F1:', 60.342763541804359)
('Macro_Precision:', 60.752334592420986)
('Macro_Recall:', 62.328372626437236)

[YanLiang1102]

The topics information does not help might because of this: most of the topics are tail topics.

token:military conflict, test_count:258, train_count:981
token:rail accident, test_count:18, train_count:36
token:limited overs final, test_count:2, train_count:6
token:concert tour, test_count:49, train_count:167
token:event, test_count:16, train_count:36
token:news event, test_count:21, train_count:61
token:flood, test_count:6, train_count:30
token:aircraft occurrence, test_count:24, train_count:76
token:military operation, test_count:1, train_count:1
token:nuclear weapons test, test_count:4, train_count:12
token:civil conflict, test_count:31, train_count:106
token:civilian attack, test_count:49, train_count:196
token:concert, test_count:6, train_count:13
token:historical event, test_count:12, train_count:59
token:terrorist attack, test_count:18, train_count:50
token:olympic event, test_count:4, train_count:10
token:operational plan, test_count:3, train_count:7
token:hurricane, test_count:97, train_count:314
token:recurring event, test_count:34, train_count:94
token:football match, test_count:8, train_count:67
token:music festival, test_count:41, train_count:101
token:wildfire, test_count:12, train_count:10
token:wrestling event, test_count:23, train_count:55
token:airliner accident, test_count:10, train_count:33
token:international football competition, test_count:9, train_count:43
token:cricket tournament, test_count:11, train_count:56
token:aircraft accident, test_count:7, train_count:27
token:athleticrace, test_count:8, train_count:12
token:cycling championship, test_count:1, train_count:1
token:rugby match, test_count:2, train_count:2
token:games, test_count:6, train_count:23
token:earthquake, test_count:17, train_count:31
token:legislative session, test_count:2, train_count:2
token:winter storm, test_count:7, train_count:19
token:canadian football game, test_count:2, train_count:0
token:horse race, test_count:2, train_count:16
token:badminton event, test_count:1, train_count:0
token:military attack, test_count:1, train_count:3
token:summit, test_count:1, train_count:3
token:international ice hockey competition, test_count:7, train_count:16
token:summit meeting, test_count:1, train_count:10
token:mma event, test_count:3, train_count:7
token:athletics competition, test_count:1, train_count:4
token:international handball competition, test_count:1, train_count:4
token:cycling championships, test_count:1, train_count:0
token:individual golf tournament, test_count:3, train_count:16
token:pro bowl, test_count:1, train_count:7
token:u.s. federal election campaign, test_count:1, train_count:2
token:commonwealth games event, test_count:1, train_count:0
token:swimming event, test_count:1, train_count:1
token:athletics race, test_count:1, train_count:6
token:university boat race, test_count:4, train_count:2
token:hurling championship, test_count:1, train_count:1
token:field hockey, test_count:1, train_count:4
token:australian rules football grand final, test_count:2, train_count:2
token:international baseball tournament, test_count:1, train_count:1
token:tennis event, test_count:1, train_count:4
token:rugby tournament, test_count:1, train_count:8

[YanLiang1102]

Add in the topic2event type distribution as prior. (similar to adding vocab for each topic into the extraction work, will this work?? not sure..)

[YanLiang1102]

The topic will help the extraction by assuming two things: 1. given a topic, certain event type happens more often than others.
2. similar topic embedding can infer similar event types in the sentences. First of all, are they true???