Skip to content

Latest commit

 

History

History
75 lines (53 loc) · 3.04 KB

readme.md

File metadata and controls

75 lines (53 loc) · 3.04 KB
Raw

Social Bots Detection via Fusing BERT and Graph Convolutional Networks

├── run.py         # train model
├── model
├── utils
├── preprocess.py  # load dataset and generate training data
├── result         # store the result file
├── data           # store the training data
└── Twibot-22      # store the training data
    ├── data           # store the training data
    ├── model
    └── run.py      # train model
  • implement details:
  1. The original model use GatConv module from dgl, using ELU as the activate function of gat.
  2. Due to memory limitation, we set the batch size as 32.
  3. Due to memory limitation, the graph of Twibot-20 and midterm-2018 only consist of 3000 word(Top 3000 in order of word frequncy)
  4. In the implementation of Twibot-22, we use pytorch geometric neighbor loader to sample graph data.

How to reproduce:

  1. preprocess the the dataset by running

    python preprocess.py --source_path ${dataset}

    this command will create related features in corresponding directory.

  2. train the model by running:

    python run.py --dataset ${dataset}

    the final result will be saved into ${dataset}.txt

Result:

dataset acc precison recall f1
Twibot-22 mean 0.7188 0.2255 0.1990 0.2114
Twibot-22 std 0.0182 0.3088 0.2724 0.2895
Twibot-20 mean 0.6636 0.6764 0.7319 0.7005
Twibot-20 std 0.0100 0.0226 0.0749 0.0260
botometer-feedback-2019 mean 0.5962 0.2750 0.0857 0.1303
botometer-feedback-2019 std 0.0316 0.2820 0.0852 0.1301
cresci-rtbust-2019 mean 0.5000 0.5813 0.3514 0.4108
cresci-rtbust-2019 std 0.0488 0.1112 0.2058 0.1300
cresci-stock-2018 mean 0.5074 0.5278 0.7040 0.5818
cresci-stock-2018 std 0.0134 0.0075 0.2614 0.1205
midterm-2018 mean 0.8287 0.8440 0.9766 0.9050
midterm-2018 std 0.0148 0.0093 0.0366 0.0109
cresci-2017 mean 0.7585 0.7585 1.0000 0.8627
cresci-2017 std 0.0000 0.0000 0.0000 0.0000
gilani-2017 mean 0.4847 0.2543 0.6000 0.3572
gilani-2017 std 0.0834 0.2321 0.5477 0.3261
cresci-2015 mean 0.8778 0.8652 0.9556 0.9080
cresci-2015 std 0.0063 0.0064 0.0202 0.0060
baseline acc on Twibot-22 f1 on Twibot-22 type tags
Guo et al 0.7188 0.2114 F BERT GAT