We include a notebook that is able to reproduce the results recorded in Table 8 for the Hate Speech and Offensive Language Detection task. The notebook first reproduces the results for each method and then allows one to quickly obtain the results in Table 8 a single expert at a time.
- hatespeech_defer.ipynb reproduces the results in Table 8 for a single expert p,q.