How to use custom tokenizer in spark NLP prediction pipeline?

[sakib-NSL]

Hello everyone,
I am a beginner in Spark NLP. I have trained a Japanese Dataset on spark NLP using Bert embeddings. I have used spacy tokenizer and converted it to BIO format and then used the data in training. The result is satisfactory with the test data. But when I use the same test data on prediction pipeline, the performance decreases. I have used Tokenizer() and WordSegmenterModel() (alternatively) in prediction pipeline but did not work. Can I use a customized different tokeizer in pipeline?

Here is the training pipeline

Here is prediction pipeline

Questions:

1. Can I use a custom tokenizer in prediction pipeline?
2. Is there other reason for the test data's difference in performance in two pipelines?
3. This one might seem a little irrelevent here, I apologise for that. For bert Embedding, I am using .setMaxSentenceLength(1024) and it still works. Doesn't Bert has the limitation of using maximum 512 tokens? Are you splitting the text for long sequence or am I missing a point here ?

Thank you in advance.

[maziyarpanahi]

Hi @sakib-NSL

• Could you please provide a sample notebook for me to try and reproduce this?
• It is not possible to set setMaxSentenceLength to anything larger than 512, we immediately throw an exception saying BERT model has a max of 512
• I would suspect the tokenization being very different. (if possible I would compare the tokens coming from WordSegmenterModel which is the only tokenizer that supports Japanese and see many tokens where actually missed)

Having a simple sample code I can look deeper into this and see if we need to improve our WordSegmenterModel() and its models