Add remove emoji

[rzsgrt]

Hi, do you think its great to have remove emoticon in preprocessing? Its on top of this one. Would help on it.

[jbesomi]

Hey @rezasugiarto, thank you for opening an issue! 🎉

It sounds like a good idea. Can you please provide us with an example with a sample code? We will need to make sure the solution is fast enough and correct.

If you haven't done yet, please read CONTRIBUTING.md.

Also, you might find this python toolkit useful: python-ftfy.

Regards,

[polvoazul]

I have code that does this! It is not very fast, but it works. I also have code that tokenizes emails and telephone numbers. Would you be intested in a PR to include these?

[rzsgrt]

Hi, sorry I haven't done it yet. maybe we can collaborate on it @polvoazul

[jbesomi]

Hey @polvoazul, thank you for your comment.

Can you give us further information regarding your code? Yes, they might be useful.

Regarding tokenizing emails and telephone numbers, for now we are simply using a regular expression but we were considering switch to spacy #131. What's and how does it works your solution?

[rzsgrt]

Hi guys

This is snippet from what i do in my other project

import emoji

def remove_emoji(text:str)->str:
    text = emoji.get_emoji_regexp().sub(u"",text)
    return text

What do you guys think?

[henrifroese]

I believe the import / new dependency of the emoji package might be unnecessary. In your sample code, you use the RegEx from emoji.get_emoji_regexp(). Why not just locally look at and copy the RegEx once, and put it in the code, so we don't need to import it in the library? That should work

[rzsgrt]

Ohh, i think i missed it. It would be perfect i guess