Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chatbot output russian chars issue #63

Open
insatile opened this issue Apr 13, 2020 · 1 comment
Open

Chatbot output russian chars issue #63

insatile opened this issue Apr 13, 2020 · 1 comment

Comments

@insatile
Copy link

Can you tell me please, what am I doing wrong?
Was training model on raw russian text file( > 4mb), formatted as in scotus file:

marfo4ka43: Ты кто вообще?)
kyindarkkk: чел с соседнего офиса с дредами)
marfo4ka43: ахаха
kyindarkkk: xDD
marfo4ka43: мы на обеде до столовки ходим, так себе прогулка
But at the end, when running chatbot.py it only returns spaces, numbers and english characters:
привет алешка
kyindarkkk: 5
погода сегодня так себе
kyindarkkk:
ты думаешь?
kyindarkkk:
кто то просто нас не понял
kyindarkkk: ? 3Fllus s GO 1738
The question is, how to make it train/output russian characters too

@uninstallgentoo
Copy link

By default open() use locale.getpreferredencoding(False). So you need to set encoding explicitly when you open the file in utils:106
io.open(input_file, mode='rt', encoding='utf-8')

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants