Skip to content

Latest commit

 

History

History
135 lines (73 loc) · 3.14 KB

README.md

File metadata and controls

135 lines (73 loc) · 3.14 KB
Raw

Somiao Pinyin: Train your own Chinese Input Method with Seq2seq Model

中文Blog

Personalized Chinese Pinyin Input Method with Seq2seq model

Original code in https://github.com/Kyubyong/neural_chinese_transliterator for research purpose.

This repository intends to experiment with different training data and interactive user inputs, and possibly develop towards a real data-personalized and model-localized Pinyin Input product.

Requrements

  • Python (>=3.5)

  • TensorFlow (>=r1.2)

  • xpinyin (for Chinese pinyin annotation)

  • distance (for calculating the similarity score between two strings)

  • tqdm

Usage

Training:

  • STEP 1. Download Leipzig Chinese Corpus

    Extract it and copy zho_news_2007-2009_1M-sentences.txt to data/ folder.

    Or use your own Chinese Corpus with the same format.

  • STEP 2. Build a Pinyin-Chinese parallel corpus.

#python3 build_corpus.py
  • STEP 3. Run prepro.py to make vocabulary and training data.
#python3 prepro.py

  • STEP 4. Adjust hyperparameters in hyperparams.py if necessary.

  • STEP 5. Train the model

#python3 train.py

Inference with command line input:

For command line input testing, run:

python3 eval.py

You may change the main function name to use the original testing data evaluation.

Testing with pre-trained models:

Download the pre-trained model from blog, unzip it to generate /log and /data.

Remember to overwrite the pickle files in /data with the pre-trained model data.

Then run for command line input testing:

python3 eval.py

Sample Results

Model is trained from Chinese News in 2007-2009. So many now common Chinese sayings are not learned.

请输入测试拼音：nihao
你好

请输入测试拼音：chenggongle
成功了

请输入测试拼音：wolegequ
我了个曲

请输入测试拼音：taibangla
太棒啦

请输入测试拼音：dacolehuizenmeyang
打破了会怎么样

请输入测试拼音：pujinghehujintaotongdianhua
普京和胡锦涛通电话

请输入测试拼音：xiangbuqilaishinianqianfashengleshenme
想不起来十年前发生了什么

请输入测试拼音：meiguohongzhawomenzainansilafudedashiguan
美国轰炸我们在南斯拉夫的大事馆

请输入测试拼音：liudehuanageshihouhaonianqing
刘德华那个时候好年轻

请输入测试拼音：shishihouxunlianyixiabilibilideyuliaole
是时候训练一下比例比例的预料了

TODOLIST

  • Pretrained models on different contexts

  • Model selection for using different models while input different things (chatting? writing scientific papers? etc...)

  • Function to record LOCALLY what user has input as personalized corpus

  • User Interface

  • ...