Sentencepiece re-release
Pre-release
Pre-release
Releases a new version of Sentencepiece with major refactorings:
- Builds with Bazel
- Re-uses existing open source libraries whenever possible
- Refactors internal dependencies
- New sets of features for configuring tokenizers
- Separation from Tensorflow