Code

Pretokenized span 개선#170

Code

Merged

bab2min merged 5 commits intomainbab2min/Kiwi:mainfrom fix_pretokenizedbab2min/Kiwi:fix_pretokenizedCopy head branch name to clipboardJun 13, 2024

+55-21

Commits

Commits on Jun 13, 2024

• Refactor Types.h and Joiner.cpp for improved tag comparison

  bab2min

  committed
  93f180a

  Copy full SHA for 93f180a

• Fix null pointer exception in Kiwi.cpp

  bab2min

  committed
  b593c23

  Copy full SHA for b593c23

• Add auto inference for regularity to pretokenized tokens

  bab2min

  committed
  02195e1

  Copy full SHA for 02195e1

• Add support for pretokenized spans in KiwiCpp tests

  bab2min

  committed
  5895432

  Copy full SHA for 5895432

• Fix compilation errors at gcc 4.8.x

  bab2min

  committed
  69f5a21

  Copy full SHA for 69f5a21