Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Tokenizers #687

Closed
wants to merge 142 commits into from
Closed

Add Tokenizers #687

wants to merge 142 commits into from

Commits on Apr 25, 2023

  1. Added string tensor implementation with explicit pointer unpack

    @slyalin
    slyalin committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    70f867a View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2023

  1. Merged from master

    @slyalin
    slyalin committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    1fac3de View commit details
    Browse the repository at this point in the history

Commits on May 2, 2023

  1. Started to migrate to extension-only support of string operations wit… 

    …h and without string support in OV core. Moved StringTensorUnpack and reworked it to be aligned with the new approach. Reworked sentece piece op and translation code to be compatible with several variants of string tensor representation and the plugin wrapping hack.
    @slyalin
    slyalin committed May 2, 2023
    Configuration menu
    Copy the full SHA
    821dee5 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2023

  1. Started to merge string/tokenizer related stuff from a dedicated OV b… 

    …ranch to contrib in form compatible with both master and the branch with string tensors support. Added CaseFoldUTF8 from that branch.
    @slyalin
    slyalin committed May 10, 2023
    Configuration menu
    Copy the full SHA
    b9b0693 View commit details
    Browse the repository at this point in the history

  2. Rename CaseFoldUTF8 to name from opset proposal: CaseFold, added Norm… 

    …alizeUnicode
    @slyalin
    slyalin committed May 10, 2023
    Configuration menu
    Copy the full SHA
    c785ec1 View commit details
    Browse the repository at this point in the history

Commits on May 11, 2023

  1. Added a stub for RegexNormalization operation, WA for CPU bug with em… 

    …pty constants, register StringTensorPack and StringTensorUnpack as OV operations to be able to read IRs with those operations
    @slyalin
    slyalin committed May 11, 2023
    Configuration menu
    Copy the full SHA
    1d129ac View commit details
    Browse the repository at this point in the history

  2. Implemented Reshape for decomposed string tensors

    @slyalin
    slyalin committed May 11, 2023
    Configuration menu
    Copy the full SHA
    71bc5bf View commit details
    Browse the repository at this point in the history

Commits on May 12, 2023

  1. Added RaggedTensorPack, sophisticated stup for RegexSplit and overrid… 

    …den Const translator for TF to intercept string constants
    @slyalin
    slyalin committed May 12, 2023
    Configuration menu
    Copy the full SHA
    6c5eec0 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2023

  1. Fixes for both master and element::string branches of OpenVINO; bette… 

    …r conditional compilation based on available features in OpenVINO
    @slyalin
    slyalin committed May 15, 2023
    Configuration menu
    Copy the full SHA
    29dfe38 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2023

  1. Debug output of indices in RaggedTensorPack

    @slyalin
    slyalin committed May 16, 2023
    Configuration menu
    Copy the full SHA
    40063c1 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2023

  1. Implemented a stub for WordpieceTokenizer. Supported conversion of a … 

    …combination of WordpieceTokenizeWithOffsets and LookupTableFindV2 from TensorFlow
    @slyalin
    slyalin committed May 17, 2023
    Configuration menu
    Copy the full SHA
    cc47b12 View commit details
    Browse the repository at this point in the history

  2. Disabled debug output

    @slyalin
    slyalin committed May 17, 2023
    Configuration menu
    Copy the full SHA
    7644231 View commit details
    Browse the repository at this point in the history

Commits on May 18, 2023

  1. Define default values for custom operations attributes to make attrib… 

    …ute initialization optional (needed for core.make_node)
    @slyalin
    slyalin committed May 18, 2023
    Configuration menu
    Copy the full SHA
    80b8023 View commit details
    Browse the repository at this point in the history

Commits on May 20, 2023

  1. Added fast_tokenizer lib to the build. Implemented CaseFold based on … 

    …fast_tokenizer.
    @slyalin
    slyalin committed May 20, 2023
    Configuration menu
    Copy the full SHA
    46c82b8 View commit details
    Browse the repository at this point in the history

  2. Removed debug output

    @slyalin
    slyalin committed May 20, 2023
    Configuration menu
    Copy the full SHA
    d7ca2ab View commit details
    Browse the repository at this point in the history

  3. Implemented RaggedToDense always in pad_right=true mode and with bool… 

    …ean mask extra output
    @slyalin
    slyalin committed May 20, 2023
    Configuration menu
    Copy the full SHA
    2baac3d View commit details
    Browse the repository at this point in the history

Commits on May 23, 2023

  1. Provided real implementations for NormalizeUnicode, RegexNormalizatio… 

    …n and RegexSplit based on paddle fast_tokenizer lib. Limited implementation, not all of the features of ops and TF translated ops are implemented.
    @slyalin
    slyalin committed May 23, 2023
    Configuration menu
    Copy the full SHA
    d270dd6 View commit details
    Browse the repository at this point in the history

  2. Implemented WordpieceTokenizer with fast_tokenizer library

    @slyalin
    slyalin committed May 23, 2023
    Configuration menu
    Copy the full SHA
    119d6e9 View commit details
    Browse the repository at this point in the history

Commits on May 25, 2023

  1. Renamed behaviours to be verbs instead of adjectives

    @slyalin
    slyalin committed May 25, 2023
    Configuration menu
    Copy the full SHA
    4d4ad89 View commit details
    Browse the repository at this point in the history

  2. Added modified version of HF tokenizer parser from Artur; implemented… 

    … necessary steps to complete HF bert preprocessing conversion (not validated)
    @slyalin
    slyalin committed May 25, 2023
    Configuration menu
    Copy the full SHA
    f4eee84 View commit details
    Browse the repository at this point in the history

  3. Renamed apply_tokenizer to connect_tokeniser and removed obsolete han… 

    …dling of model name
    @slyalin
    slyalin committed May 25, 2023
    Configuration menu
    Copy the full SHA
    1e50352 View commit details
    Browse the repository at this point in the history

Commits on May 31, 2023

  1. CombineSegments is implemented, used in HF converter. Stitching of to… 

    …kenizer and main model is fixed partially (still produces topologically incorrect model)
    @slyalin
    slyalin committed May 31, 2023
    Configuration menu
    Copy the full SHA
    0966b8a View commit details
    Browse the repository at this point in the history

  2. Fixed stitching of two models by connecting with names of inputs/outp… 

    …uts, now Bert and its tokenizer are connected together correctly
    @slyalin
    slyalin committed May 31, 2023
    Configuration menu
    Copy the full SHA
    61d7983 View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2023

  1. WA for CPU bug with scalar inputs, correct truncation and dynamic pad… 

    …ding, fix bugs for batches processing
    @slyalin
    slyalin committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    5609ee6 View commit details
    Browse the repository at this point in the history

  2. Fixed conversion of HF tokenizer if part of outputs are omitted. Disa… 

    …bled debug output
    @slyalin
    slyalin committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    062acf3 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2023

  1. Add BPE Tokenizer

    @apaniukov
    apaniukov committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    0f772dc View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2023

  1. Add BytesToChars Node for BBPE

    @apaniukov
    apaniukov committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    10e3d18 View commit details
    Browse the repository at this point in the history

  2. Delete print

    @apaniukov
    apaniukov committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    c413cb6 View commit details
    Browse the repository at this point in the history

  3. Clip max value for max_length to int32

    @apaniukov
    apaniukov committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    8c8994c View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2023

  1. Fix RegexNormalization and Splitter, Add Digits Splitter

    @apaniukov
    apaniukov committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    8750ae6 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2023

  1. Bug fixes

    @apaniukov
    apaniukov committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    be6dc3f View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2023

  1. Add decoding step, BytesToChars refactoring 

    Has a bug with internal dimension for VocabNode
    @apaniukov
    apaniukov committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    e4dcdda View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2023

  1. Fix some regex bugs for byte-level splitter

    @apaniukov
    apaniukov committed Jun 30, 2023
    Configuration menu
    Copy the full SHA
    b45e5ec View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. Fix bug with VocabDecoder shape

    @apaniukov
    apaniukov committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    5f03ed0 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2023

  1. Minor changes for natively supported strings

    @slyalin
    slyalin committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    2a65502 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'artur/string_tensors_add_bpe' into stri… 

    …ng_tensors
    @slyalin
    slyalin committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    2e34b92 View commit details
    Browse the repository at this point in the history

  3. Suppressed minor^Carnings about int32 -> unsigned implicit

    @slyalin
    slyalin committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    a6f9110 View commit details
    Browse the repository at this point in the history

  4. Restructured sentence_piece directory to tokenizer directory: split a… 

    …ll ops, translators and helper into individual files. To build use tokenizer custom op name in cmake instead of sentence_piece.
    @slyalin
    slyalin committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    5c29254 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2023

  1. Add regex to detokenizer pipeline, all splitters have 5 inputs

    @apaniukov
    apaniukov committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    f8d0e0d View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2023

  1. Add Caching for RegexNormalization

    @apaniukov
    apaniukov committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    10c10c5 View commit details
    Browse the repository at this point in the history

  2. Add Caching for RegexSplit

    @apaniukov
    apaniukov committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    4eb12f8 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2023

  1. Add Wordpiece Cache

    @apaniukov
    apaniukov committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    c5efaf0 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2023

  1. Add NodeFactory

    @apaniukov
    apaniukov committed Jul 31, 2023
    Configuration menu
    Copy the full SHA
    239acc4 View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2023

  1. Fix regex nodes init

    @apaniukov
    apaniukov committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    38552b0 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2023

  1. Fix Wordpiece Cache

    @apaniukov
    apaniukov committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    597ccd4 View commit details
    Browse the repository at this point in the history

  2. Add BPE Cache

    @apaniukov
    apaniukov committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    e6933b7 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2023

  1. Fix RegexNormalization

    @apaniukov
    apaniukov committed Aug 11, 2023
    Configuration menu
    Copy the full SHA
    bd7f9d9 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2023

  1. Refactor CombineSegments and Padding

    @apaniukov
    apaniukov committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    99c603f View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. Refactoring

    @apaniukov
    apaniukov committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    6cc9b36 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2023

  1. Clean-up commented code

    @apaniukov
    apaniukov committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    973c52d View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2023

  1. Sentencepiece Model Encoder from Transformers Tokenizer

    @apaniukov
    apaniukov committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    1fa02b2 View commit details
    Browse the repository at this point in the history

  2. Add tests for tokenizers

    @apaniukov
    apaniukov committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    e37f89d View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2023

  1. Add detokenizer for Sentencepiece models

    @apaniukov
    apaniukov committed Oct 2, 2023
    Configuration menu
    Copy the full SHA
    88bf7c6 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2023

  1. Update README.md

    @apaniukov
    apaniukov authored Oct 4, 2023
    Configuration menu
    Copy the full SHA
    bb1b57a View commit details
    Browse the repository at this point in the history

  2. Update README.md

    @apaniukov
    apaniukov authored Oct 4, 2023
    Configuration menu
    Copy the full SHA
    6b4be05 View commit details
    Browse the repository at this point in the history

  3. Update README.md

    @apaniukov
    apaniukov authored Oct 4, 2023
    Configuration menu
    Copy the full SHA
    539797f View commit details
    Browse the repository at this point in the history

  4. OVTokenizer as python package

    @apaniukov
    apaniukov committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    79c3e09 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'openvinotoolkit:master' into tokenizer-fix-decode

    @apaniukov
    apaniukov authored Oct 4, 2023
    Configuration menu
    Copy the full SHA
    203ffbb View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2023

  1. Update README.md

    @apaniukov
    apaniukov committed Oct 5, 2023
    Configuration menu
    Copy the full SHA
    45c0068 View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2023

  1. Merge branch 'master' into tokenizer-fix-decode

    @apaniukov
    apaniukov authored Oct 6, 2023
    Configuration menu
    Copy the full SHA
    372465b View commit details
    Browse the repository at this point in the history

  2. Add sentencepiece detokenizer test

    @apaniukov
    apaniukov committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    64567ea View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/tokenizer-fix-decode' into token… 

    …izer-fix-decode
    @apaniukov
    apaniukov committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    f54076e View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2023

  1. Unified interface for fast and sentencepiece tokenizers

    @apaniukov
    apaniukov committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    c42d1bd View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2023

  1. Add Full Pipeline example for Sentencepiece 

    Move greedy decoding pipeline from detokenizer to model
    @apaniukov
    apaniukov committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    8b29443 View commit details
    Browse the repository at this point in the history

  2. Update third-party-programs.txt

    @apaniukov
    apaniukov committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    2ee3707 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2023

  1. Merge branch 'master' into tokenizer-fix-decode

    @apaniukov
    apaniukov authored Oct 12, 2023
    Configuration menu
    Copy the full SHA
    4b57fcc View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2023

  1. Add Constants

    @apaniukov
    apaniukov committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    803d831 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2023

  1. Add CPP pack/unpack_strings functions 

    Refactor greedy decoding
    @apaniukov
    apaniukov committed Oct 16, 2023
    Configuration menu
    Copy the full SHA
    72f6d9f View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2023

  1. Merge branch 'master' into tokenizer-fix-decode

    @apaniukov
    apaniukov authored Oct 17, 2023
    Configuration menu
    Copy the full SHA
    386cb02 View commit details
    Browse the repository at this point in the history

  2. Move tests to tokenizer dir

    @apaniukov
    apaniukov committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    79bd05f View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2023

  1. Fix import

    @apaniukov
    apaniukov committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    24a60b3 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/tokenizer-fix-decode' into token… 

    …izer-fix-decode
    @apaniukov
    apaniukov committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    f01afee View commit details
    Browse the repository at this point in the history

  3. Fix imports

    @apaniukov
    apaniukov committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    b22569f View commit details
    Browse the repository at this point in the history

  4. Sort Imports

    @apaniukov
    apaniukov committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    96673f5 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2023

  1. Add Streaming Sentencepiece Decoder

    @apaniukov
    apaniukov committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    0e7ae87 View commit details
    Browse the repository at this point in the history

  2. Change Authors

    @apaniukov
    apaniukov committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    5ebdb1f View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2023

  1. Update modules/custom_operations/user_ie_extensions/tokenizer/utils.cpp 

    Co-authored-by: Zlobin Vladimir <[email protected]>
    @apaniukov @Wovchena
    apaniukov and Wovchena authored Oct 23, 2023
    Configuration menu
    Copy the full SHA
    6a55877 View commit details
    Browse the repository at this point in the history

  2. Configure tests

    @apaniukov
    apaniukov committed Oct 23, 2023
    Configuration menu
    Copy the full SHA
    06d5159 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2023

  1. Skip Java Tests

    @apaniukov
    apaniukov committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    fa5360d View commit details
    Browse the repository at this point in the history

  2. Add Regression Test

    @apaniukov
    apaniukov committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    e855193 View commit details
    Browse the repository at this point in the history

  3. Skip traceback

    @apaniukov
    apaniukov committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    d495d3b View commit details
    Browse the repository at this point in the history

  4. Add Win64 Fast Tokenizer lib

    @apaniukov
    apaniukov committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    d7bebd0 View commit details
    Browse the repository at this point in the history

  5. Fix WorkingDir

    @apaniukov
    apaniukov committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    b2e35ed View commit details
    Browse the repository at this point in the history

  6. Return TB

    @apaniukov
    apaniukov committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    f81bd18 View commit details
    Browse the repository at this point in the history

  7. Fix dependencies install

    @apaniukov
    apaniukov committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    0bd23b5 View commit details
    Browse the repository at this point in the history

  8. Add byte tokens handling for sentencepiece

    @apaniukov
    apaniukov committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    12ac9f8 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2023

  1. Drop black, use ruff format instead

    @apaniukov
    apaniukov committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    9e6ae6f View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2023

  1. Temp remove tokenizers from windows CI

    @apaniukov
    apaniukov committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    f5d2d4c View commit details
    Browse the repository at this point in the history

  2. CI check

    @apaniukov
    apaniukov committed Oct 26, 2023
    Configuration menu
    Copy the full SHA
    cf039b9 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2023

  1. Compile fast_tokenizers from source code

    @ilya-lavrenov
    ilya-lavrenov committed Oct 28, 2023
    Configuration menu
    Copy the full SHA
    795306d View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2023

  1. Export pack_strings() and unpack_strings()

    @Wovchena
    Wovchena committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    9c200c2 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2023

  1. Merge pull request #1 from ilya-lavrenov/tokenizer-fix-decode 

    Compile fast_tokenizers from source code
    @apaniukov
    apaniukov authored Oct 31, 2023
    Configuration menu
    Copy the full SHA
    0e9b960 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into tokenizer-fix-decode

    @apaniukov
    apaniukov authored Oct 31, 2023
    Configuration menu
    Copy the full SHA
    95aa47c View commit details
    Browse the repository at this point in the history

  3. Merge branch 'tokenizer-fix-decode' into export-pack_strings-and-unpa… 

    …ck_strings
    @Wovchena
    Wovchena authored Oct 31, 2023
    Configuration menu
    Copy the full SHA
    e1de338 View commit details
    Browse the repository at this point in the history

  4. Build tokenizer target on windows

    @apaniukov
    apaniukov committed Oct 31, 2023
    Configuration menu
    Copy the full SHA
    f23e59b View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2023

  1. Merge branch 'tokenizer-fix-decode' into export-pack_strings-and-unpa… 

    …ck_strings
    @Wovchena
    Wovchena committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    dbec117 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2023

  1. Add icu4c patch

    @apaniukov
    apaniukov committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    ce25397 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2023

  1. Added include dir to nlohmann headers

    @ilya-lavrenov
    ilya-lavrenov committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    d46f594 View commit details
    Browse the repository at this point in the history

  2. Fixed compilation on ubuntu 18.04 arm64

    @ilya-lavrenov
    ilya-lavrenov committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    6f213ab View commit details
    Browse the repository at this point in the history

  3. Fixed Windows

    @ilya-lavrenov
    ilya-lavrenov committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    6ed52e4 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #3 from ilya-lavrenov/nlohmann 

    Added include dir to nlohmann headers
    @apaniukov
    apaniukov authored Nov 8, 2023
    Configuration menu
    Copy the full SHA
    ca62321 View commit details
    Browse the repository at this point in the history

  5. Supported prebuild Fast Tokenizers on all platforms

    @ilya-lavrenov
    ilya-lavrenov committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    52bfe5a View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2023

  1. Merge branch 'master' into tokenizer-fix-decode

    @apaniukov
    apaniukov authored Nov 9, 2023
    Configuration menu
    Copy the full SHA
    b504013 View commit details
    Browse the repository at this point in the history

  2. Add tiktoken support WIP

    @apaniukov
    apaniukov committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    cc663dc View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/tokenizer-fix-decode' into token… 

    …izer-fix-decode
    @apaniukov
    apaniukov committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    4c9ceed View commit details
    Browse the repository at this point in the history

  4. Unskip java tests

    @apaniukov
    apaniukov committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    745e969 View commit details
    Browse the repository at this point in the history

  5. Merge pull request #4 from ilya-lavrenov/prebuilt-fast-tokenizers 

    Supported prebuild Fast Tokenizers on all platforms
    @apaniukov
    apaniukov authored Nov 9, 2023
    Configuration menu
    Copy the full SHA
    48564b7 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2023

  1. Fixed compilation with re2 on Windows

    @ilya-lavrenov
    ilya-lavrenov committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    056eb9f View commit details
    Browse the repository at this point in the history

  2. Merge pull request #5 from ilya-lavrenov/windows-re2 

    Fixed compilation with re2 on Windows
    @apaniukov
    apaniukov authored Nov 10, 2023
    Configuration menu
    Copy the full SHA
    309b8e9 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'tokenizer-fix-decode' into export-pack_strings-and-unpa… 

    …ck_strings
    @Wovchena
    Wovchena committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    b193cb2 View commit details
    Browse the repository at this point in the history

  4. Move unpack_strings(), create sepparate include dir

    @Wovchena
    Wovchena committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    debcb5d View commit details
    Browse the repository at this point in the history

  5. openvino_extensions

    @Wovchena
    Wovchena committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    b739ffd View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2023

  1. Fixed link stage on Windows

    @ilya-lavrenov
    ilya-lavrenov committed Nov 13, 2023
    Configuration menu
    Copy the full SHA
    e70a3f2 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #6 from ilya-lavrenov/windows-linkage 

    Fixed link stage on Windows
    @apaniukov
    apaniukov authored Nov 13, 2023
    Configuration menu
    Copy the full SHA
    2ce27cd View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2023

  1. i64 is default tokenizer output type

    @apaniukov
    apaniukov committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    3022a5a View commit details
    Browse the repository at this point in the history

  2. Add support for more tiktoken tokenizers

    @apaniukov
    apaniukov committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    c467a8c View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2023

  1. Merge branch 'tokenizer-fix-decode' into export-pack_strings-and-unpa… 

    …ck_strings
    @Wovchena
    Wovchena committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    1ec4c5f View commit details
    Browse the repository at this point in the history

  2. Check Azure CI

    @apaniukov
    apaniukov committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    8505b51 View commit details
    Browse the repository at this point in the history

  3. Fix Azure Win CI

    @apaniukov
    apaniukov committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    82639e6 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #2 from Wovchena/export-pack_strings-and-unpack_st… 

    …rings

Export pack_strings() and unpack_strings()
    @apaniukov
    apaniukov authored Nov 15, 2023
    Configuration menu
    Copy the full SHA
    fb37580 View commit details
    Browse the repository at this point in the history

  5. Define python version for setupvars.bat

    @apaniukov
    apaniukov committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    a45b826 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/tokenizer-fix-decode' into token… 

    …izer-fix-decode
    @apaniukov
    apaniukov committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    35cc136 View commit details
    Browse the repository at this point in the history

  7. Add support for tiktoken detokenizers

    @apaniukov
    apaniukov committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    244a593 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2023

  1. Merge branch 'master' into tokenizer-fix-decode

    @andrei-kochin
    andrei-kochin authored Nov 16, 2023
    Configuration menu
    Copy the full SHA
    84686f4 View commit details
    Browse the repository at this point in the history

  2. Add ChatGLM tokenization support.

    @apaniukov
    apaniukov committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    ad1c589 View commit details
    Browse the repository at this point in the history

  3. Add ChatGLM detokenization and tests

    @apaniukov
    apaniukov committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    0f63c3d View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2023

  1. Add ChatGLM detokenization and tests

    @apaniukov
    apaniukov committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    0f1c1cc View commit details
    Browse the repository at this point in the history

  2. Fix mac sha256

    @apaniukov
    apaniukov committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    3edb73b View commit details
    Browse the repository at this point in the history

  3. Skip Lin Java Tests

    @apaniukov
    apaniukov committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    48bba34 View commit details
    Browse the repository at this point in the history

  4. Add Mac Tokenziers Tests and Skip Mac Java Step

    @apaniukov
    apaniukov committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    fe507ff View commit details
    Browse the repository at this point in the history

  5. Fix Mac SHA

    @apaniukov
    apaniukov committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    4b0c4ec View commit details
    Browse the repository at this point in the history

  6. Del WA for CPU Bug

    @apaniukov
    apaniukov committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    4656238 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2023

  1. Fix Mac CI Pipeline

    @apaniukov
    apaniukov committed Nov 18, 2023
    Configuration menu
    Copy the full SHA
    2f5cc1c View commit details
    Browse the repository at this point in the history

  2. Change Mac CI

    @apaniukov
    apaniukov committed Nov 18, 2023
    Configuration menu
    Copy the full SHA
    1568727 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2023

  1. Fixed compilation

    @ilya-lavrenov
    ilya-lavrenov committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    fa822c2 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #7 from ilya-lavrenov/compilation-fix 

    Fixed compilation
    @apaniukov
    apaniukov authored Nov 20, 2023
    Configuration menu
    Copy the full SHA
    6ddb2a6 View commit details
    Browse the repository at this point in the history

  3. Add setupvars to mac CI

    @apaniukov
    apaniukov committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    14f993b View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into tokenizer-fix-decode

    @apaniukov
    apaniukov authored Nov 20, 2023
    Configuration menu
    Copy the full SHA
    cae3098 View commit details
    Browse the repository at this point in the history

  5. Change detokenizer output type

    @apaniukov
    apaniukov committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    b59204d View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/tokenizer-fix-decode' into token… 

    …izer-fix-decode
    @apaniukov
    apaniukov committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    e54b42e View commit details
    Browse the repository at this point in the history

  7. Fix SegFault on AddedTokens For BPE tokenizer

    @apaniukov
    apaniukov committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    6c3bae3 View commit details
    Browse the repository at this point in the history

  8. Add SP Space handling for decoder

    @apaniukov
    apaniukov committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    d34d401 View commit details
    Browse the repository at this point in the history