Add Tokenizers #687
Closed
Add Tokenizers #687
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…h and without string support in OV core. Moved StringTensorUnpack and reworked it to be aligned with the new approach. Reworked sentece piece op and translation code to be compatible with several variants of string tensor representation and the plugin wrapping hack.
…ranch to contrib in form compatible with both master and the branch with string tensors support. Added CaseFoldUTF8 from that branch.
…pty constants, register StringTensorPack and StringTensorUnpack as OV operations to be able to read IRs with those operations
…den Const translator for TF to intercept string constants
…r conditional compilation based on available features in OpenVINO
…combination of WordpieceTokenizeWithOffsets and LookupTableFindV2 from TensorFlow
…ute initialization optional (needed for core.make_node)
…ean mask extra output
…n and RegexSplit based on paddle fast_tokenizer lib. Limited implementation, not all of the features of ops and TF translated ops are implemented.
… necessary steps to complete HF bert preprocessing conversion (not validated)
…dling of model name
…kenizer and main model is fixed partially (still produces topologically incorrect model)
…uts, now Bert and its tokenizer are connected together correctly
…ding, fix bugs for batches processing
…bled debug output
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run openvino_contrib-mac |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run openvino_contrib-mac |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run openvino_contrib-mac |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run openvino_contrib-mac |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run openvino_contrib-mac |
|
Azure Pipelines successfully started running 1 pipeline(s). |
apaniukov
force-pushed
the
tokenizer-fix-decode
branch
2 times, most recently
from
5c3b656
to
d34d401
Compare
|
/azp run openvino_contrib-mac |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Wovchena
suggested changes
Nov 21, 2023
There was a problem hiding this comment.
I can see Windows compiles now. But ov_tokenizer.init_extension() fails for me:
import os
import sys
import ov_tokenizer
if hasattr(os, "add_dll_directory"):
for path in os.environ.get("PATH", "").split(";"):
if os.path.isdir(path):
os.add_dll_directory(path)
ov_tokenizer.init_extension(sys.argv[1])py llm/cpp/convert_tokenizers.py c:/Users/vzlobin/r/openvino.genai/build/thirdparty/openvino_contrib/modules/custom_operations/user_ie_extensions/Release/user_ov_extensions.dll C:\Users\vzlobin\r\tiny-llama-fast-tokenizer
Traceback (most recent call last):
File "C:\Users\vzlobin\r\openvino.genai\llm\cpp\convert_tokenizers.py", line 9, in <module>
ov_tokenizer.init_extension(sys.argv[1])
File "C:\Users\vzlobin\r\openvino.genai\thirdparty\openvino_contrib\modules\custom_operations\user_ie_extensions\tokenizer\python\ov_tokenizer\node_factory.py", line 21, in init_extension
factory.add_extension(extension_path)
File "C:\Users\vzlobin\Downloads\w_openvino_toolkit_windows_2023.2.0.13089.cfd42bd2cb0_x86_64\python\openvino\runtime\utils\node_factory.py", line 118, in add_extension
self.factory.add_extension(lib_path)
RuntimeError: Cannot load library 'c:/Users/vzlobin/r/openvino.genai/build/thirdparty/openvino_contrib/modules/custom_operations/user_ie_extensions/Release/user_ov_extensions.dll': 126 from cwd: C:\Users\vzlobin\r\openvino.genai
|
/azp run openvino_contrib-mac |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Replaced by #767 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Details:
This PR extends OV Opset with tokenization-related operations
Ticket: