fix(deps): update dependency tiktoken to v1.0.17

[renovate]

This PR contains the following updates:

Package	Change	Age	Adoption	Passing	Confidence
tiktoken	1.0.16 -> 1.0.17

---

Release Notes

dqbd/tiktoken (tiktoken)

v1.0.17

Compare Source

Patch Changes

• 53bd14a: Add o1 class of models

---

Configuration

📅 Schedule: Branch creation - "* 0-4 * * 3" (UTC), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[github-actions]

openai debug - [puLL-Merge] - dqbd/tiktoken@@dqbd/[email protected]..@dqbd/[email protected]

Description

This PR aims to introduce the o1 class of models into the tiktoken project. The changes span across multiple files to ensure that the new model class is recognized throughout the code base. Additionally, it includes script adjustments in the package.json to copy specific rank files to the distribution folder during the build process.

Changes

Changes

js/CHANGELOG.md

• Added entries for versions 1.0.14 and 1.0.15, describing the addition of o1 class of models.

js/package.json

• Updated the version from 1.0.13 to 1.0.15.
• Modified the build script to include a step that copies rank files to the dist folder.
• Updated files field to exclude specific rank files from the distribution.

js/src/core.ts

• Added multiple entries for o1 class of models in the getEncodingNameForModel function.

tiktoken/model_to_encoding.json

• Added mappings for various o1 models to use "o200k_base" encoding.

wasm/CHANGELOG.md, wasm/alias/CHANGELOG.md

• Added entries for version 1.0.17 describing the addition of o1 class of models.

wasm/alias/package.json, wasm/package.json

• Updated the version from 1.0.16 to 1.0.17.

wasm/src/lib.rs

• Added o1 models to the TiktokenModel type definition.
• Updated encoding_for_model function to recognize and correctly map o1 models.

Possible Issues

1. Redundant File Paths in package.json: Make sure that excluded rank files are necessary and correct. Including unnecessary exclusions could lead to unforeseen issues.
2. Model Naming Conventions: Verify that adding these new model names won't create conflicts with existing model names elsewhere in the codebase.
3. Update Version Synchronization: Given that this PR is updating versions in multiple places, ensure all these updates are consistent and won't cause any dependency mismatch.

Security Hotspots

None identified.

Changes

Changes

• Changelog Updates:

  • js/CHANGELOG.md
  • wasm/CHANGELOG.md
  • wasm/alias/CHANGELOG.md

• Package Details and Build Script:

  • js/package.json
  • wasm/alias/package.json
  • wasm/package.json

• Core Functionality for Model Recognition:

  • js/src/core.ts
  • wasm/src/lib.rs
  • tiktoken/model_to_encoding.json

[github-actions]

anthropic debug - [puLL-Merge] - dqbd/tiktoken@@dqbd/[email protected]..@dqbd/[email protected]

Description

This pull request updates the tiktoken library to support new models in the "o1" class. It includes version bumps, new model mappings, and adjustments to the build process for the JavaScript version.

Changes

Changes

1. js/CHANGELOG.md

   • Added entries for versions 1.0.15 and 1.0.14, mentioning the addition of o1 class models and missing rank files.

2. js/package.json

   • Updated version to 1.0.15
   • Modified build script to copy rank files to the dist folder
   • Added exclusions for certain files in the dist/ranks directory

3. js/src/core.ts

   • Added new model names to the getEncodingNameForModel function, mapping them to "o200k_base" encoding

4. tiktoken/model_to_encoding.json

   • Added new model mappings for o1 class models, all using "o200k_base" encoding

5. wasm/CHANGELOG.md and wasm/alias/CHANGELOG.md

   • Added entry for version 1.0.17, mentioning the addition of o1 class models

6. wasm/alias/package.json and wasm/package.json

   • Updated version to 1.0.17

7. wasm/src/lib.rs

   • Added new model names to the TiktokenModel type
   • Updated the encoding_for_model function to include mappings for new o1 class models

Possible Issues

1. The build script change in js/package.json assumes that the ranks folder exists in the src directory. If this folder is missing or located elsewhere, it could cause build failures.

2. The exclusion of certain files from the dist/ranks directory in js/package.json might lead to missing files if they are actually required for the package to function correctly.

Security Hotspots

There are no apparent security issues in this change. The modifications are primarily focused on adding support for new models and updating version numbers, which don't introduce any obvious security vulnerabilities.

[github-actions]

bedrock debug - [puLL-Merge] - dqbd/tiktoken@@dqbd/[email protected]..@dqbd/[email protected]

Description

This pull request updates the tiktoken library to include support for new "o1" class models and updates version numbers. The changes encompass both JavaScript and WebAssembly implementations, as well as updating the model-to-encoding mapping.

Changes

Changes

1. js/CHANGELOG.md:

   • Added entries for versions 1.0.15 and 1.0.14, noting the addition of o1 class models and missing rank files.

2. js/package.json:

   • Updated version to 1.0.15.
   • Modified build script to copy rank files to the dist folder.
   • Added exclusions for certain files in the dist/ranks directory.

3. js/src/core.ts:

   • Added new o1 class models to the getEncodingNameForModel function, mapping them to the "o200k_base" encoding.

4. tiktoken/model_to_encoding.json:

   • Added new o1 class models and their corresponding "o200k_base" encoding.

5. wasm/CHANGELOG.md and wasm/alias/CHANGELOG.md:

   • Added entry for version 1.0.17, noting the addition of o1 class models.

6. wasm/alias/package.json and wasm/package.json:

   • Updated version to 1.0.17.

7. wasm/src/lib.rs:

   • Added new o1 class models to the TiktokenModel type and the encoding_for_model function, mapping them to the "o200k_base" encoding.

Possible Issues

The addition of new models might require testing to ensure compatibility with existing systems that use this library. There's a potential for unexpected behavior if these new models have any unique characteristics not accounted for in the current implementation.

Security Hotspots

There are no apparent security hotspots in this change. The modifications are primarily focused on adding support for new models and updating version numbers, which don't directly impact security.