Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add warning about token length being greater than GPT limits #26

Open
hargup opened this issue Mar 17, 2023 · 2 comments
Open

Add warning about token length being greater than GPT limits #26

hargup opened this issue Mar 17, 2023 · 2 comments

Comments

@hargup
Copy link

hargup commented Mar 17, 2023

I have an internal library for counting tokens on files and directories, should probably opensource it.
@tddschn
Copy link

tddschn commented Mar 19, 2023

Is it based on OpenAI's tiktoken library? @hargup

@hargup
Copy link
Author

hargup commented Mar 19, 2023

Yes it is based on tiktoken, just open sourced it. https://github.com/felvin-search/token-count

shanecp added a commit to shanecp/gpt-repository-loader that referenced this issue Apr 3, 2023
@shanecp
- Add token limit flag and file limit, so output can be split by tokens
f8a7e90 
- Update docs to support
- Add requirements file.

This gives a solution for mpoon#35, mpoon#37. Perhaps mpoon#26
@shanecp shanecp mentioned this issue Apr 3, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hargup @tddschn