Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix (examples/llm): Fix infinite loop in LLM entrypoint with WikiText2 #1044

Merged
merged 4 commits into from
Oct 8, 2024

Conversation

pablomlago
Copy link
Contributor

Fix for issue #1036. When fuse_sequences is False, and the seqlen is greater than 842 (maximum sequence length in WikiText2), the condition in line 131 of optimum/amd/brevitas/data_utils.py always evaluates to True, resulting in an infinite loop (see commit d0d06070225804dc6fecde00911ae38b53080123 in optimum-amd).

Consequently, we opted for enabling passing fuse_sequences as an argument and warning the user, when fuse_sequences=False, than an overly large seqlen can result in the infinite loop indicated in issue #1036.

@pablomlago
Copy link
Contributor Author

pablomlago commented Oct 8, 2024

The distribution of the sequence lengths in WikiText2, when fuse_sequences is set to False, is shown in the following picture:
length_distribution_wikitext2_train
Note that, for the default value seqlen=2048, the infinite loop is entered, as the maximum sequence length is 842.

@pablomlago pablomlago changed the title [DO NOT MERGE] Fix (examples/llm): Fix infinite loop in LLM entrypoint with WikiText2 Fix (examples/llm): Fix infinite loop in LLM entrypoint with WikiText2 Oct 8, 2024
@nickfraser nickfraser added the next release PRs which should be merged for the next release label Oct 8, 2024
@Giuseppe5
Copy link
Collaborator

Can you update the readme?
Generate the usage description with
python main.py -h
And copy and past it in the readme

@Giuseppe5 Giuseppe5 merged commit 4d8b153 into Xilinx:dev Oct 8, 2024
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
next release PRs which should be merged for the next release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants