skip hpugraph usage for first token to save memory

[polisettyvarma]

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[polisettyvarma]

@regisss please add @dvarshney-habana to reviewers
please review this @dvarshney-habana and @regisss

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[polisettyvarma]

please review now @regisss, @dvarshney-habana and @puneeshkhanna

[regisss]

CI failed because _get_hpu_graphs_kwargs should be linked to Transformers' `generate here:

optimum-habana/optimum/habana/transformers/modeling_utils.py

Line 103 in af3da93

# Generation is modified to run faster in lazy mode

Or, maybe easier, you can just declare this method outside of this class and use it that way without self.

[polisettyvarma]

CI failed because _get_hpu_graphs_kwargs should be linked to Transformers' `generate here:

optimum-habana/optimum/habana/transformers/modeling_utils.py

Line 103 in af3da93

# Generation is modified to run faster in lazy mode

Or, maybe easier, you can just declare this method outside of this class and use it that way without self.

I took the first approach, please review @regisss

[regisss]

@polisettyvarma CI failed, we need to return hpu_graphs_kwargs at the end of _get_hpu_graphs_kwargs.
There is also one merge conflict to solve, probably due to the trim logit PR that I just merged. There shouldn't be much to do I think.

[polisettyvarma]

please review now @regisss

[yafshar]

@regisss, @polisettyvarma which model is using this feature? I do not see bypass_hpu_graphs in any models' arguments. Would you please point me to a model which is using this?

[regisss]

@regisss, @polisettyvarma which model is using this feature? I do not see bypass_hpu_graphs in any models' arguments. Would you please point me to a model which is using this?

@yafshar You won't see it in the code of any model. It's managed by the HPU graph API which wraps the forward of the model.
For example, in a Docker container running the SynapseAI 1.15 Docker image, you can find it at the beginning of the wrapped_hpugraph_forward method in /usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/hpu/graphs.py.