refactor: avoid downloading tokenizer if truncate is False

[anakin87]

Related Issues

• fixes AmazonBedrockGenerator shouldn't load tokenizer if truncate is set to False. #1125

Proposed Changes:

If truncate is False, do not initialize the PromptHandler (which internally downloads the tokenizer).
In the Generator, the PromptHandler is only used to truncate the text if needed.

How did you test it?

CI + a new unit test.

Notes for the reviewer

This change only applies to AmazonBedrockGenerator.

I would have liked to do the same for the corresponding Chat Generator.
Unfortunately, in the Chat Generator, the PromptHandler (and its internal tokenizer) have 2 different purposes:

• transform messages into text with special tokens (via tokenizer.apply_chat_template)
• truncate the text, like in the Generator

Therefore, we cannot easily get rid of the tokenizer.

(I hope we can prioritize #977 soon: this would greatly simplify the Chat Generator.)

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

[Amnah199]

LG!