change deafult use_cache param to True, to align with the former implementation and make CI pass

[kaixuanliu]

No description provided.

[kaixuanliu]

In #1292, args.use_kv_cache was set to False by default, which will greatly slow down the performance, and cause CI failed.

[kaixuanliu]

@regisss @libinta Pls help review

[skaulintel]

@kaixuanliu

Hi kaixun. With this PR, we still see 3/10 lava tests fail with the below messages.

FAILED tests/test_image_to_text_example.py::test_image_to_text_bf16[token0-llava-hf/llava-1.5-7b-hf-1-87.2901500056982] - assert 79.04646425810226 >= ((2 - 1.05) * 87.2901500056982)
FAILED tests/test_image_to_text_example.py::test_image_to_text_fp8[token0-llava-hf/llava-1.5-7b-hf-1-115.48515989461843] - assert 102.4314524915627 >= ((2 - 1.05) * 115.48515989461843)
FAILED tests/test_image_to_text_example.py::test_image_to_text_fp8[token0-llava-hf/llava-1.5-13b-hf-1-78.2635142547838] - assert 69.15517294520089 >= ((2 - 1.05) * 78.2635142547838)

[tthakkal]

Suggested change

	type=str2bool,
	default=True,
	action="store_true",
	default=None,

this should work, don't need to set to True by default

[kaixuanliu]

This cannot solve the problem, as if we set default to None, it will pass down this param to modeling part, and change the value of use_cache in generation_config here: L671, which will slow down the performance as well.

[imangohari1]

@skaulintel @vidyasiv
I am not quite sure that the threshould should be adjusted.
these tests have actually passed before on 1.17(CI#160) so we need to at least understand the cause.

[vidyasiv]

@skaulintel @vidyasiv I am not quite sure that the threshould should be adjusted. these tests have actually passed before on 1.17(CI#160) so we need to at least understand the cause.

Updating the padding to 0 (already merged) lowers performance and improves accuracy per testing for #1366 for llava 1.5 models so we have to update perf.

[vidyasiv]

@regisss PR #1343 and #1366 are now identical.

[regisss]

@regisss PR #1343 and #1366 are now identical.

Okay, let's close this one then since #1366 is better documented.