Limit GPU memory allocation of literals to be a predefined value

[eddieliao]

Add either a hard-coded or compile-time argument to set a predefined limit for GPU memory allocation of literals. Proof of concept for eventual weight streaming implementation.

[eddieliao]

HIP has a env var that can be set called HIP_HIDDEN_FREE_MEM which allows one to hide a certain amount of free memory from hipMemGetInfo(). Would prefer a env var that can just limit max free memory but I guess this works as well?
https://github.com/ROCm/clr/blob/4ec3a977b21da0025609fad0564edaa98436e077/rocclr/utils/flags.hpp#L182-L184

[eddieliao]

Created a proof of concept with fixed budget that can be set at compile time. Attached are the outputs of perf runs for no streaming, streaming with unlimited budget, and streaming with fixed budget.
resnet_perf.txt
resnet_perf_streaming_no_budget.txt
resnet_perf_streaming_50000000.txt