Added functions to control the cache eviction

[gobinathsr]

Added functions to control the cache eviction

Root Cause:
Assigned credit for a pacer period is utilized without considering the bytes in flight.
Hence bytes flight increases a lot above the allowed maximum threshold level and leading to the cache eviction & leading to cache miss.

Fix info:
Added the below functions to control the cache eviction

1. is_credit_available() - When credit availably checked, also checking bytes in flight is with in the maximum allowed bytes
   Default enabled with the macro SPDK_NVMF_RDMA_IO_PACER_ALLOW_WITHIN_CREDIT_ONLY

2. is_entry_size_within_limit() - When processing the request from the pacer check if the request size is within in the allowed bytes range
   Default disabled with the macro SPDK_NVMF_RDMA_IO_PACER_ALLOW_IO_WITHIN_AVAIL - Reason to allow processing a single request to consider the slow disk/other slowness in processing the request

Added the macros to configure the tuner type and the proposals. Hence with the change will send the parameters from the test script to configure dynamically.

[Features/Functional areas and/or component interfaces affected]
General SPDK

[Whether the change affects how hardware is programmed]
Yes - Configures request out of pacer queue to be processed

Local testing:
Tested with Tuner2 - Test14 test case and below is the test result

test_14
CPU mask: 0xF0 Num cores:4 IO pacer period:5600 Adjusted period:22400
./test_NoArg.sh: line 336: 708492 Terminated tail -f > rpc_pipe
| QD | BW | BW Max | WIRE BW | AVG LAT, us | BW STDDEV | L3 Hit Rate | Bufs in-flight (MiB) | Pacer period, us
| 256 | 185.6 | 191.0 | 188.9947 | 2891.1 | .6 | 99.3 | 29.6 (3.7) | 22.5

[gobinathsr]

Pls ignore the first 3 commit added for testing with pull request. On merging, will squash and merge as single commit.

[ldewangan]

It seems multiple changes are there in single patch.
Can you do the split?

1. Fix the allowed bytes and bytes_in_flight issue
2. Fix the pacer expiry and credit allocation method
3. Add the tuner’s algorithms

[swx-snap]

Can one of the admins verify this patch?