[Tiered caching] Defining a new setting for tiered disk cache storage path

[sgup432]

Description

Adding new setting for disk cache storage path specifically within tiered cache. Earlier we used specific cache implementation setting which might be confusing to user. This changes that now.

Example setting

indices.request.cache.tiered_spillover.disk.store.storage.path: /tmp

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for a9cbe24: SUCCESS

[sgup432]

@jainankitk

[github-actions]

❌ Gradle check result for c29acc1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for cdbe57a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[sgup432]

As suggested by @jainankitk , will pause on this. As we are keeping a empty default value inside storage setting, but still rely on a non empty default storagePath value coming from outside. Might be confusing. Will see if there is a way to do it.

[jainankitk]

While the changes look fine as such, I have couple of concerns with including this storage path setting:

• Even though default setting is configured as blank, we are internally using some default value which is not very transparent for the users. Ideally they should be able to see that value using _cluster/settings?include_defaults=true
• Any disk storage path outside of data directory will not be accounted for within the DiskThreshold circuit breakers, that puts OpenSearch cluster at risk of going down