Ability to customise cache TTL

[Maxim-Filimonov]

Proposal

Ability to customise cache expiration age.
Suggested api:
On specific prompt - cache: {ttl: {days: 30}}
Allowed values: {ttl: {hours, days, minutes}}
Sdk sums up and converts them to ms.

To keep it backwards compatible I’m thinking if cache: true it defaults to {ttl: {days: 3}} when true is specified.

Not sure if there is invalidation api already if one doesn’t exists it might be required.

Use-Case

We have certain queries that are used to generate content in our system. There is no need to run them again every time user requests the same content. Moreover, unless model has been changed or the prompt itself there should be no need to run them again at all.
As mentioned in our discussion current cache implementation resets age every time content is accessed which is perfect for our use case. We don’t necessarily want to store years of content only accessed once

Is this a feature you are interested in implementing yourself?

Maybe