Local schema cache weighted invalidation based on schema size

[mattsewall-stripe]

Summary

• While running schema registry we have observed scenarios where large schemas that contain many versions do not get invalidated quickly enough from the local schema cache. (Similar to large objects in an avalanche rising to the surface, large schemas with many references stay in the cache for max time period.)
• Rather than increase the total heap space of our schema registry nodes or decrease the total size of the cache we've solved this issue with weight based local schema cache invalidation. Our schemas vary greatly in size and some of them are very large. It’s not efficient for us to size heap to accommodate a full cache of very large schemas without OOMing.
• No performance degradation has been observed in our schema registry instance, but would love to work through mainline benchmark strategies as needed. Our hypothesis for this reasoning is that our cache hit % has stayed constant between sized based cache and weight based cache.
• Included in this pr is metrics to track the local schema cache in both size and hits + misses.
• By default this feature is not enabled.

Motivation

• We have observed scenarios operating the schema registry where during periods of heavy write traffic the leader node will OOM based on the local cache becoming too large. This write traffic is extremely spiky and contains large schemas that contain many versions (important for checking compatibility).
• This change hopefully serves to provide operational clarity into managing the local schema cache + provide a fix for this failure scenario.

[mattsewall-stripe]

The sum of canonical schema length must be under 1,000,000 characters is what this variable means. From the guava cache docs there's a lot of fuzziness as to how things get invalidated within the cache itself (based on usage and % of contribution to weight total?) but we've set this limit such that large schemas should be getting invalidated over time. What this essentially means is that more smaller schemas will be cached (and that's good!) as we should be able to cache quite a few of them. No significant performance degradation has materialized from making this change.

[rayokota]

Thanks for the PR @mattsewall-stripe!

The build failed with a few checkstyle errors

[mattsewall-stripe]

Turned off by default but in my opinion likely could be turned on by default.

[mattsewall-stripe]

@rayokota

Hey! I apologize for the delay but have built project locally and ran tests :)

[mattsewall-stripe]

Hi! @rayokota can you take another look? Coming around to this again

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}