Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ZstdSerializer, change Key to include "domain" #58

Merged
merged 1 commit into from
Mar 6, 2024

Conversation

bisho
Copy link
Member

@bisho bisho commented Mar 5, 2024

Changes:

  • Add ZstdSerializer. Can be configured to default to zlib for backwards-compatibility. Can use zstd dicts. Can decompress and deserialize anything MixedSerialized supported before.
  • Change Key to include "domain"
  • Change Key to include skip compression, so if the data is compressed on user space, it wont be re-compressed again, or if the data is too sensitive and needs to skip compression performance hit.
  • Change Key, removing the old unicode flag. It is no longer needed (with rust serialization key encoding just works, no mater unicode, binary or anything).

The idea is that domain can be used to mark keys as belonging to a use-case domain, and specify custom Zstd dictionaries for some domains. Also domains can be used to track per-domain counters (hits, hit ratio, etc), but this is not implemented yet.

@bisho bisho marked this pull request as ready for review March 5, 2024 13:30
@bisho bisho changed the title zstd serializer Add ZstdSerializer, change Key to include "domain" Mar 5, 2024
@bisho bisho force-pushed the zstd_serializer branch from d056746 to 1d2ab94 Compare March 5, 2024 13:38
@bisho bisho force-pushed the zstd_serializer branch from 1d2ab94 to 24a1c7a Compare March 5, 2024 14:09
@bisho bisho force-pushed the zstd_serializer branch from 24a1c7a to ef22e7e Compare March 5, 2024 14:12
@bisho bisho force-pushed the zstd_serializer branch from ef22e7e to 9a284d2 Compare March 5, 2024 14:13
@bisho bisho force-pushed the zstd_serializer branch from 9a284d2 to c059e64 Compare March 5, 2024 14:15
@bisho bisho force-pushed the zstd_serializer branch from c059e64 to bd6882f Compare March 5, 2024 14:19
@bisho bisho force-pushed the zstd_serializer branch from bd6882f to 9177323 Compare March 5, 2024 14:20
@bisho bisho force-pushed the zstd_serializer branch from 9177323 to 369e9a0 Compare March 5, 2024 14:27
@bisho bisho changed the base branch from rust_build_cmd to main March 5, 2024 16:24
@bisho bisho changed the base branch from main to rust_build_cmd March 5, 2024 16:58
tests/serializer_test.py Outdated Show resolved Hide resolved
src/meta_memcache/serializer.py Outdated Show resolved Hide resolved
src/meta_memcache/serializer.py Show resolved Hide resolved
@bisho bisho force-pushed the zstd_serializer branch from 369e9a0 to 7e96f8e Compare March 5, 2024 22:16
Copy link
Member

@tonycosentini tonycosentini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love the domain concept - super simple.

Would it make sense to also allow different serializers for different domains? My hunch is no because it makes things very complicated (in particular, the application code now controls more details around compatiblity...).

@bisho
Copy link
Member Author

bisho commented Mar 6, 2024

Love the domain concept - super simple.

Would it make sense to also allow different serializers for different domains? My hunch is no because it makes things very complicated (in particular, the application code now controls more details around compatiblity...).

We can definitely build a new serializer that routes serialization depending on domain, it can use composition to reuse existing ones.

Serializing in jsonb our data-only objects might be very interesting if we get out of the python-only world and need to share cache. I think the domain thing in the key allows to build anything we want, but I would try to keep a single serializer per client. Just make a proxy serializer if you need custom things.

@bisho bisho merged commit 0bd497a into rust_build_cmd Mar 6, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants