-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unexpected response when container is near memory limit #2804
Comments
Hmm, that is weird. It sounds like a protocol desync has occurred. This is very bad, and I plan to overhaul the IO core soon (prototype is active, etc). However, you could enable "high integrity mode" (see the configuration page, https://stackexchange.github.io/StackExchange.Redis/Configuration.html) - this will both confirm whether desync is the issue, and protect you from incorrect data fetched and all the consequences thereof. If it does report a desync, please let us know! |
Note that high integrity mode has a slight performance impact, since it adds an additional (tiny) command per operation; in almost all cases, this is effectively invisible and is not a concern. |
Can you also please elaborate on which type of container we're on here? We're assuming Linux from k8s, but which specific container base image? I don't think it'll be relevant but with networking bits, we'd like to know as much about the environment as possible. |
Yep Linux/containerd. Base image is the standard Ubuntu distro from |
We're seeing the same issue posted here:
#2577
Triggered when calling
HashGetAsync
and when the app is running OOM. App is running on k8s in a container with a mem limit.StackExchange.Redis version:
2.7.17
on .NET 8, Asp.NET Core application.Difficult to reproduce, it seems to happen when the application is close to the container mem limit. We see this happen alongside System.OutOfMemoryExceptions.
Additionally, (and much scarier) behavior is we're also occasionally getting the wrong value on GETs in this state. E.g. We check the cache for
key:123
, but the data we get back is what we expect forkey:234
. We're not sure if it's the GET returning the wrong data or if the initial SET was setting the wrong data.The text was updated successfully, but these errors were encountered: