Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexpected response when container is near memory limit #2804

Open
lazorfuzz opened this issue Oct 18, 2024 · 4 comments
Open

Unexpected response when container is near memory limit #2804

lazorfuzz opened this issue Oct 18, 2024 · 4 comments

Comments

@lazorfuzz
Copy link

lazorfuzz commented Oct 18, 2024

We're seeing the same issue posted here:
#2577

StackExchange.Redis.RedisConnectionException: Unexpected response to HMGET: Integer: 1

Triggered when calling HashGetAsync and when the app is running OOM. App is running on k8s in a container with a mem limit.

StackExchange.Redis version: 2.7.17 on .NET 8, Asp.NET Core application.

Difficult to reproduce, it seems to happen when the application is close to the container mem limit. We see this happen alongside System.OutOfMemoryExceptions.

Additionally, (and much scarier) behavior is we're also occasionally getting the wrong value on GETs in this state. E.g. We check the cache for key:123, but the data we get back is what we expect for key:234. We're not sure if it's the GET returning the wrong data or if the initial SET was setting the wrong data.

@mgravell
Copy link
Collaborator

Hmm, that is weird. It sounds like a protocol desync has occurred. This is very bad, and I plan to overhaul the IO core soon (prototype is active, etc). However, you could enable "high integrity mode" (see the configuration page, https://stackexchange.github.io/StackExchange.Redis/Configuration.html) - this will both confirm whether desync is the issue, and protect you from incorrect data fetched and all the consequences thereof. If it does report a desync, please let us know!

@mgravell
Copy link
Collaborator

Note that high integrity mode has a slight performance impact, since it adds an additional (tiny) command per operation; in almost all cases, this is effectively invisible and is not a concern.

@NickCraver
Copy link
Collaborator

Can you also please elaborate on which type of container we're on here? We're assuming Linux from k8s, but which specific container base image? I don't think it'll be relevant but with networking bits, we'd like to know as much about the environment as possible.

@lazorfuzz
Copy link
Author

Yep Linux/containerd. Base image is the standard Ubuntu distro from mcr.microsoft.com/dotnet/aspnet:8.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants