Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: slow regression tests tests #4117

Merged
merged 3 commits into from
Nov 13, 2024
Merged

fix: slow regression tests tests #4117

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c5556b5
fix: slow CI tests
kostasrim Nov 12, 2024
d50845f
tune
kostasrim Nov 12, 2024
cb2e94c
comments
kostasrim Nov 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion tests/dragonfly/replication_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,6 @@ async def wait_for_replicas_state(*clients, state="online", node_role="slave", t
pytest.param(
8, [8, 8], dict(key_target=1_000_000, units=16), 50_000, False, marks=M_STRESS
),
pytest.param(8, [8, 8], dict(key_target=1_000_000, units=16), 50_000, True, marks=M_STRESS),
],
)
@pytest.mark.parametrize("mode", [({}), ({"cache_mode": "true"})])
Expand Down
53 changes: 20 additions & 33 deletions tests/dragonfly/snapshot_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -566,55 +566,42 @@ async def test_tiered_entries_throttle(async_client: aioredis.Redis):
assert await StaticSeeder.capture(async_client) == start_capture


@dfly_args({"proactor_threads": 1})
@dfly_args({"serialization_max_chunk_size": 4096, "proactor_threads": 1})
@pytest.mark.parametrize(
"query",
"cont_type",
[
("HSET"),
("SADD"),
("HASH"),
("SET"),
("ZSET"),
("LIST"),
],
)
@pytest.mark.slow
async def test_big_value_serialization_memory_limit(df_factory, query):
async def test_big_value_serialization_memory_limit(df_factory, cont_type):
dbfilename = f"dump_{tmp_file_name()}"
instance = df_factory.create(dbfilename=dbfilename)
instance.start()
client = instance.client()

ten_mb = 10_000_000
one_gb = 1_000_000_000
elements = 1000
element_size = 1_000_000 # 1mb

def ten_mb_random_string():
return "".join(random.choices(string.ascii_letters, k=ten_mb))

one_gb = 1_000_000_000 # 1GB

upper_limit = one_gb * 1.1 # 1GB + 100MB

i = 0

while instance.rss < one_gb:
if query == "HSET":
i = i + 1
await client.execute_command(f"HSET foo_key foo_{i} {ten_mb_random_string()}")
elif query == "SADD":
await client.execute_command(f"SADD foo_key {ten_mb_random_string()}")
elif query == "ZSET":
await client.execute_command(f"ZADD foo_key {i} {ten_mb_random_string()}")
elif query == "LIST":
await client.execute_command(f"LPUSH foo_key {ten_mb_random_string()}")

async def check_memory_usage(instance):
while True:
assert instance.rss < upper_limit
await asyncio.sleep(0.01)

checker = asyncio.create_task(check_memory_usage(instance))
await client.execute_command(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add a comment here what it exectly does.
i.e add 1 db entry of given type with elements num each one of size element size

Now my question is why do we end up with more than 2g rss if we have one entry of 1000 elemets each is 1Mb size

Copy link
Contributor Author

@kostasrim kostasrim Nov 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RSS will grow during DEBUG populate, For example run:

debug populate 1 prefix 1000000 TYPE hash RAND ELEMENTS 1000

Creates a hash table with 1GB total size. Now do INFO MEMORY. RSS is: 2062901248 (2GB ~ doubled).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not answer the question why, but we can continue with this PR and please create another github ticket so we can follow up on this to investigate why we have this overhead

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uh you meant what is causing the rss spike. I do not know, it was an observation. I created an issue #4124

f"debug populate 1 prefix {element_size} TYPE {cont_type} RAND ELEMENTS {elements}"
)

info = await client.info("ALL")
# rss double's because of DEBUG POPULATE
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do you mean, why does it doubles because of DEBUG POPULATE?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see my other comment

assert info["used_memory_peak_rss"] > (one_gb * 2)
# if we execute SAVE below without big value serialization we trigger the assertion below.
# note the peak would reach (one_gb * 3) without it.
await client.execute_command("SAVE")
info = await client.info("ALL")

upper_limit = 2_250_000_000 # 2.25 GB
assert info["used_memory_peak_rss"] < upper_limit

checker.cancel()
await client.execute_command("FLUSHALL")
await client.close()

Expand Down
Loading