Test 'long_index' failed due to link_buf assertion failure

[koujl]

Original issue (eBay internal JIRA): https://jirap.corp.ebay.com/browse/SDSTOR-15786

2024-10-29 14:44:54 - Starting Test 'long_index'
2024-10-29 14:44:54 - Creating 400 GiB PVC test-index (up to 60 retries, 5 secs apart)
2024-10-29 14:44:54 - Creating 400 GiB PVC test-index (response): None None
2024-10-29 14:44:54 - Creating 5000 GiB PVC test-data1 (up to 60 retries, 5 secs apart)
2024-10-29 14:44:54 - Creating 5000 GiB PVC test-data1 (response): None None
2024-10-29 14:44:54 - Creating 5000 GiB PVC test-data2 (up to 60 retries, 5 secs apart)
2024-10-29 14:44:54 - Creating 5000 GiB PVC test-data2 (response): None None
2024-10-29 14:44:55 - Creating 5000 GiB PVC test-data3 (up to 60 retries, 5 secs apart)
2024-10-29 14:44:55 - Creating 5000 GiB PVC test-data3 (response): None None
2024-10-29 14:44:55 - create_client_pod(): using image hub.tess.io/sds/homestore:master-nightly-debug-4.0
2024-10-29 14:44:55 - Creating pod test-index-long-index-001-jbdw9 with the entrypoint 'ulimit -a; echo './core.%h.%e.%t' > /proc/sys/kernel/core_pattern; export USER_SET_CP_WD_TMR_SEC=30 && /usr/local/bin/scripts/long_running.py --test_suits=index_long_running --dirpath=/usr/local/bin/ --dev_list=/dev/index,/dev/data1,/dev/data2,/dev/data3'
2024-10-29 14:44:55 - Client pod spec specified: long_running_hdd
2024-10-29 14:44:55 - PVC passed on for client pod: []
2024-10-29 14:44:55 - Creating client pod test-index-long-index-001-jbdw9 (up to 60 retries, 5 secs apart)
2024-10-29 14:44:56 - Creating client pod test-index-long-index-001-jbdw9 (response): None None
2024-10-29 14:44:56 - Waiting up to 1800s for pod test-index-long-index-001-jbdw9 to be running (0s elapsed)
2024-10-29 14:45:56 - Waiting up to 1800s for pod test-index-long-index-001-jbdw9 to be running (60s elapsed) (Done)
2024-10-29 14:45:56 - Getting log for 86:sds-tess86-05/test-index-long-index-001-jbdw9...
2024-10-29 14:45:56 - Homestore version found: HomeStore DEBUG version: 6.5.4-522
2024-10-29 14:45:57 - Waiting up to 864000s for pod test-index-long-index-001-jbdw9 to complete/pass/fail (0s elapsed)
2024-10-29 15:15:57 - Waiting up to 864000s for pod test-index-long-index-001-jbdw9 to complete/pass/fail (1800s elapsed)
2024-10-29 15:45:57 - Waiting up to 864000s for pod test-index-long-index-001-jbdw9 to complete/pass/fail (3600s elapsed)
2024-10-29 16:15:57 - Waiting up to 864000s for pod test-index-long-index-001-jbdw9 to complete/pass/fail (5400s elapsed)
2024-10-29 16:45:58 - Waiting up to 864000s for pod test-index-long-index-001-jbdw9 to complete/pass/fail (7200s elapsed)
2024-10-29 17:15:58 - Waiting up to 864000s for pod test-index-long-index-001-jbdw9 to complete/pass/fail (9000s elapsed)
2024-10-29 17:45:58 - Waiting up to 864000s for pod test-index-long-index-001-jbdw9 to complete/pass/fail (10800s elapsed)
2024-10-29 18:15:59 - Waiting up to 864000s for pod test-index-long-index-001-jbdw9 to complete/pass/fail (12600s elapsed)
2024-10-29 18:45:59 - Waiting up to 864000s for pod test-index-long-index-001-jbdw9 to complete/pass/fail (14400s elapsed)
2024-10-29 19:15:59 - test-index-long-index-001-jbdw9: Test FAILED
2024-10-29 19:16:00 - Failure: Traceback (most recent call last):
  File "/usr/local/bin/run.py", line 257, in run_tests
    test_func()
  File "/usr/local/lib/python3.11/site-packages/nuobject_verify/tests/homestore_long_running_4_0.py", line 46, in long_index
    self.long_running_test(entrypoint, "index")
  File "/usr/local/lib/python3.11/site-packages/nuobject_verify/tests/homestore_long_running_4_0.py", line 101, in long_running_test
    self.verify.wait_for_clients_pass_fail(max_time=24*60*60* 10, interval=1800)  # 10 days
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nuobject_verify/verify.py", line 89, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nuobject_verify/verify.py", line 3398, in wait_for_clients_pass_fail
    raise VerifyException(f"Pod {pod_name}: Test FAILED!")
nuobject_verify.verify.VerifyException: Pod test-index-long-index-001-jbdw9: Test FAILED!

Pod test-index-long-index-001-jbdw9: Test FAILED!

HomeStore log print:

test_index_btree: /root/.conan/data/homestore/6.5.4-522/_/_/build/249fb080b0b87115f3eab85b3e72a64f4d0f55c0/src/lib/index/wb_cache.cpp:406: void homestore::IndexWBCache::link_buf(const IndexBufferPtr&, const IndexBufferPtr&, bool, homestore::CPContext*): Assertion `0' failed.
Aborted (core dumped)
Long running test started
options: {'test_suits': 'index_long_running', 'dirpath': '/usr/local/bin/', 'op_list': '', 'log_mods': '', 'threads': 2, 'fibers': 2, 'preload_size': 262144, 'num_entries': 2097152, 'num_iters': 100000000, 'run_time': 14400, 'dev_list': ' --device_list=/dev/index,/dev/data1,/dev/data2,/dev/data3', 'cleanup_after_shutdown': False, 'init_device': True, 'max_keys_in_node': 5, 'min_keys_in_node': 2, 'num_rounds': 10000, 'num_entries_per_rounds': 60}
Test completed
Long running test completed
Long running clean shutdown started
Test failed: Command '/usr/local/bin/test_index_btree --gtest_filter=BtreeConcurrentTest/0.ConcurrentAllOps --gtest_break_on_failure --cleanup_after_shutdown=False --init_device=True  --preload_size=262144  --run_time=14400 --num_iters=100000000 --num_entries=2097152 --num_threads=2 --num_fibers=2  --device_list=/dev/index,/dev/data1,/dev/data2,/dev/data3 ' returned non-zero exit status 134.
Test failed: Test failed for type 0
Traceback (most recent call last):
  File "/usr/local/bin/scripts/index_test.py", line 17, in run_test
    subprocess.check_call(f"{options['dirpath']}test_index_btree {cmd_opts}", stderr=subprocess.STDOUT, shell=True)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/usr/local/bin/test_index_btree --gtest_filter=BtreeConcurrentTest/0.ConcurrentAllOps --gtest_break_on_failure --cleanup_after_shutdown=False --init_device=True  --preload_size=262144  --run_time=14400 --num_iters=100000000 --num_entries=2097152 --num_threads=2 --num_fibers=2  --device_list=/dev/index,/dev/data1,/dev/data2,/dev/data3 ' returned non-zero exit status 134.

[koujl]

Test command:

/usr/local/bin/test_index_btree --gtest_filter=BtreeConcurrentTest/0.ConcurrentAllOps --gtest_break_on_failure --cleanup_after_shutdown=False --init_device=True  --preload_size=262144  --run_time=14400 --num_iters=100000000 --num_entries=2097152 --num_threads=2 --num_fibers=2