Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot replay requests when amplification factor of requests is set too high #348

Open
rainjuns opened this issue Nov 2, 2024 · 1 comment
Open

Cannot replay requests when amplification factor of requests is set too high #348

rainjuns opened this issue Nov 2, 2024 · 1 comment

Comments

@rainjuns
Copy link

rainjuns commented Nov 2, 2024 •
edited
Loading

Describe the bug

If the amplification factor of requests (i.e., test_config::replayGeneratorConfig::ampFactor) is set too high, KVReplayGenerator stops generating the request and CacheLib continuously shows a hit ratio 0.00% in the output.

I first try to find if the requests are constantly generated and find out that KVReplayGenerator::genRequests() ends up in an infinite loop.

https://github.com/facebook/CacheLib/blob/75457b785f3de23903acf651e74aa537992b7345/cachelib/cachebench/workload/KVReplayGenerator.h#L513C1-L519C4

While the consumers (i.e., stressors) have to consume the requests from the queue of stressor (let's say 0), every consumer (i.e., KVReplayGenerator::getReq()) also ends up in an infinite loop.

https://github.com/facebook/CacheLib/blob/75457b785f3de23903acf651e74aa537992b7345/cachelib/cachebench/workload/KVReplayGenerator.h#L504C1-L529C2

Because every consumer is stuck in the infinite loop, waiting for requests from queues corresponding to stressors other than 0, the requests in the queue of stressor 0 remain full.

Using GDB and reviewing the source code, I identified three problems.

  1. One of stressor threads keeps waiting in it.wait() in findFn defined in Cache<Allocator>::find(Key key).
  2. One of reader threads is stuck in obtaining TimedMutex so that it cannot finish onGetComplete.
  3. I am not sure this part but, OnGetComplete acquires a lock, hindering removeFromFillMap from acquiring the lock later. So, it seems that lock must be released after checking hasTombStone(hk).
    https://github.com/facebook/CacheLib/blob/75457b785f3de23903acf651e74aa537992b7345/cachelib/allocator/nvmcache/NvmCache.h#L1186C1-L1186C31
    https://github.com/facebook/CacheLib/blob/75457b785f3de23903acf651e74aa537992b7345/cachelib/allocator/nvmcache/NvmCache.h#L1143C1-L1144C75

Please note that even if (2) is not happening, (1) still happens.

I attached the GDB stacktrace below.

To Reproduce
For the fast debugging, I just return Status::NotFound in lookup() function of navy cache.

Here's the JSON file that I used.

{
  "cache_config": {
    "cacheSizeMB": 430,
    "nvmCacheSizeMB": 95230,
    "nvmCachePaths": [],
    "writeAmpDeviceList": [],
    "navyAdmissionWriteRateMB": 50,
    "cacheDir": "/root/cachelib_metadata",
    "allocFactor": 1.08,
    "maxAllocSize": 524288,
    "minAllocSize": 64,
    "navySizeClasses": [],
    "navyReaderThreads": 72,
    "navyWriterThreads": 36,
    "navySmallItemMaxSize": 2048,
    "navySegmentedFifoSegmentRatio": [
      1.0
    ],
    "navyHitsReinsertionThreshold": 1,
    "navyBlockSize": 4096,
    "nvmAdmissionRetentionTimeThreshold": 7200,
    "navyParcelMemoryMB": 6048,
    "enableChainedItem": true,
    "htBucketPower": 29,
    "moveOnSlabRelease": false,
    "poolRebalanceIntervalSec": 2,
    "poolResizeIntervalSec": 2,
    "rebalanceStrategy": "hits"
  },
  "test_config": {
    "traceFileNames": [
      "/root/shared/facebook/kvcache/202206/kvcache_traces_1.csv",
      "/root/shared/facebook/kvcache/202206/kvcache_traces_2.csv"
    ],
    "opRatePerSec": 550000,
    "opRateBurstSize": 200,
    "enableLookaside": false,
    "repeatTraceReplay": false,
    "repeatOpCount": true,
    "onlySetIfMiss": false,
    "numOps": 100000000000,
    "numThreads": 16,
    "prepopulateCache": true,
    "replayGeneratorConfig": {
      "ampFactor": 100
    },
    "generator": "replay"
  }
}

Expected behavior
Every consumer thread must not in the infinite loop at a certain point.

Screenshots
Here is a part of log messages that I printed for the purpose of debugging.

kv getReq while ... is printed out in the while statement of KVReplayGenerator::getReq().
genReq while ... is printed out in the while statement of KVReplayGenerator::genRequests()

1102 14:18:03.240128 247359 KVReplayGenerator.h:521] kv getReq while, shard id: 12, reqQ empty: true, reqQ full: false
I1102 14:18:03.298928 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:03.298959 247209 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:04.108278 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:04.240181 247357 KVReplayGenerator.h:521] kv getReq while, shard id: 10, reqQ empty: true, reqQ full: false
I1102 14:18:04.298998 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:04.299035 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:05.108232 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:05.240127 247352 KVReplayGenerator.h:521] kv getReq while, shard id: 5, reqQ empty: true, reqQ full: false
I1102 14:18:05.299079 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:05.299106 247232 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:06.108125 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:06.240125 247347 KVReplayGenerator.h:521] kv getReq while, shard id: 0, reqQ empty: true, reqQ full: false
I1102 14:18:06.299164 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:06.299178 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:07.108197 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:07.240201 247349 KVReplayGenerator.h:521] kv getReq while, shard id: 3, reqQ empty: true, reqQ full: false
I1102 14:18:07.299233 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:07.299241 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:08.108180 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:08.240135 247356 KVReplayGenerator.h:521] kv getReq while, shard id: 9, reqQ empty: true, reqQ full: false
I1102 14:18:08.299300 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:08.299303 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:09.108258 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:09.240124 247355 KVReplayGenerator.h:521] kv getReq while, shard id: 8, reqQ empty: true, reqQ full: false
I1102 14:18:09.299386 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:09.299402 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:10.108220 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:10.240131 247351 KVReplayGenerator.h:521] kv getReq while, shard id: 4, reqQ empty: true, reqQ full: false
I1102 14:18:10.299445 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:10.299473 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:11.108255 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:11.240190 247348 KVReplayGenerator.h:521] kv getReq while, shard id: 1, reqQ empty: true, reqQ full: false
I1102 14:18:11.299536 247265 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:11.299564 247232 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:12.108252 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:12.240130 247353 KVReplayGenerator.h:521] kv getReq while, shard id: 6, reqQ empty: true, reqQ full: false
I1102 14:18:12.299608 247265 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:12.299629 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:13.108189 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:13.240127 247352 KVReplayGenerator.h:521] kv getReq while, shard id: 5, reqQ empty: true, reqQ full: false
I1102 14:18:13.299734 247265 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:13.299748 247209 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:14.108222 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:14.240138 247349 KVReplayGenerator.h:521] kv getReq while, shard id: 3, reqQ empty: true, reqQ full: false
I1102 14:18:14.299856 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:14.299855 247232 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:15.108183 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:15.240139 247351 KVReplayGenerator.h:521] kv getReq while, shard id: 4, reqQ empty: true, reqQ full: false
I1102 14:18:15.299937 247209 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:15.299981 247265 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:16.108125 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:16.240120 247352 KVReplayGenerator.h:521] kv getReq while, shard id: 5, reqQ empty: true, reqQ full: false
I1102 14:18:16.300070 247209 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:16.300092 247232 ThreadPoolJobQueue.cpp:79] running workers: 0
I1102 14:18:17.108263 247197 KVReplayGenerator.h:487] genReq while, nreqs: 23732699, finished: false, shard id: 13, reqQ empty: false, reqQ full: true
I1102 14:18:17.240131 247359 KVReplayGenerator.h:521] kv getReq while, shard id: 12, reqQ empty: true, reqQ full: false
I1102 14:18:17.300180 247265 ThreadPoolJobQueue.cpp:82] go to wait, status: 0
I1102 14:18:17.300220 247209 ThreadPoolJobQueue.cpp:79] running workers: 0

Here is the output of GDB. Certain stressor thread(s) is(are) waiting.

#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007ffff747caff in folly::detail::(anonymous namespace)::nativeFutexWaitImpl (waitMask=4294967295, absSteadyTime=0x0,
    absSystemTime=0x0, expected=4294967293, addr=0x7fe7100bc428) at /root/workspace/kvcache/cachelib/external/folly/folly/detail/Futex.cpp:126
#2  folly::detail::futexWaitImpl (futex=futex@entry=0x7fe7100bc428, expected=expected@entry=4294967293, absSystemTime=absSystemTime@entry=0x0,
    absSteadyTime=absSteadyTime@entry=0x0, waitMask=waitMask@entry=4294967295)
    at /root/workspace/kvcache/cachelib/external/folly/folly/detail/Futex.cpp:254
#3  0x00007ffff74ee6ce in folly::detail::futexWait<std::atomic<unsigned int> > (waitMask=4294967295, expected=4294967293, futex=0x7fe7100bc428)
    at /root/workspace/kvcache/cachelib/external/folly/folly/detail/Futex-inl.h:94
#4  folly::detail::MemoryIdler::futexWait<std::atomic<unsigned int>, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (
    idleTimeout=..., stackToRetain=1024, timeoutVariationFrac=0.5, waitMask=4294967295, expected=4294967293,
    fut=std::atomic<unsigned int> = { 4294967293 }) at /root/workspace/kvcache/cachelib/external/folly/folly/detail/MemoryIdler.h:128
#5  folly::detail::MemoryIdler::futexWait<std::atomic<unsigned int>, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (
    timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, expected=4294967293, fut=...)
    at /root/workspace/kvcache/cachelib/external/folly/folly/detail/MemoryIdler.h:106
#6  folly::fibers::Baton::waitThread (this=0x7fe7100bc428) at /root/workspace/kvcache/cachelib/external/folly/folly/fibers/Baton.cpp:70
#7  0x00007ffff74ee9ae in folly::fibers::Baton::wait<folly::fibers::Baton::wait()::<lambda()> > (mainContextFunc=..., this=<optimized out>)
    at /root/workspace/kvcache/cachelib/external/folly/folly/fibers/Baton-inl.h:54
#8  0x000055555563a14d in facebook::cachelib::detail::ReadHandleImpl<facebook::cachelib::CacheItem<facebook::cachelib::LruCacheTrait> >::ItemWaitContext::wait (this=0x7fe7100bc420) at /root/workspace/kvcache/cachelib/../cachelib/allocator/Handle.h:271
#9  facebook::cachelib::detail::ReadHandleImpl<facebook::cachelib::CacheItem<facebook::cachelib::LruCacheTrait> >::ItemWaitContext::wait (
    this=0x7fe7100bc420) at /root/workspace/kvcache/cachelib/../cachelib/allocator/Handle.h:266
#10 facebook::cachelib::detail::ReadHandleImpl<facebook::cachelib::CacheItem<facebook::cachelib::LruCacheTrait> >::wait (this=0x7fe723ff6450)
    at /root/workspace/kvcache/cachelib/../cachelib/allocator/Handle.h:230
#11 facebook::cachelib::cachebench::Cache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::find(facebook::cachelib::KAllocation::Key)::{lambda()#1}::operator()() const (__closure=__closure@entry=0x7fe723ff6260)
    at /root/workspace/kvcache/cachelib/../cachelib/cachebench/cache/Cache.h:1018
#12 0x0000555555648205 in facebook::cachelib::cachebench::Cache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::find (
    this=<optimized out>, key=...) at /root/workspace/kvcache/cachelib/../cachelib/cachebench/cache/Cache.h:1036
#13 0x0000555555652763 in facebook::cachelib::cachebench::CacheStressor<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::stressByDiscreteDistribution (this=0x555555bac730, stats=...) at /root/workspace/kvcache/opt/cachelib/include/folly/Range.h:306
#14 0x00007ffff6fb8db4 in std::execute_native_thread_routine (__p=0x7fe71c000b70) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:104
#15 0x00007ffff6c3fa94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#16 0x00007ffff6cccc3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

One of reader threads seems to be waiting to obtain TimeMutex.

#0  0x00007ffff6c8fadf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7fe756fd4480, rem=0x7fe756fd4480)
    at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007ffff6c9ca27 in __GI___nanosleep (req=<optimized out>, rem=<optimized out>) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x00005555556cfbb8 in std::this_thread::sleep_for<long, std::ratio<1l, 1000000000l> > (
    __rtime=<synthetic pointer>std::chrono::duration = { <optimized out>ns }) at /usr/include/c++/13/bits/this_thread_sleep.h:80
#3  folly::detail::Sleeper::wait (this=<optimized out>)
    at /root/workspace/kvcache/opt/cachelib/include/folly/synchronization/detail/Sleeper.h:56
#4  folly::MicroSpinLock::lock (this=0x7fe7142cc821) at /root/workspace/kvcache/opt/cachelib/include/folly/synchronization/MicroSpinLock.h:81
#5  folly::SpinLock::lock (this=0x7fe7142cc821) at /root/workspace/kvcache/opt/cachelib/include/folly/SpinLock.h:45
#6  std::unique_lock<folly::SpinLock>::lock (this=0x7fe756fd4470) at /usr/include/c++/13/bits/unique_lock.h:141
#7  std::unique_lock<folly::SpinLock>::unique_lock (__m=..., this=0x7fe756fd4470) at /usr/include/c++/13/bits/unique_lock.h:71
#8  folly::fibers::TimedMutex::lockHelper<folly::fibers::TimedMutex::lock()::{lambda(folly::fibers::TimedMutex::MutexWaiter&)#1}>(folly::fibers::TimedMutex::lock()::{lambda(folly::fibers::TimedMutex::MutexWaiter&)#1}&&) (this=this@entry=0x7fe7142cc820, waitFunc=...)
    at /root/workspace/kvcache/opt/cachelib/include/folly/fibers/TimedMutex-inl.h:30
#9  0x000055555579d510 in folly::fibers::TimedMutex::lock (this=<optimized out>)
    at /root/workspace/kvcache/opt/cachelib/include/folly/fibers/TimedMutex-inl.h:85
#10 std::unique_lock<folly::fibers::TimedMutex>::lock (this=0x7fe756fd4540) at /usr/include/c++/13/bits/unique_lock.h:141
#11 std::unique_lock<folly::fibers::TimedMutex>::unique_lock (__m=..., this=0x7fe756fd4540) at /usr/include/c++/13/bits/unique_lock.h:71
#12 facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::getFillLockForShard (shard=7136,
    this=0x7fe7140ed620) at /root/workspace/kvcache/cachelib/../cachelib/allocator/nvmcache/NvmCache.h:431
#13 facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::getFillLock (hk=...,
    this=<optimized out>) at /root/workspace/kvcache/cachelib/../cachelib/allocator/nvmcache/NvmCache.h:435
#14 facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::removeFromFillMap (
    this=0x7fe7140ed620, hk=...) at /root/workspace/kvcache/cachelib/../cachelib/allocator/nvmcache/NvmCache.h:382
#15 0x00005555557b9bdf in facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::onGetComplete(facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::GetCtx&, facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::BufferViewT<unsigned char const>)::{lambda()#1}::operator()() const (
    __closure=0x7fe756fd4638) at /root/workspace/kvcache/cachelib/../cachelib/allocator/nvmcache/NvmCache.h:1090
#16 folly::catch_exception<facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::onGetComplete(facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::GetCtx&, facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::BufferViewT<unsigned char const>)::{lambda()#1}&, void (*)() noexcept, , void>(facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::onGetComplete(facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::GetCtx&, facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::BufferViewT<unsigned char const>)::{lambda()#1}&, void (*&&)() noexcept) (c=<optimized out>, t=...)
    at /root/workspace/kvcache/opt/cachelib/include/folly/lang/Exception.h:290
#17 folly::detail::ScopeGuardImpl<facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::onGetComplete(facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::GetCtx&, facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::BufferViewT<unsigned char const>)::{lambda()#1}, true>::execute() (
    this=0x7fe756fd4630) at /root/workspace/kvcache/opt/cachelib/include/folly/ScopeGuard.h:181
#18 folly::detail::ScopeGuardImpl<facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::onGetComplete(facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::GetCtx&, facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::BufferViewT<unsigned char const>)::{lambda()#1}, true>::~ScopeGuardImpl() (
    this=0x7fe756fd4630, __in_chrg=<optimized out>) at /root/workspace/kvcache/opt/cachelib/include/folly/ScopeGuard.h:154
#19 facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::onGetComplete (val=..., hk=...,
    status=<optimized out>, ctx=..., this=<optimized out>) at /root/workspace/kvcache/cachelib/../cachelib/allocator/nvmcache/NvmCache.h:1146
#20 facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::find(facebook::cachelib::HashedKey)::{lambda(facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::Buffer)#1}::operator()(facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::Buffer) const (v=..., k=..., s=<optimized out>, __closure=<optimized out>)
    at /root/workspace/kvcache/cachelib/../cachelib/allocator/nvmcache/NvmCache.h:669
#21 folly::detail::function::call_<facebook::cachelib::NvmCache<facebook::cachelib::CacheAllocator<facebook::cachelib::LruCacheTrait> >::find(facebook::cachelib::HashedKey)::{lambda(facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::Buffer)#1}, true, false, void, facebook::cachelib::navy::Status, facebook::cachelib::HashedKey&&, facebook::cachelib::navy::Buffer&&>(facebook::cachelib::navy::Status, facebook::cachelib::HashedKey&&, facebook::cachelib::navy::Buffer&&, folly::detail::function::Data&) (p=...)
    at /root/workspace/kvcache/opt/cachelib/include/folly/Function.h:341
#22 0x00005555558e054b in folly::detail::function::FunctionTraits<void (facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::Buffer)>::operator()(facebook::cachelib::navy::Status, facebook::cachelib::HashedKey, facebook::cachelib::navy::Buffer) (
    args#2=..., args#1=..., args#0=<optimized out>, this=0x7ffff3b72e10) at /root/workspace/kvcache/opt/cachelib/include/folly/Function.h:370
#23 operator() (__closure=0x7ffff3b72e00) at /root/workspace/kvcache/cachelib/navy/engine/EnginePair.cpp:180
#24 0x0000555555879903 in folly::detail::function::FunctionTraits<facebook::cachelib::navy::JobExitCode ()>::operator()() (this=0x7fe710090f50)
    at /root/workspace/kvcache/opt/cachelib/include/folly/Function.h:370
#25 operator() (__closure=0x7fe710090f40) at /root/workspace/kvcache/cachelib/navy/scheduler/ThreadPoolJobScheduler.cpp:192
#26 folly::detail::function::call_<facebook::cachelib::navy::OrderedThreadPoolJobScheduler::scheduleJobLocked(JobParams, uint64_t)::<lambda()>, false, false, facebook::cachelib::navy::JobExitCode>(folly::detail::function::Data &) (p=...)
    at /root/workspace/kvcache/opt/cachelib/include/folly/Function.h:343
#27 0x000055555587d017 in folly::detail::function::FunctionTraits<facebook::cachelib::navy::JobExitCode ()>::operator()() (
    this=<optimized out>) at /root/workspace/kvcache/opt/cachelib/include/folly/Function.h:368
#28 operator() (__closure=<optimized out>) at /root/workspace/kvcache/cachelib/navy/scheduler/ThreadPoolJobQueue.cpp:158
#29 0x000055555587d5d3 in facebook::cachelib::navy::JobQueue::runJob (this=<optimized out>, entry=...)
    at /root/workspace/kvcache/cachelib/navy/scheduler/ThreadPoolJobQueue.cpp:170
#30 0x000055555587d8f3 in facebook::cachelib::navy::JobQueue::process (this=0x55555609d130)
    at /root/workspace/kvcache/cachelib/navy/scheduler/ThreadPoolJobQueue.cpp:91
#31 0x00007ffff6fb8db4 in std::execute_native_thread_routine (__p=0x555555be4f50) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:104
#32 0x00007ffff6c3fa94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
--Type <RET> for more, q to quit, c to continue without paging--
#33 0x00007ffff6cccc3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Desktop (please complete the following information):

  • OS: ubuntu 24.04
  • CacheLib commit: #eaa5ca93 (submodules are from #f3b5582)
@rainjuns
Copy link
Author

rainjuns commented Nov 7, 2024

Currently, I am not sure if this problem can be reproduced.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rainjuns