It seems to be deadlock when executing the script "benchmark.sh"

[FangleiLiu]

[BUG]

Expected behavior

The script executes normally and outputs performance results。

Actual behavior

It seems to be deadlock after 8 hours of script execution。

Steps to reproduce the behavior

dev.1.3  e3afb15140b64ae88bdb416413a09e123cbb2f78
I built terarkdb  with reference to wiki documents，then executed "NUM_KEYS=100000000 NUM_THREADS=64 CACHE_SIZE=137438953472  VALUE_SIZE=40960 ./benchmark.sh bulkload". I found it seems to be deadlock after 10 hours of script execution.

[Enhancement]

Problem

Pstack info:
Thread 24 (Thread 0x7f414137f700 (LWP 209502)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=0) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62e90) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f41405ff700 (LWP 209503)):
#0 CopyForward (len=50, op=0x7d32c7cf4344 "\uC0z5&2Xs]Ou$Q}'4WKhs(BMhC/_XDEl+^sCN6"jF^xB_EC#i$g.-11ShE!BnTT{nr2c5<TO=]3"!s|^vvfNssoBacxB_EC#i$g.-11ShE!BnTT{nr2c5<TO=]*3\"!s|^vvfNssoBacP$,!M{#!6Tw/$S"9qTp\F-in6B?otJ"XtHC+yN1Wh$<F}bw+nP$,!M{#"..., src=0x7d32c7cf4312 "\uC0z5&2Xs]Ou$Q}'4WKhs(BMhC/_XDEl+^sCN6"jF^ni;YV}\uC0z5&2Xs]Ou$Q}'4WKhs(BMhC/_XDEl+^sCN6"jF^xB_EC#i$g.-11ShE!BnTT{nr2c5<TO=]3"!s|^vvfNssoBacxB_EC#i$g.-11ShE!BnTT{nr2c5<TO=]*3\"!s|^vvfNssoBacP$,!M{#"...) at /data0/liufanglei/src/terarkdb/third-party/terark-zip/src/terark/zbs/dict_zip_blob_store.cpp:2585
#1 terark::DoUnzipSwitchPreserve<3> (pos=0x7d3308fd5893 "\370xB_EC#i$g.-11ShE!BnTT{nr2c5<TO=]\2103"!s|^vvfNssoBac\206\062", end=0x7d3308fda6a5 "uѧ\216XorUeb[pd\"=->1P7.P~dUf-$T!F^|]r0[R1O>P_k>3skiYYu^XorUeb[poU6GMw11gu"2SuoR$OZV{"ht3Yl29ZfgW3W3Q#d--SWbHk[wvoU6GMw11gu"2SuoR$OZV{"ht3Yl29ZfgW3W3Q#d--SWbHk[wv9~>8a>0HYCqy6Fu/=w%kr}$aG}|4Gm`-u)>""..., recData=0x7f412bd63750, dic=, gOffsetBits=, reserveOutputMultiplier=) at /data0/liufanglei/src/terarkdb/third-party/terark-zip/src/terark/zbs/dict_zip_blob_store_unzip_func.hpp:204
#2 0x0000000000a49155 in read_record_append_tpl<false, 2, (terark::DictZipBlobStore::Options::EntropyAlgo)0, 0, terark::DictZipBlobStore::fspread_record_append_tpl(terark::BlobStore::pread_func_t, void, size_t, size_t, terark::valvec, terark::valvec) const [with bool ZipOffset = false; int CheckSumLevel = 2; terark::DictZipBlobStore::Options::EntropyAlgo Entropy = (terark::DictZipBlobStore::Options::EntropyAlgo)0; int EntropyInterLeave = 0]::<lambda(size_t, size_t)> > (readRaw=..., recData=0x7f412bd63750, recId=418, this=0x7f166cd7fd80) at /data0/liufanglei/src/terarkdb/third-party/terark-zip/src/terark/zbs/dict_zip_blob_store.cpp:2895
#3 terark::DictZipBlobStore::fspread_record_append_tpl<false, 2, (terark::DictZipBlobStore::Options::EntropyAlgo)0, 0> (this=0x7f166cd7fd80, fspread=, lambda=, baseOffset=, recID=418, recData=0x7f412bd63750, rdbuf=0x7f41405f9cf0) at /data0/liufanglei/src/terarkdb/third-party/terark-zip/src/terark/zbs/dict_zip_blob_store.cpp:2686
#4 0x0000000000a2cad0 in fspread_record_append (rdbuf=0x7f41405f9cf0, recData=0x7f412bd63750, recID=418, baseOffset=50376, lambda=0x7f062a7d0e78, fspread=0x7aff20 <rocksdb::FsPread(void*, size_t, size_t, terark::valvec)>, this=) at /data0/liufanglei/src/terarkdb/third-party/terark-zip/src/terark/zbs/blob_store.hpp:127
#5 terark::BlobStore::fspread_record_append (this=, fspread=0x7aff20 <rocksdb::FsPread(void, size_t, size_t, terark::valvec)>, lambda=0x7f062a7d0e78, baseOffset=50376, recID=418, recData=0x7f412bd63750) at /data0/liufanglei/src/terarkdb/third-party/terark-zip/src/terark/zbs/blob_store.cpp:120
#6 0x00000000007bbd16 in rocksdb::TerarkZipTableIterator::UnzipIterRecord (this=0x7f412bd63660, hasRecord=) at /data0/liufanglei/src/terarkdb/table/terark_zip_table_reader.cc:545
#7 0x00000000007bc4c5 in rocksdb::TerarkZipTableIterator::SeekInternal (this=0x7f412bd63660, seek_key=..., seek_tag=0) at /data0/liufanglei/src/terarkdb/table/terark_zip_table_reader.cc:471
#8 0x00000000007bc621 in rocksdb::TerarkZipTableIterator::Seek (this=0x7f412bd63660, target=...) at /data0/liufanglei/src/terarkdb/table/terark_zip_table_reader.cc:355
#9 0x0000000000743afd in Seek (k=..., this=0x7d33085c5de0) at /data0/liufanglei/src/terarkdb/table/iterator_wrapper.h:72
#10 rocksdb::MergingIterator::Seek (this=0x7f41405fa600, target=...) at /data0/liufanglei/src/terarkdb/table/merging_iterator.cc:134
#11 0x00000000006284d0 in Seek (k=..., this=0x7f41405fa370) at /data0/liufanglei/src/terarkdb/table/iterator_wrapper.h:183
#12 rocksdb::(anonymous namespace)::AdjustRange (ic=ic@entry=0x7f4141a83528, iter=iter@entry=0x7f41405fa370, arena=0x7f41405fa5f0, largest_key=..., ranges=std::vector of length 41913, capacity 65536 = {...}) at /data0/liufanglei/src/terarkdb/db/map_builder.cc:633
#13 0x0000000000631c61 in rocksdb::MapBuilder::Build (this=this@entry=0x7f41405fb0e0, inputs=std::vector of length 2, capacity 2 = {...}, push_range=std::vector of length 1, capacity 1 = {...}, output_level=5, output_path_id=0, cfd=0x7f4141a83500, version=0x7f412e780000, edit=0x7f412b672308, output=0x7f41405fb020) at /data0/liufanglei/src/terarkdb/db/map_builder.cc:1464
#14 0x00000000007dd30b in rocksdb::CompactionJob::InstallCompactionResults (this=this@entry=0x7f41405fbc30, mutable_cf_options=...) at /data0/liufanglei/src/terarkdb/db/compaction_job.cc:2343
#15 0x00000000007deb4d in rocksdb::CompactionJob::Install (this=this@entry=0x7f41405fbc30, mutable_cf_options=...) at /data0/liufanglei/src/terarkdb/db/compaction_job.cc:1083
#16 0x00000000005aefec in rocksdb::DBImpl::BackgroundCompaction (this=this@entry=0x7f4141ae5400, made_progress=made_progress@entry=0x7f41405fc086, job_context=job_context@entry=0x7f41405fc0a0, log_buffer=log_buffer@entry=0x7f41405fc2c0, prepicked_compaction=prepicked_compaction@entry=0x7f3f89634e30) at /data0/liufanglei/src/terarkdb/db/db_impl_compaction_flush.cc:2819
#17 0x00000000005b4ad9 in rocksdb::DBImpl::BackgroundCallCompaction (this=this@entry=0x7f4141ae5400, prepicked_compaction=prepicked_compaction@entry=0x7f3f89634e30, bg_thread_pri=bg_thread_pri@entry=rocksdb::Env::LOW) at /data0/liufanglei/src/terarkdb/db/db_impl_compaction_flush.cc:2300
#18 0x00000000005b505f in rocksdb::DBImpl::BGWorkCompaction (arg=) at /data0/liufanglei/src/terarkdb/db/db_impl_compaction_flush.cc:2052
#19 0x00000000007897bd in operator() (this=0x7f41405fcc30) at /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/std_function.h:706
#20 rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=1) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:252
#21 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62ea0) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#22 0x0000000000d0225f in execute_native_thread_routine ()
#23 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f413fdfe700 (LWP 209504)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=2) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62ec0) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f413f1ff700 (LWP 209505)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=3) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62eb0) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f413e7fe700 (LWP 209506)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=4) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62ed0) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f413d1ff700 (LWP 209507)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=5) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62ee0) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f413c5fe700 (LWP 209508)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=6) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62ef0) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f413b9ff700 (LWP 209509)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=7) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62f00) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f413affe700 (LWP 209510)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=8) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62f10) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f413a3ff700 (LWP 209511)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=9) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62f20) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f4139bfe700 (LWP 209512)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=10) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62f30) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f4138fff700 (LWP 209513)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=11) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62f40) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f41383fe700 (LWP 209514)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=12) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62f50) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f41377ff700 (LWP 209515)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=13) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62f60) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f4136bff700 (LWP 209516)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a460e0, thread_id=thread_id@entry=14) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62f70) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f41361fe700 (LWP 209517)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a461c0, thread_id=thread_id@entry=0) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62f80) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f41355ff700 (LWP 209518)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a461c0, thread_id=thread_id@entry=1) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62f90) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f41349fe700 (LWP 209519)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a461c0, thread_id=thread_id@entry=2) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62fb0) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f4133dff700 (LWP 209520)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a461c0, thread_id=thread_id@entry=3) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62fa0) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f41331ff700 (LWP 209521)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f414286eaec in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /lib64/libstdc++.so.6
#2 0x0000000000789609 in rocksdb::ThreadPoolImpl::Impl::BGThread (this=this@entry=0x7f4141a461c0, thread_id=thread_id@entry=4) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:185
#3 0x0000000000789954 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x7f4141a62fc0) at /data0/liufanglei/src/terarkdb/util/threadpool_imp.cc:290
#4 0x0000000000d0225f in execute_native_thread_routine ()
#5 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f41305ff700 (LWP 209522)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000527edd in rocksdb::port::CondVar::Wait (this=this@entry=0x7f4141b051d0) at /data0/liufanglei/src/terarkdb/port/port_posix.cc:91
#2 0x00000000006c5235 in rocksdb::InstrumentedCondVar::WaitInternal (this=this@entry=0x7f4141b051d0) at /data0/liufanglei/src/terarkdb/monitoring/instrumented_mutex.cc:116
#3 0x00000000006c5314 in rocksdb::InstrumentedCondVar::Wait (this=this@entry=0x7f4141b051d0) at /data0/liufanglei/src/terarkdb/monitoring/instrumented_mutex.cc:86
#4 0x000000000077a308 in rocksdb::DeleteScheduler::BackgroundEmptyTrash (this=0x7f4141b050e0) at /data0/liufanglei/src/terarkdb/util/delete_scheduler.cc:226
#5 0x0000000000d0225f in execute_native_thread_routine ()
#6 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f4107d45700 (LWP 209554)):
#0 0x00007f4143304d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000527f42 in rocksdb::port::CondVar::TimedWait (this=this@entry=0x7f4141a682a8, abs_time_us=abs_time_us@entry=1610069578258914) at /data0/liufanglei/src/terarkdb/port/port_posix.cc:105
#2 0x000000000058e383 in wait (delay=, this=0x7f4141a68240) at /data0/liufanglei/src/terarkdb/util/repeatable_thread.h:93
#3 thread (this=0x7f4141a68240) at /data0/liufanglei/src/terarkdb/util/repeatable_thread.h:130
#4 operator() (__closure=) at /data0/liufanglei/src/terarkdb/util/repeatable_thread.h:34
#5 __invoke_impl<void, rocksdb::RepeatableThread::RepeatableThread(std::function<void()>, const string&, rocksdb::Env, uint64_t, uint64_t)::<lambda()> > (__f=) at /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/invoke.h:60
#6 __invoke<rocksdb::RepeatableThread::RepeatableThread(std::function<void()>, const string&, rocksdb::Env*, uint64_t, uint64_t)::<lambda()> > (__fn=) at /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/invoke.h:95
#7 _M_invoke<0> (this=) at /opt/rh/devtoolset-7/root/usr/include/c++/7/thread:234
#8 operator() (this=) at /opt/rh/devtoolset-7/root/usr/include/c++/7/thread:243
#9 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN7rocksdb16RepeatableThreadC4ESt8functionIFvvEERKSsPNS3_3EnvEmmEUlvE_EEEEE6_M_runEv (this=) at /opt/rh/devtoolset-7/root/usr/include/c++/7/thread:186
#10 0x0000000000d0225f in execute_native_thread_routine ()
#11 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f410c7ff700 (LWP 209555)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000527edd in rocksdb::port::CondVar::Wait (this=this@entry=0x7f4141ae5930) at /data0/liufanglei/src/terarkdb/port/port_posix.cc:91
#2 0x00000000006c5235 in rocksdb::InstrumentedCondVar::WaitInternal (this=this@entry=0x7f4141ae5930) at /data0/liufanglei/src/terarkdb/monitoring/instrumented_mutex.cc:116
#3 0x00000000006c5314 in rocksdb::InstrumentedCondVar::Wait (this=this@entry=0x7f4141ae5930) at /data0/liufanglei/src/terarkdb/monitoring/instrumented_mutex.cc:86
#4 0x00000000005a6b71 in rocksdb::DBImpl::RunManualCompaction (this=this@entry=0x7f4141ae5400, cfd=cfd@entry=0x7f4141a83500, input_level=input_level@entry=0, output_level=output_level@entry=-2, output_path_id=, max_subcompactions=0, begin=0x0, end=0x0, files_being_compact=0x7f410c7fca50, exclusive=true, disallow_trivial_move=false) at /data0/liufanglei/src/terarkdb/db/db_impl_compaction_flush.cc:1462
#5 0x00000000005ab006 in rocksdb::DBImpl::CompactRange (this=0x7f4141ae5400, options=..., column_family=, begin=0x0, end=0x0) at /data0/liufanglei/src/terarkdb/db/db_impl_compaction_flush.cc:735
#6 0x0000000000466e3d in rocksdb::DB::CompactRange (this=0x7f4141ae5400, options=..., begin=0x0, end=0x0) at /data0/liufanglei/src/terarkdb/include/rocksdb/db.h:857
#7 0x000000000046cf9e in rocksdb::Benchmark::Compact (this=0x7ffed2238520, thread=) at /data0/liufanglei/src/terarkdb/tools/db_bench_tool.cc:5695
#8 0x0000000000473394 in rocksdb::Benchmark::ThreadBody (v=0x7ed85a728890) at /data0/liufanglei/src/terarkdb/tools/db_bench_tool.cc:2898
#9 0x00000000005283e2 in rocksdb::(anonymous namespace)::StartThreadWrapper (arg=0x7f4141a62e60) at /data0/liufanglei/src/terarkdb/env/env_posix.cc:1069
#10 0x00007f4143300dd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f4141fd5ead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f4143718980 (LWP 209500)):
#0 0x00007f4143304965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000527edd in rocksdb::port::CondVar::Wait (this=this@entry=0x7ffed2237cc8) at /data0/liufanglei/src/terarkdb/port/port_posix.cc:91
#2 0x0000000000482eb8 in rocksdb::Benchmark::RunBenchmark (this=this@entry=0x7ffed2238520, n=n@entry=1, name=..., method=(void (rocksdb::Benchmark::*)(rocksdb::Benchmark * const, rocksdb::ThreadState *)) 0x46cf10 rocksdb::Benchmark::Compact(rocksdb::ThreadState*)) at /data0/liufanglei/src/terarkdb/tools/db_bench_tool.cc:2970
#3 0x0000000000489822 in rocksdb::Benchmark::Run (this=this@entry=0x7ffed2238520) at /data0/liufanglei/src/terarkdb/tools/db_bench_tool.cc:2838
#4 0x0000000000462361 in rocksdb::db_bench_tool (argc=, argv=) at /data0/liufanglei/src/terarkdb/tools/db_bench_tool.cc:5871
#5 0x00007f4141efa3d5 in __libc_start_main () from /lib64/libc.so.6
#6 0x0000000000437637 in _start ()

CMD info:
liufang+ 209500 100 2.1 2239499132 16795548 pts/1 Sl+ 00:42 521:15 ./db_bench --benchmarks=compact --use_existing_db=1 --disable_auto_compactions=1 --sync=0 --db=/data4/liufl/terarkdb/terarkdb_test/db_test --wal_dir=/data4/liufl/terarkdb/terarkdb_test/wal_test --num=100000000 --num_levels=6 --key_size=20 --value_size=40960 --block_size=8192 --cache_size=137438953472 --cache_numshardbits=6 --compression_max_dict_bytes=0 --compression_ratio=0.5 --compression_type=snappy --level_compaction_dynamic_level_bytes=true --bytes_per_sync=8388608 --cache_index_and_filter_blocks=0 --pin_l0_filter_and_index_blocks_in_cache=1 --benchmark_write_rate_limit=0 --hard_rate_limit=3 --rate_limit_delay_max_milliseconds=1000000 --write_buffer_size=134217728 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --verify_checksum=1 --delete_obsolete_files_period_micros=62914560 --max_bytes_for_level_multiplier=8 --statistics=0 --stats_per_interval=1 --stats_interval_seconds=60 --histogram=1 --memtablerep=skip_list --bloom_bits=10 --open_files=-1 --level0_file_num_compaction_trigger=4 --level0_slowdown_writes_trigger=12 --level0_stop_writes_trigger=20 --max_background_jobs=20 --max_write_buffer_number=8 --threads=1

Disk Info:
/dev/nvme4n1 3.6T 2.1T 1.4T 61% /data4

Memory Info:
total used free shared buff/cache available
Mem: 754G 21G 14G 27M 718G 730G
Swap: 4.0G 169M 3.8G

Machine Info:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
座： 2
NUMA 节点： 2
厂商 ID： GenuineIntel
CPU 系列： 6
型号： 85
型号名称： Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
步进： 4
CPU MHz： 999.902
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS： 4800.00
虚拟化： VT-x
L1d 缓存： 32K
L1i 缓存： 32K
L2 缓存： 1024K
L3 缓存： 28160K
NUMA 节点0 CPU： 0-19,40-59
NUMA 节点1 CPU： 20-39,60-79

Solution

[FangleiLiu]

The Issue should be caused by memory exhaustion，such as free memory is 14G in the "memory info"。
At first, I only focused on available memory in the "memory info".
It is speculated that too many files were opened，causing the buffer/cache to be occupied。
Maybe it's related to my optionsetting. I'll try to catch up with the issue and close it temporarily.