Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Infamous memory usage #761

Open
3 tasks done
deajan opened this issue Oct 26, 2024 · 0 comments
Open
3 tasks done

Infamous memory usage #761

deajan opened this issue Oct 26, 2024 · 0 comments
Assignees
@kdeldycke
Labels
🐛 bug Something isn't working, or a fix is proposed

Comments

@deajan
Copy link

deajan commented Oct 26, 2024 •
edited
Loading

Preliminary checks

Describe the bug

So I already read about the OOM killing mdedup because if high memory usage, but I'm quite a bit puzzled here.
I ran the following on a set of 590k emails:

/mnt/venv/bin/mdedup -b raw --action move-selected --strategy select-biggest --export /mnt/DEDUP /mnt/EML /mnt/EML-OST /mnt/EML-PST

Once about 23k emails were done, mdedup had a memory usage of 42GB, which is roughly 1,8MB per email ! At that pace, I'd need 1TB to dedup my emails.
This number climbed almost proportionnaly, which makes me think that, since I used the -b raw parameter. mdedup does not only keep paths and corresponding sets of hashes in memory, but full emails.

I've done another test without -b switch, and this time I got about 15GB for 23k emails which still is a very high number. At that pace I'd need about 400GB for my emails to be deduped.

I've made a quick calculation, keeping 10 SHA256 hashes (32 bytes) in memory per email would require about 180MB of data (regardless of any data representation) plus let's say 1KB per filename which would require another whopping 576MB, for a total of less than a GB of data.
Now I can imagine that the dedup table would take some memory, but 1TB of memory ? Even ZFS isn't that hungry.

This program is highly memory inefficient, since I think it keeps way more than just sets of hashes. Does it keep the whole in memory representation of an email object per email ?

Are my findings normal or off the charts ?

I've had earlier issues with high memory consumption with lots of json data.
I'd advise to have a look at msgspec library which saved me about 40% memory usage on a json dict with millions of entries.
Anyway, I'd advise you to only keep hashes in memory instead of whatever class object repr you're keeping.

Additional context

mdedup output just before I had to stop it:

Hashed mails  [#-----------------------------------]  23756/596456  04:35:21

Memory usage of mdedup

ps -eo size,pid,user,command --sort -size |     awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |    cut -d "" -f2 | cut -d "-" -f1 | head

     42956.84 Mb /mnt/venv/bin/python /mnt/venv/bin/mdedup
     17007.59 Mb /usr/libexec/qemu
     13616.19 Mb /usr/libexec/qemu
     10567.80 Mb /usr/libexec/qemu
      9330.69 Mb /usr/libexec/qemu
      8982.71 Mb /usr/libexec/qemu
      8976.46 Mb /usr/libexec/qemu
      8935.57 Mb /usr/libexec/qemu
      8746.17 Mb /usr/libexec/qemu

Memory map of mdedup

pmap $(ps aux | grep [m]dedup | awk '{print $2}')                                                                                        13073:   /mnt/venv/bin/python /mnt/venv/bin/mdedup -b raw --action move-selected --strategy select-biggest --export /mnt/DEDUP /mnt/EML /mnt/EML-OST /mnt/EML-PST
00005609cf128000      4K r---- python3.9
00005609cf129000      4K r-x-- python3.9
00005609cf12a000      4K r---- python3.9
00005609cf12b000      4K r---- python3.9
00005609cf12c000      4K rw--- python3.9
00005609d0564000 18028940K rw---   [ anon ]
00007fe4b0f36000    256K rw---   [ anon ]
00007fe4b19f6000 3828736K rw---   [ anon ]
00007fe59b4ff000 9425956K rw---   [ anon ]
00007fe7daa0d000 13230060K rw---   [ anon ]
00007feb0223f000 935788K rw---   [ anon ]
00007feb3b41b000   8704K rw---   [ anon ]
00007feb3bc9c000   5120K rw---   [ anon ]
00007feb3c19d000   2560K rw---   [ anon ]
00007feb3c45c000   2816K rw---   [ anon ]
00007feb3c729000   2816K rw---   [ anon ]
00007feb3ca0a000   2048K rw---   [ anon ]
00007feb3cc44000   2048K rw---   [ anon ]
00007feb3ce5b000   1792K rw---   [ anon ]
00007feb3d053000   1536K rw---   [ anon ]
00007feb3d1ef000   1536K rw---   [ anon ]
00007feb3d373000   1792K rw---   [ anon ]
00007feb3d561000   9216K rw---   [ anon ]
00007feb3de61000     12K r---- libgcc_s-11-20231218.so.1
00007feb3de64000     72K r-x-- libgcc_s-11-20231218.so.1
00007feb3de76000     12K r---- libgcc_s-11-20231218.so.1
00007feb3de79000      4K ----- libgcc_s-11-20231218.so.1
00007feb3de7a000      4K r---- libgcc_s-11-20231218.so.1
00007feb3de7b000      4K rw--- libgcc_s-11-20231218.so.1
00007feb3de7c000     20K r---- libnss_myhostname.so.2
00007feb3de81000     68K r-x-- libnss_myhostname.so.2
00007feb3de92000     44K r---- libnss_myhostname.so.2
00007feb3de9d000      4K ----- libnss_myhostname.so.2
00007feb3de9e000     16K r---- libnss_myhostname.so.2
00007feb3dea2000      4K rw--- libnss_myhostname.so.2
00007feb3dead000   1024K rw---   [ anon ]
00007feb3dfad000     24K r---- _decimal.cpython-39-x86_64-linux-gnu.so
00007feb3dfb3000    172K r-x-- _decimal.cpython-39-x86_64-linux-gnu.so
00007feb3dfde000     64K r---- _decimal.cpython-39-x86_64-linux-gnu.so
00007feb3dfee000      4K r---- _decimal.cpython-39-x86_64-linux-gnu.so
00007feb3dfef000     12K rw--- _decimal.cpython-39-x86_64-linux-gnu.so
00007feb3dff2000   1536K rw---   [ anon ]
00007feb3e172000     20K r---- _pickle.cpython-39-x86_64-linux-gnu.so
00007feb3e177000     64K r-x-- _pickle.cpython-39-x86_64-linux-gnu.so
00007feb3e187000     24K r---- _pickle.cpython-39-x86_64-linux-gnu.so
00007feb3e18d000      4K ----- _pickle.cpython-39-x86_64-linux-gnu.so
00007feb3e18e000      4K r---- _pickle.cpython-39-x86_64-linux-gnu.so
00007feb3e18f000      8K rw--- _pickle.cpython-39-x86_64-linux-gnu.so
00007feb3e191000    256K rw---   [ anon ]
00007feb3e1d1000     60K r---- libsqlite3.so.0.8.6
00007feb3e1e0000    964K r-x-- libsqlite3.so.0.8.6
00007feb3e2d1000    196K r---- libsqlite3.so.0.8.6
00007feb3e302000     16K r---- libsqlite3.so.0.8.6
00007feb3e306000     12K rw--- libsqlite3.so.0.8.6
00007feb3e309000     24K r---- _sqlite3.cpython-39-x86_64-linux-gnu.so
00007feb3e30f000     40K r-x-- _sqlite3.cpython-39-x86_64-linux-gnu.so
00007feb3e319000     16K r---- _sqlite3.cpython-39-x86_64-linux-gnu.so
00007feb3e31d000      4K ----- _sqlite3.cpython-39-x86_64-linux-gnu.so
00007feb3e31e000      4K r---- _sqlite3.cpython-39-x86_64-linux-gnu.so
00007feb3e31f000      8K rw--- _sqlite3.cpython-39-x86_64-linux-gnu.so
00007feb3e321000    256K rw---   [ anon ]
00007feb3e361000     88K r---- libreadline.so.8.1
00007feb3e377000    184K r-x-- libreadline.so.8.1
00007feb3e3a5000     40K r---- libreadline.so.8.1
00007feb3e3af000     12K r---- libreadline.so.8.1
00007feb3e3b2000     24K rw--- libreadline.so.8.1
00007feb3e3b8000   1800K rw---   [ anon ]
00007feb3e57a000     20K r---- _yaml.cpython-39-x86_64-linux-gnu.so
00007feb3e57f000    308K r-x-- _yaml.cpython-39-x86_64-linux-gnu.so
00007feb3e5cc000     36K r---- _yaml.cpython-39-x86_64-linux-gnu.so
00007feb3e5d5000      4K r---- _yaml.cpython-39-x86_64-linux-gnu.so
00007feb3e5d6000      8K rw--- _yaml.cpython-39-x86_64-linux-gnu.so
00007feb3e5d8000    516K rw---   [ anon ]
00007feb3e659000     16K r---- libexpat.so.1.8.10
00007feb3e65d000    124K r-x-- libexpat.so.1.8.10
00007feb3e67c000     40K r---- libexpat.so.1.8.10
00007feb3e686000      4K ----- libexpat.so.1.8.10
00007feb3e687000      8K r---- libexpat.so.1.8.10
00007feb3e689000      4K rw---   [ anon ]
00007feb3e68d000      4K r---- fcntl.cpython-39-x86_64-linux-gnu.so
00007feb3e68e000      8K r-x-- fcntl.cpython-39-x86_64-linux-gnu.so
00007feb3e690000      8K r---- fcntl.cpython-39-x86_64-linux-gnu.so
00007feb3e692000      4K r---- fcntl.cpython-39-x86_64-linux-gnu.so
00007feb3e693000      4K rw--- fcntl.cpython-39-x86_64-linux-gnu.so
00007feb3e694000     16K r---- pyexpat.cpython-39-x86_64-linux-gnu.so
00007feb3e698000     28K r-x-- pyexpat.cpython-39-x86_64-linux-gnu.so
00007feb3e69f000     12K r---- pyexpat.cpython-39-x86_64-linux-gnu.so
00007feb3e6a2000      4K ----- pyexpat.cpython-39-x86_64-linux-gnu.so
00007feb3e6a3000      4K r---- pyexpat.cpython-39-x86_64-linux-gnu.so
00007feb3e6a4000      8K rw--- pyexpat.cpython-39-x86_64-linux-gnu.so
00007feb3e6a6000   1280K rw---   [ anon ]
00007feb3e7e6000     12K r---- _multibytecodec.cpython-39-x86_64-linux-gnu.so
00007feb3e7e9000     20K r-x-- _multibytecodec.cpython-39-x86_64-linux-gnu.so
00007feb3e7ee000      8K r---- _multibytecodec.cpython-39-x86_64-linux-gnu.so
00007feb3e7f0000      4K r---- _multibytecodec.cpython-39-x86_64-linux-gnu.so
00007feb3e7f1000      4K rw--- _multibytecodec.cpython-39-x86_64-linux-gnu.so
00007feb3e7f2000     52K r---- md__mypyc.cpython-39-x86_64-linux-gnu.so
00007feb3e7ff000    120K r-x-- md__mypyc.cpython-39-x86_64-linux-gnu.so
00007feb3e81d000     36K r---- md__mypyc.cpython-39-x86_64-linux-gnu.so
00007feb3e826000      4K ----- md__mypyc.cpython-39-x86_64-linux-gnu.so
00007feb3e827000      4K r---- md__mypyc.cpython-39-x86_64-linux-gnu.so
00007feb3e828000     20K rw--- md__mypyc.cpython-39-x86_64-linux-gnu.so
00007feb3e82d000      4K r---- md.cpython-39-x86_64-linux-gnu.so
00007feb3e82e000      4K r-x-- md.cpython-39-x86_64-linux-gnu.so
00007feb3e82f000      4K r---- md.cpython-39-x86_64-linux-gnu.so
00007feb3e830000      4K r---- md.cpython-39-x86_64-linux-gnu.so
00007feb3e831000      4K rw--- md.cpython-39-x86_64-linux-gnu.so
00007feb3e832000    768K rw---   [ anon ]
00007feb3e8f2000      8K r---- _csv.cpython-39-x86_64-linux-gnu.so
00007feb3e8f4000     12K r-x-- _csv.cpython-39-x86_64-linux-gnu.so
00007feb3e8f7000      8K r---- _csv.cpython-39-x86_64-linux-gnu.so
00007feb3e8f9000      4K ----- _csv.cpython-39-x86_64-linux-gnu.so
00007feb3e8fa000      4K r---- _csv.cpython-39-x86_64-linux-gnu.so
00007feb3e8fb000      4K rw--- _csv.cpython-39-x86_64-linux-gnu.so
00007feb3e8fc000   2560K rw---   [ anon ]
00007feb3eb7c000      8K r---- _json.cpython-39-x86_64-linux-gnu.so
00007feb3eb7e000     28K r-x-- _json.cpython-39-x86_64-linux-gnu.so
00007feb3eb85000      8K r---- _json.cpython-39-x86_64-linux-gnu.so
00007feb3eb87000      4K r---- _json.cpython-39-x86_64-linux-gnu.so
00007feb3eb88000      4K rw--- _json.cpython-39-x86_64-linux-gnu.so
00007feb3eb89000   1024K rw---   [ anon ]
00007feb3ec89000     16K r---- _asyncio.cpython-39-x86_64-linux-gnu.so
00007feb3ec8d000     20K r-x-- _asyncio.cpython-39-x86_64-linux-gnu.so
00007feb3ec92000     16K r---- _asyncio.cpython-39-x86_64-linux-gnu.so
00007feb3ec96000      4K r---- _asyncio.cpython-39-x86_64-linux-gnu.so
00007feb3ec97000      8K rw--- _asyncio.cpython-39-x86_64-linux-gnu.so
00007feb3ec99000    256K rw---   [ anon ]
00007feb3ecd9000      8K r---- binascii.cpython-39-x86_64-linux-gnu.so
00007feb3ecdb000     16K r-x-- binascii.cpython-39-x86_64-linux-gnu.so
00007feb3ecdf000      8K r---- binascii.cpython-39-x86_64-linux-gnu.so
00007feb3ece1000      4K ----- binascii.cpython-39-x86_64-linux-gnu.so
00007feb3ece2000      4K r---- binascii.cpython-39-x86_64-linux-gnu.so
00007feb3ece3000      4K rw--- binascii.cpython-39-x86_64-linux-gnu.so
00007feb3ece4000    256K rw---   [ anon ]
00007feb3ed24000    120K r---- libssl.so.3.0.7
00007feb3ed42000    372K r-x-- libssl.so.3.0.7
00007feb3ed9f000    116K r---- libssl.so.3.0.7
00007feb3edbc000     40K r---- libssl.so.3.0.7
00007feb3edc6000     16K rw--- libssl.so.3.0.7
00007feb3edca000     72K r---- _ssl.cpython-39-x86_64-linux-gnu.so
00007feb3eddc000     44K r-x-- _ssl.cpython-39-x86_64-linux-gnu.so
00007feb3ede7000     52K r---- _ssl.cpython-39-x86_64-linux-gnu.so
00007feb3edf4000      4K ----- _ssl.cpython-39-x86_64-linux-gnu.so
00007feb3edf5000      4K r---- _ssl.cpython-39-x86_64-linux-gnu.so
00007feb3edf6000     36K rw--- _ssl.cpython-39-x86_64-linux-gnu.so
00007feb3edff000   1792K rw---   [ anon ]
00007feb3efbf000     12K r---- _struct.cpython-39-x86_64-linux-gnu.so
00007feb3efc2000     20K r-x-- _struct.cpython-39-x86_64-linux-gnu.so
00007feb3efc7000     16K r---- _struct.cpython-39-x86_64-linux-gnu.so
00007feb3efcb000      4K r---- _struct.cpython-39-x86_64-linux-gnu.so
00007feb3efcc000      4K rw--- _struct.cpython-39-x86_64-linux-gnu.so
00007feb3efcd000      8K r---- libuuid.so.1.3.0
00007feb3efcf000     20K r-x-- libuuid.so.1.3.0
00007feb3efd4000      4K r---- libuuid.so.1.3.0
00007feb3efd5000      4K r---- libuuid.so.1.3.0
00007feb3efd6000      4K rw---   [ anon ]
00007feb3efdb000      8K r---- _queue.cpython-39-x86_64-linux-gnu.so
00007feb3efdd000      4K r-x-- _queue.cpython-39-x86_64-linux-gnu.so
00007feb3efde000      4K r---- _queue.cpython-39-x86_64-linux-gnu.so
00007feb3efdf000      4K r---- _queue.cpython-39-x86_64-linux-gnu.so
00007feb3efe0000      4K rw--- _queue.cpython-39-x86_64-linux-gnu.so
00007feb3efe4000     56K r---- libtinfo.so.6.2
00007feb3eff2000     60K r-x-- libtinfo.so.6.2
00007feb3f001000     56K r---- libtinfo.so.6.2
00007feb3f00f000     16K r---- libtinfo.so.6.2
00007feb3f013000      4K rw--- libtinfo.so.6.2
00007feb3f018000      8K r---- resource.cpython-39-x86_64-linux-gnu.so
00007feb3f01a000      4K r-x-- resource.cpython-39-x86_64-linux-gnu.so
00007feb3f01b000      4K r---- resource.cpython-39-x86_64-linux-gnu.so
00007feb3f01c000      4K r---- resource.cpython-39-x86_64-linux-gnu.so
00007feb3f01d000      4K rw--- resource.cpython-39-x86_64-linux-gnu.so
00007feb3f01e000     12K r---- readline.cpython-39-x86_64-linux-gnu.so
00007feb3f021000     12K r-x-- readline.cpython-39-x86_64-linux-gnu.so
00007feb3f024000      8K r---- readline.cpython-39-x86_64-linux-gnu.so
00007feb3f026000      4K ----- readline.cpython-39-x86_64-linux-gnu.so
00007feb3f027000      4K r---- readline.cpython-39-x86_64-linux-gnu.so
00007feb3f028000      4K rw--- readline.cpython-39-x86_64-linux-gnu.so
00007feb3f029000     12K r---- termios.cpython-39-x86_64-linux-gnu.so
00007feb3f02c000      4K r-x-- termios.cpython-39-x86_64-linux-gnu.so
00007feb3f02d000      8K r---- termios.cpython-39-x86_64-linux-gnu.so
00007feb3f02f000      4K r---- termios.cpython-39-x86_64-linux-gnu.so
00007feb3f030000      8K rw--- termios.cpython-39-x86_64-linux-gnu.so
00007feb3f032000    768K rw---   [ anon ]
00007feb3f0f2000      4K r---- libpthread.so.0
00007feb3f0f3000      4K r-x-- libpthread.so.0
00007feb3f0f4000      4K r---- libpthread.so.0
00007feb3f0f5000      4K r---- libpthread.so.0
00007feb3f0f6000      4K rw--- libpthread.so.0
00007feb3f0f7000      4K r---- _contextvars.cpython-39-x86_64-linux-gnu.so
00007feb3f0f8000      4K r-x-- _contextvars.cpython-39-x86_64-linux-gnu.so
00007feb3f0f9000      4K r---- _contextvars.cpython-39-x86_64-linux-gnu.so
00007feb3f0fa000      4K r---- _contextvars.cpython-39-x86_64-linux-gnu.so
00007feb3f0fb000      4K rw--- _contextvars.cpython-39-x86_64-linux-gnu.so
00007feb3f0fc000      4K r---- _uuid.cpython-39-x86_64-linux-gnu.so
00007feb3f0fd000      4K r-x-- _uuid.cpython-39-x86_64-linux-gnu.so
00007feb3f0fe000      4K r---- _uuid.cpython-39-x86_64-linux-gnu.so
00007feb3f0ff000      4K r---- _uuid.cpython-39-x86_64-linux-gnu.so
00007feb3f100000      4K rw--- _uuid.cpython-39-x86_64-linux-gnu.so
00007feb3f101000     64K r---- _regex.cpython-39-x86_64-linux-gnu.so
00007feb3f111000    220K r-x-- _regex.cpython-39-x86_64-linux-gnu.so
00007feb3f148000    512K r---- _regex.cpython-39-x86_64-linux-gnu.so
00007feb3f1c8000      4K r---- _regex.cpython-39-x86_64-linux-gnu.so
00007feb3f1c9000     36K rw--- _regex.cpython-39-x86_64-linux-gnu.so
00007feb3f1d2000     12K r---- unicodedata.cpython-39-x86_64-linux-gnu.so
00007feb3f1d5000     16K r-x-- unicodedata.cpython-39-x86_64-linux-gnu.so
00007feb3f1d9000   1056K r---- unicodedata.cpython-39-x86_64-linux-gnu.so
00007feb3f2e1000      4K r---- unicodedata.cpython-39-x86_64-linux-gnu.so
00007feb3f2e2000      4K rw--- unicodedata.cpython-39-x86_64-linux-gnu.so
00007feb3f2e3000    768K rw---   [ anon ]
00007feb3f3a3000     12K r---- liblzma.so.5.2.5
00007feb3f3a6000    108K r-x-- liblzma.so.5.2.5
00007feb3f3c1000     44K r---- liblzma.so.5.2.5
00007feb3f3cc000      4K ----- liblzma.so.5.2.5
00007feb3f3cd000      4K r---- liblzma.so.5.2.5
00007feb3f3ce000      4K rw---   [ anon ]
00007feb3f3cf000     12K r---- _lzma.cpython-39-x86_64-linux-gnu.so
00007feb3f3d2000     16K r-x-- _lzma.cpython-39-x86_64-linux-gnu.so
00007feb3f3d6000     12K r---- _lzma.cpython-39-x86_64-linux-gnu.so
00007feb3f3d9000      4K r---- _lzma.cpython-39-x86_64-linux-gnu.so
00007feb3f3da000      4K rw--- _lzma.cpython-39-x86_64-linux-gnu.so
00007feb3f3db000      8K r---- libbz2.so.1.0.8
00007feb3f3dd000     52K r-x-- libbz2.so.1.0.8
00007feb3f3ea000      8K r---- libbz2.so.1.0.8
00007feb3f3ec000      4K r---- libbz2.so.1.0.8
00007feb3f3ed000      4K rw--- libbz2.so.1.0.8
00007feb3f3f2000      8K r---- grp.cpython-39-x86_64-linux-gnu.so
00007feb3f3f4000      4K r-x-- grp.cpython-39-x86_64-linux-gnu.so
00007feb3f3f5000      4K r---- grp.cpython-39-x86_64-linux-gnu.so
00007feb3f3f6000      4K r---- grp.cpython-39-x86_64-linux-gnu.so
00007feb3f3f7000      4K rw--- grp.cpython-39-x86_64-linux-gnu.so
00007feb3f3f8000      8K r---- _bz2.cpython-39-x86_64-linux-gnu.so
00007feb3f3fa000      8K r-x-- _bz2.cpython-39-x86_64-linux-gnu.so
00007feb3f3fc000      4K r---- _bz2.cpython-39-x86_64-linux-gnu.so
00007feb3f3fd000      4K ----- _bz2.cpython-39-x86_64-linux-gnu.so
00007feb3f3fe000      4K r---- _bz2.cpython-39-x86_64-linux-gnu.so
00007feb3f3ff000      4K rw--- _bz2.cpython-39-x86_64-linux-gnu.so
00007feb3f400000   2048K rw---   [ anon ]
00007feb3f600000    692K r---- libcrypto.so.3.0.7
00007feb3f6ad000   2416K r-x-- libcrypto.so.3.0.7
00007feb3f909000    820K r---- libcrypto.so.3.0.7
00007feb3f9d6000      4K ----- libcrypto.so.3.0.7
00007feb3f9d7000    344K r---- libcrypto.so.3.0.7
00007feb3fa2d000     12K rw--- libcrypto.so.3.0.7
00007feb3fa30000     12K rw---   [ anon ]
00007feb3fa36000     20K r---- _datetime.cpython-39-x86_64-linux-gnu.so
00007feb3fa3b000     48K r-x-- _datetime.cpython-39-x86_64-linux-gnu.so
00007feb3fa47000     24K r---- _datetime.cpython-39-x86_64-linux-gnu.so
00007feb3fa4d000      4K r---- _datetime.cpython-39-x86_64-linux-gnu.so
00007feb3fa4e000     12K rw--- _datetime.cpython-39-x86_64-linux-gnu.so
00007feb3fa51000    256K rw---   [ anon ]
00007feb3fa93000      8K r---- _posixsubprocess.cpython-39-x86_64-linux-gnu.so
00007feb3fa95000      8K r-x-- _posixsubprocess.cpython-39-x86_64-linux-gnu.so
00007feb3fa97000      4K r---- _posixsubprocess.cpython-39-x86_64-linux-gnu.so
00007feb3fa98000      4K r---- _posixsubprocess.cpython-39-x86_64-linux-gnu.so
00007feb3fa99000      4K rw--- _posixsubprocess.cpython-39-x86_64-linux-gnu.so
00007feb3fa9a000    256K rw---   [ anon ]
00007feb3fada000     16K r---- _socket.cpython-39-x86_64-linux-gnu.so
00007feb3fade000     40K r-x-- _socket.cpython-39-x86_64-linux-gnu.so
00007feb3fae8000     36K r---- _socket.cpython-39-x86_64-linux-gnu.so
00007feb3faf1000      4K r---- _socket.cpython-39-x86_64-linux-gnu.so
00007feb3faf2000      4K rw--- _socket.cpython-39-x86_64-linux-gnu.so
00007feb3faf3000    256K rw---   [ anon ]
00007feb3fb33000      8K r---- _blake2.cpython-39-x86_64-linux-gnu.so
00007feb3fb35000     24K r-x-- _blake2.cpython-39-x86_64-linux-gnu.so
00007feb3fb3b000      8K r---- _blake2.cpython-39-x86_64-linux-gnu.so
00007feb3fb3d000      4K r---- _blake2.cpython-39-x86_64-linux-gnu.so
00007feb3fb3e000      4K rw--- _blake2.cpython-39-x86_64-linux-gnu.so
00007feb3fb3f000     12K r---- libz.so.1.2.11
00007feb3fb42000     56K r-x-- libz.so.1.2.11
00007feb3fb50000     24K r---- libz.so.1.2.11
00007feb3fb56000      4K ----- libz.so.1.2.11
00007feb3fb57000      4K r---- libz.so.1.2.11
00007feb3fb58000      4K rw---   [ anon ]
00007feb3fb59000      8K r---- select.cpython-39-x86_64-linux-gnu.so
00007feb3fb5b000     12K r-x-- select.cpython-39-x86_64-linux-gnu.so
00007feb3fb5e000      8K r---- select.cpython-39-x86_64-linux-gnu.so
00007feb3fb60000      4K ----- select.cpython-39-x86_64-linux-gnu.so
00007feb3fb61000      4K r---- select.cpython-39-x86_64-linux-gnu.so
00007feb3fb62000      4K rw--- select.cpython-39-x86_64-linux-gnu.so
00007feb3fb63000     16K r---- _hashlib.cpython-39-x86_64-linux-gnu.so
00007feb3fb67000     24K r-x-- _hashlib.cpython-39-x86_64-linux-gnu.so
00007feb3fb6d000     16K r---- _hashlib.cpython-39-x86_64-linux-gnu.so
00007feb3fb71000      4K ----- _hashlib.cpython-39-x86_64-linux-gnu.so
00007feb3fb72000      4K r---- _hashlib.cpython-39-x86_64-linux-gnu.so
00007feb3fb73000      4K rw--- _hashlib.cpython-39-x86_64-linux-gnu.so
00007feb3fb74000      8K r---- _random.cpython-39-x86_64-linux-gnu.so
00007feb3fb76000      8K r-x-- _random.cpython-39-x86_64-linux-gnu.so
00007feb3fb78000      4K r---- _random.cpython-39-x86_64-linux-gnu.so
00007feb3fb79000      4K r---- _random.cpython-39-x86_64-linux-gnu.so
00007feb3fb7a000      4K rw--- _random.cpython-39-x86_64-linux-gnu.so
00007feb3fb7b000      4K r---- _bisect.cpython-39-x86_64-linux-gnu.so
00007feb3fb7c000      4K r-x-- _bisect.cpython-39-x86_64-linux-gnu.so
00007feb3fb7d000      4K r---- _bisect.cpython-39-x86_64-linux-gnu.so
00007feb3fb7e000      4K r---- _bisect.cpython-39-x86_64-linux-gnu.so
00007feb3fb7f000      4K rw--- _bisect.cpython-39-x86_64-linux-gnu.so
00007feb3fb80000    512K rw---   [ anon ]
00007feb3fc00000    160K r---- libc.so.6
00007feb3fc28000   1492K r-x-- libc.so.6
00007feb3fd9d000    352K r---- libc.so.6
00007feb3fdf5000      4K ----- libc.so.6
00007feb3fdf6000     16K r---- libc.so.6
00007feb3fdfa000      8K rw--- libc.so.6
00007feb3fdfc000     52K rw---   [ anon ]
00007feb3fe0a000     16K r---- math.cpython-39-x86_64-linux-gnu.so
00007feb3fe0e000     28K r-x-- math.cpython-39-x86_64-linux-gnu.so
00007feb3fe15000     16K r---- math.cpython-39-x86_64-linux-gnu.so
00007feb3fe19000      4K r---- math.cpython-39-x86_64-linux-gnu.so
00007feb3fe1a000      4K rw--- math.cpython-39-x86_64-linux-gnu.so
00007feb3fe1b000    768K rw---   [ anon ]
00007feb3fedf000      8K r---- zlib.cpython-39-x86_64-linux-gnu.so
00007feb3fee1000     16K r-x-- zlib.cpython-39-x86_64-linux-gnu.so
00007feb3fee5000     12K r---- zlib.cpython-39-x86_64-linux-gnu.so
00007feb3fee8000      4K r---- zlib.cpython-39-x86_64-linux-gnu.so
00007feb3fee9000      4K rw--- zlib.cpython-39-x86_64-linux-gnu.so
00007feb3feea000      4K r---- _opcode.cpython-39-x86_64-linux-gnu.so
00007feb3feeb000      4K r-x-- _opcode.cpython-39-x86_64-linux-gnu.so
00007feb3feec000      4K r---- _opcode.cpython-39-x86_64-linux-gnu.so
00007feb3feed000      4K r---- _opcode.cpython-39-x86_64-linux-gnu.so
00007feb3feee000      4K rw--- _opcode.cpython-39-x86_64-linux-gnu.so
00007feb3feef000     16K r---- array.cpython-39-x86_64-linux-gnu.so
00007feb3fef3000     24K r-x-- array.cpython-39-x86_64-linux-gnu.so
00007feb3fef9000     16K r---- array.cpython-39-x86_64-linux-gnu.so
00007feb3fefd000      4K ----- array.cpython-39-x86_64-linux-gnu.so
00007feb3fefe000      4K r---- array.cpython-39-x86_64-linux-gnu.so
00007feb3feff000      4K rw--- array.cpython-39-x86_64-linux-gnu.so
00007feb3ff00000   1024K rw---   [ anon ]
00007feb40000000    360K r---- libpython3.9.so.1.0
00007feb4005a000   1748K r-x-- libpython3.9.so.1.0
00007feb4020f000    976K r---- libpython3.9.so.1.0
00007feb40303000      4K ----- libpython3.9.so.1.0
00007feb40304000     24K r---- libpython3.9.so.1.0
00007feb4030a000    220K rw--- libpython3.9.so.1.0
00007feb40341000    136K rw---   [ anon ]
00007feb40366000      4K r---- _heapq.cpython-39-x86_64-linux-gnu.so
00007feb40367000      8K r-x-- _heapq.cpython-39-x86_64-linux-gnu.so
00007feb40369000     12K r---- _heapq.cpython-39-x86_64-linux-gnu.so
00007feb4036c000      4K r---- _heapq.cpython-39-x86_64-linux-gnu.so
00007feb4036d000      4K rw--- _heapq.cpython-39-x86_64-linux-gnu.so
00007feb4036e000    256K rw---   [ anon ]
00007feb403ae000    340K r---- LC_CTYPE
00007feb40403000     12K rw---   [ anon ]
00007feb40406000     52K r---- libm.so.6
00007feb40413000    448K r-x-- libm.so.6
00007feb40483000    368K r---- libm.so.6
00007feb404df000      4K r---- libm.so.6
00007feb404e0000      4K rw--- libm.so.6
00007feb404e4000     28K r--s- gconv-modules.cache
00007feb404eb000      8K rw---   [ anon ]
00007feb404ed000      8K r---- ld-linux-x86-64.so.2
00007feb404ef000    152K r-x-- ld-linux-x86-64.so.2
00007feb40515000     44K r---- ld-linux-x86-64.so.2
00007feb40521000      8K r---- ld-linux-x86-64.so.2
00007feb40523000      8K rw--- ld-linux-x86-64.so.2
00007ffc65d45000    136K rw---   [ stack ]
00007ffc65dab000     16K r----   [ anon ]
00007ffc65daf000      8K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total         45531420K

Environment

$ mdedup --version
mdedup, version 7.5.0

Running on RHEL 9.4 x64.

Additional context

Sorry if my wording was a bit harsh, I'm just totally frightened by those high memory usage numbers.
@deajan deajan added the 🐛 bug Something isn't working, or a fix is proposed label Oct 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kdeldycke kdeldycke

Labels
🐛 bug Something isn't working, or a fix is proposed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kdeldycke @deajan