Free disk space

[y0d4a]

Hi,

I've been struggling to get Malcolm to run reliably on its own.
I keep running out of free space, primarily due to uploaded PCAP files.

In the configuration menu, I've set thresholds to monitor and manage free space when it drops below 80%, but it doesn't seem to be working.
The partition keeps filling up, and I can’t get Malcolm to run for more than two days before running into issues.

Any advice or suggestions would be greatly appreciated!

[mmguero]

Managing disk usage can be tricky because there are a lot of different components and different types of data in Malcolm that are all sort of "doing their own thing" when it comes to pruning old data to make sure there's room for the new.

Let's double check the settings you've got that have to do with this and then if we can identify which one(s) need to change you can either directly change them in the .env files or I can point you at the right question in the configuration wizard.

In your Malcolm directory, run this command to get the values for the variables controlling the storage management for the various aspects of Malcolm's data, and you'll see something like (although probably not exactly like) this example output:

$ grep -P "^\w*(PRUNE|CLEAN|FREE|MANAGE_PCAP)\w*\s*=" config/*.env
config/arkime.env:MANAGE_PCAP_FILES=false
config/arkime.env:ARKIME_FREESPACEG=10%
config/dashboards-helper.env:OPENSEARCH_INDEX_SIZE_PRUNE_LIMIT=0
config/dashboards-helper.env:OPENSEARCH_INDEX_SIZE_PRUNE_NAME_SORT=false
config/filebeat.env:FILEBEAT_CLEAN_INACTIVE=180m
config/filebeat.env:FILEBEAT_CLEAN_REMOVED=true
config/filebeat.env:LOG_CLEANUP_MINUTES=15
config/filebeat.env:ZIP_CLEANUP_MINUTES=20
config/zeek.env:EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE=1TB
config/zeek.env:EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT=0
config/zeek.env:EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS=60

If you can post your results for that command we can go from there.

[y0d4a]

Yeah, i suppose that is the issue.
I tried couple of options, this is last conf. which is running:

config/arkime.env:MANAGE_PCAP_FILES=false
config/arkime.env:ARKIME_FREESPACEG=10%
config/dashboards-helper.env:OPENSEARCH_INDEX_SIZE_PRUNE_LIMIT=1.5TB
config/dashboards-helper.env:OPENSEARCH_INDEX_SIZE_PRUNE_NAME_SORT=false
config/filebeat.env:FILEBEAT_CLEAN_INACTIVE=180m
config/filebeat.env:FILEBEAT_CLEAN_REMOVED=true
config/filebeat.env:LOG_CLEANUP_MINUTES=360
config/filebeat.env:ZIP_CLEANUP_MINUTES=720
config/zeek.env:EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE=1.5TB
config/zeek.env:EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT=0
config/zeek.env:EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS=300

I have 1.2T partition for malcolm folder (where is all data, including pcaps)

[mmguero]

Okay, if everything is on one partition we need to make a few adjustments. These can all be made in the configuration wizard, but I'll show you the values you'll want to change in the config files themselves first to get it working:

• MANAGE_PCAP_FILES needs to be true to allow the pruning process to delete older PCAP files to make room for newer ones
• ARKIME_FREESPACEG needs to be set to a value that means "Start deleting PCAP files when the free space drops below this amount;" in other words, setting it to 10% means "the oldest PCAP files will be deleted when my free space is below 10% of the drive" (or when free space is < ~120 gigabytes in your case, with a 1.2 TB partition)
• OPENSEARCH_INDEX_SIZE_PRUNE_LIMIT needs to be set to some lower value as well; this is the maximum size we're going to allow the opensearch indices to be before we start deleting old ones. If your partition is 1.2TB, I'd set this value to something like 750GB.
• EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE is a threshold for extracted (carved) files. This probably doesn't need to be as large, you could set it to maybe 100GB.

So with those changes, extracted files will not consume more than 100GB, opensearch indices will not consume more than 700G, and PCAP deletion will occur when the free space drops below 10%, meaning you're going to end up with like 280G left for PCAPs.

To make those changes, stop malcolm, edit the .env files, and then start malcolm again. Or, if you're going through the configuration wizard these questions are all under Should Malcolm delete the oldest database indices and capture artifacts based on available storage?