Malcolm v24.04.0

Malcolm v24.04.0 contains new features, improvements, bug fixes and component version updates.

v24.03.1...v24.04.0

Because some of the environment variables used for configuring Malcolm have been reorganized in the .env files found in the ./config directory, it is recommended you re-run ./scripts/configure for this release.

• Features and enhancements
  • Zeek-extracted files scanned and preserved on a Hedgehog Linux sensor can now be accessed via the extracted files download user interface (idaholab#331).
  • Improvements to creation of index templates, dashboards, and other saved objects on startup (idaholab#208) to ensure that saved objects get created correctly upon upgrade (see this comment for more details on this feature).
  • Populating the NetBox inventory via passively-gathered network traffic metadata now uses network traffic logs for DNS, NTLM, and DHCP to identify assets' host names when possible for use when populating device and VM names (idaholab#415). Autopopulated devices now have their status field set to Active rather than Stage, and uses tags instead to indicated that they were created through autopopulation.
  • Users can now specify pruning thresholds for carved files so that old files are deleted in order to avoid filling available storage (idaholab#453). See a new section of documentation on Managing disk usage for more information about this and similar settings.
  • Users can now specify a prefix that will be prepended to dashboards as they are imported into OpenSearch Dashboards or Kibana, allowing users who have dashboards from other sources to differentiate between those and Malcolm's (idaholab#455).
  • The default anomaly detectors created for the OpenSearch Anomaly Detection plugin are now created with category fields for high cardinality to allow for better breakdown of contributing values to anomalies discovered (idaholab#464).
  • Include JA4+ plugin in Arkime. See idaholab#419 for status on upcoming full JA4+ support in Malcolm.
  • Hedgehog Linux sensors can now periodically refresh their Zeek inteligence files.
    • NOTE: Due to an oversight, a value is missing from the default Hedgehog Linux configuration in this release, preventing the intel refresh cron job from executing. As a workaround, appending the line export INTEL_DIR=/opt/sensor/sensor_ctl/zeek/intel to /opt/sensor/sensor_ctl/control_vars.conf and restarting the sensor services will remedy the situation. This will be corrected in the next Malcolm release.
  • Assorted documentation improvements.
• Component version updates
  • Arkime to v5.1.2
  • OpenSearch and OpenSearch Dashboards to v2.13.0
  • Beats to v8.13.2
  • Logstash to v8.13.2
  • gunicorn to v22.0.0 to address CVE-2024-1135.
  • elasticsearch-dsl to v8.13.0
  • elasticsearch-py to v8.13.0
  • idna to v3.7 to address CVE-2024-3651
  • Fluent Bit to v3.0.3
• Bug fixes
  • The documentation for Windows host system configuration was out of date and has been updated for the latest version of Microsoft Windows Subsystem for Linux (idaholab#421).
  • An issue was fixed in which Malcolm's list of users and their password hashes could become corrupted if the file did not initially end with a newline character (idaholab#426).
  • The manner in which Zeek intel files are generated has been changed to avoid problems found in Kubernetes deployments when scaling out the number of zeek-live containers (idaholab#456). See this comment for more details.
  • Removed the version top-level element from docker-compose.yml files as it is now obsolete and caused a warning message that sometimes was not handled correctly.
  • Fix Malcolm ISO not correctly detecting if it's in a live boot ISO environment or installed mode.
  • Restart live Zeek instances with zeekctl deploy instead of zeekctl restart.
• Configuration changes (in environment variables in ./config/)
  • ARKIME_QUERY_ALL_INDICES in arkime.env can be set to control the queryAllIndices setting in Arkime's config.ini.
  • DASHBOARDS_PREFIX in dashboards-helper.env has been added for idaholab#455 (see above in Features and Enhancements).
  • LOGSTASH_NETBOX_ENRICHMENT_DATASETS in logstash.env has been changed to include zeek.dhcp, zeek.dns, and zeek.ntlm to support idaholab#415 (see above in Features and Enhancements).
  • LOGSTASH_ZEEK_IGNORED_LOGS in logstash.env has been changed to remove capture_loss and stats so that those diagnostic Zeek logs can be parsed without the user having to manually change this variable.
  • ZEEK_CRON has been removed from zeek-live.env and ZEEK_INTEL_REFRESH_CRON_EXPRESSION was removed from zeek.env and moved to the "offline" version of the container in zeek-offline.env for idaholab#456.
  • EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE, EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT, and EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS were added to zeek.env for idaholab#453. See a new section of documentation on Managing disk usage for more information about these and similar settings.

Official ISO installer images for Malcolm and Hedgehog Linux can now be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.