Skip to content

2.1.0
Compare
Choose a tag to compare
released this 15 Oct 11:10
· 2493 commits to develop since this release
2.1.0
ced415d

Install documentation:
https://github.com/certtools/intelmq/blob/2.1.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.0/docs/UPGRADING.md

Core

  • intelmq.lib.harmonization:
    • Use correct parent classes.
    • Add DateTime.convert as interface for all existing conversion functions.
    • add DateTime.convert_from_format.
    • add DateTime.convert_from_format_midnight.
    • add DateTime.convert_fuzzy.
  • intelmq.lib.pipeline:
    • Redis: Use single connection client if calling bot is not multithreaded. Gives a small speed advantage.
    • Require the bot instance as parameter for all pipeline classes.
    • New internal variable _has_message to keep the state of the pipeline.
    • Split receive and acknowledge into public-facing and private methods.
    • Add reject_message method to the Pipeline class for explicit requeue of messages.
    • AMQP:
      • Make exchange configurable.
      • If exchange is set, the queues are not declared, the queue name is for routing used by exchanges.
  • intelmq.lib.bot:
    • Log message after successful bot initialization, no log message anymore for ready pipeline.
    • Use existing current message if receive is called and the current message still exists.
    • Fix handling of received messaged after a sighup that happend during a blocking receving connection using explicit rejection (#1438).
    • New method _parse_common_parameters called before init to parse commonly used argument. Currently supported: extract_files.
  • intelmq.lib.test:
    • Fix the tests broker by providing the testing pipeline.
  • intelmq.lib.utils:
    • unzip:
      • new parameter return_names to optionally return the file names.
      • support for zip
      • new parameters try_zip, try_gzip and try_tar to control which compressions are tried.
      • rewritten to an iterative approach
    • add file_name_from_response to extract a file name from a Response object for downloaded files.
  • intelmq.lib.upgrades: Added v210_deprecations for deprecated parameters.

Harmonization

  • Add extra to reports.

Bots

Collectors

  • intelmq.bots.collectors.http.collector_http:
    • More extensive usage of intelmq.lib.utils.unzip.
    • Save the file names in the report if files have been extracted form an archive.
  • intelmq.bots.collectors.rt.collector_rt:
    • Save ticket information/metadata in the extra fields of the report.
    • Support for RT 3.8 and RT 4.4.
    • New parameters extract_attachment and extract_download for generic archive extraction and consistency. The parameter unzip_attachment is deprecated.
  • intelmq.bots.collectors.mail.*: Save email information/metadata in the extra fields of the report. See the bots documentation for a complete list of provided data.
    • intelmq.bots.collectors.mail.collector_mail_attach: Check for existence/validity of the attach_regex parameter.
    • Use the lib's unzip function for uncompressing attachments and use the .
    • intelmq.bots.collectors.mail.collector_mail_url: Save the file name of the downloaded file as extra.file_name.
  • intelmq.bots.collectors.amqp.collector_amqp: New collector to collect data from (remote) AMQP servers, for bot IntelMQ as well as external data.
    • use default SSL context for client purposes, fixes compatibility with python < 3.6 if TLS is used.

Parsers

  • intelmq.bot.parsers.html_table.parser:
    • New parameter "html_parser".
    • Use time conversion functions directly from intelmq.lib.harmonization.DateTime.convert.
    • Limit lxml dependency on 3.4 to < 4.4.0 (incompatibility).
  • intelmq.bots.parsers.netlab_360.parser: Add support for hajime scanners.
  • intelmq.bots.parsers.hibp.parser_callback: A new parser to parse data retrieved from a HIBP Enterprise Subscription.
  • intelmq.bots.parsers.shadowserver.parser:
    • Ability to detect the feed base on the reports's field extra.file_name, so the parameter feedname is no longer required and one configured parser can parse any feed (#1442).

Experts

  • Add geohash expert.
  • intelmq.bot.experts.generic_db_lookup.expert
    • new optional parameter engine with postgresql (default) and sqlite (new) as possible values.

Outputs

  • Add intelmq.bots.outputs.touch.output.
  • intelmq.bot.outputs.postgresql.output:
    • deprecated in favor of intelmq.bot.outputs.sql.output
    • Compatibility shim will be available in the 2.x series.
  • intelmq.bot.outputs.sql.output added generic SQL output bot. Comparted to
    • new optional parameter engine with postgresql (default) and sqlite (new) as possible values.
  • intelmq.bots.outputs.stomp.output: New parameters message_hierarchical_output, message_jsondict_as_string, message_with_type, single_key.

Documentation

  • Feeds:
    • Add ViriBack feed.
    • Add Have I Been Pwned Enterprise Callback.
  • intelmq.tests.bots.outputs.amqptopic.test_output: Added.
  • Move the documentation of most bots from separate README files to the central Bots.md and feeds.yaml files.

Tests

  • Travis:
    • Use UTC timezone.
  • Tests for utils.unzip.
  • Add a new asset: Zip archive with two files, same as with tar.gz archive.
  • Added tests for the Mail Attachment & Mail URL collectors.
  • Ignore logging-tests on Python 3.7 temporarily (#1342).

Tools

  • intelmqctl:
    • Use green and red text color for some interactive output to indicate obvious errors or the absence of them.
  • intelmqdump:
    • New edit action v to modify a message saved in the dump (#1284).

Contrib

  • malware name mapping:
    • Add support for MISP treat actors data, see it's README for more information.
      • And handle empty synonyms in misp's galxies data.
    • Move apply-Script to the new EventDB directory
  • EventDB: Scripts for applying malware name mapping and domain suffixes to an EventDB.

Known issues

  • MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
  • ctl: shell colorizations are logged (#1436)
  • http stream collector: retry on regular connection problems? (#1435)
  • tests: capture logging with context manager (#1342)
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
  • n6 parser: mapping is modified within each run (#905)
  • reverse DNS: Only first record is used (#877)
  • Corrupt dump files when interrupted during writing (#870)
Assets 2
Loading