Release v0.8.3

[jshlbrd]

Welcome to Substation v0.8.3, the biggest point release we've done yet -- this release adds two new major capabilities that might completely change how you think about large-scale data processing! 🎉

Key-Value Stores

The system now supports read-only and read-write key-value stores such as AWS DynamoDB, CSV files, and text files -- now any field from any event can be stored locally or remotely at scale. Here's a small taste of what you can do with this feature:

• Enrich data within and across data pipelines -- for example, enrich logs from an identity provider (e.g., Okta, Duo) with logs from an EDR (e.g., CrowdStrike, Carbon Black) to verify if users are authenticating from managed devices
• Cache data in memory to limit the number of calls to external services
• Apply Zeek Intelligence Framework files to any log source
  • We've included support for Critical Path Security's Zeek Intelligence Feeds
• Match the Emerging Threats Compromised IP Addresses list against any log source
• Create an internally curated, enterprise-scale passive DNS database

Microservices

The system has an additional, entirely new deployment pattern -- microservices! This is for anyone who doesn't need a data pipeline but still wants to transform data, and it's the best way to integrate external systems and users. We've added examples for building synchronous and asynchronous microservices, including a recipe for deploying a centralized DNS service.

Let's take some of the examples mentioned above and see how they might work as a microservice:

• Put any Substation processor in a cache-aside service to speed up data processing
• Threat intelligence service that performs indicator matching on any data
• Centralized DNS resolution service that backs up results to a cloud database

Here are a few new examples of what you can do with microservices:

• IP Address validation service that categorizes addresses by their type and geolocation
• Data sanitization service that masks sensitive data such as personally identifiable information
• Content inspection service that identifies data based on its media (MIME) type

Other Changes

• Added a recipe that shows how to reduce multiple events into a single event using aggregation (aka "slash logs")
• Miscellaneous bug fixes in the configurations as code library and processor settings