Version 3.0.1

.n6-version

@@ -1 +1 @@
-3.0.1a1
+3.0.1

CHANGELOG.md

@@ -1,13 +1,27 @@
 # Changelog
 
-Starting with the 3.0.0 release, all notable changes applied to the
+Starting with the 3.0.0 release, all notable changes to the
 [code of _n6_](https://github.com/CERT-Polska/n6) are continuously
 documented here.
 
-The format of this file is based, to much extent, on
+The format of this document is based, to much extent, on
 [Keep a Changelog](https://keepachangelog.com/).
 
 
+## [3.0.1] - 2021-12-03
+
+### Changes and Fixes
+
+- [docs] A bunch of fixes and improvements regarding the documentation,
+  including major changes to its structure, layout and styling.
+
+- [setup] In `do_setup.py`, regarding the default value of the option
+  `--additional-packages` under Python 3: the version of the `mkdocs` tool
+  has been pinned (`1.2.3`), and the `mkdocs-material` library (providing
+  the `material` docs theme) has been added (and its version is also
+  pinned: `8.0.3`).
+
+
 ## [3.0.0] - 2021-12-01
 
 **This release is a big milestone.** It includes, among others:
@@ -42,4 +56,5 @@ be able to rid of `N6Core` and `N6CoreLib`).  There are also
 `N6SDK-py2` (needed only as dependencies of `N6Core`/`N6CoreLib`).
 
 
+[3.0.1]: https://github.com/CERT-Polska/n6/compare/v3.0.0...v3.0.1
 [3.0.0]: https://github.com/CERT-Polska/n6/compare/v2.0.6a2-dev1...v3.0.0

README.md

@@ -11,7 +11,7 @@ and incidents in their networks.
 ## Basic References
 
 * **Home page:** [github.com/CERT-Polska/n6](https://github.com/CERT-Polska/n6)
-* **Documentation:** [n6.readthedocs.io](https://n6.readthedocs.io/en/latest/)
+* **Documentation:** [n6.readthedocs.io](https://n6.readthedocs.io)
 
 The project is developed for [CERT Polska](https://www.cert.pl/en/).
 Contact us via e-mail: [[email protected]](mailto:[email protected]).

do_setup.py

@@ -24,7 +24,6 @@
         'coverage<6.0',
         'astroid==1.6.6',
         'pylint==1.9.5',
-        'mkdocs==1.0.4',
         'waitress<2.0',
     ]
 else:
@@ -33,7 +32,8 @@
         'pytest-cov==2.12.1',
         'coverage',
         'pylint',
-        'mkdocs',
+        'mkdocs==1.2.3',
+        'mkdocs-material==8.0.3',
         'waitress',
     ]
 DEFAULT_LOG_CONFIG = {

docs/data_flow_overview.md

@@ -1,4 +1,9 @@
-# *n6* Architecture and Data Flow Overview
+---
+hide:
+  - toc
+---
+
+# Architecture and Data Flow Overview
 
 The *n6*'s overall architecture and data flow can be described schematically as follows:
 

docs/guides/new_source/collectors/index.md → docs/dev_guides/new_source/collectors/index.md

@@ -6,8 +6,10 @@ So the pipeline at this stage looks like this:
 
 ![`(External Data) → [Collector] → [Parser] → ...`](c_p_pipe.png)
 
-> **Note:** For a broader overview, you may want to take a look at
-> the [*n6* architecture and data flow](../../../data_flow_overview.md) diagram.
+!!! note
+
+    For a broader overview, you may want to take a look at the
+    [*n6* architecture and data flow](../../../data_flow_overview.md) diagram.
 
 To describe collectors' job more technically: a collector obtains data
 from its respective external source (e.g., by downloading files from a
@@ -65,7 +67,7 @@ that will not be shared with other collectors.  The
 class helps to take care of that.
 
 
-# The *Collectors* part's chapters
+## This part's contents
 
 * [Collector Executable Commands](command.md)
 * [Collector Base Classes](classes.md)

docs/guides/new_source/index.md → docs/dev_guides/new_source/index.md

@@ -1,17 +1,20 @@
-# Implementing a New Data Source (Developer Guide)
+# Implementing a New Data Source
+
+!!! warning "TODO note"
+
+    **This guide needs an update regarding the recent migration of *n6* from
+    Python 2.7 to 3.9**, especially that the *n6*'s data sources and data
+    pipeline stuff now works under Python 3.9 and resides in `N6DataSources`
+    and `N6DataPipeline` (where some names and other interface details
+    differ from the corresponding, Python-2.7-only, legacy stuff kept in
+    `N6Core`).
 
-**TBD: this guide needs an update regarding the stuff that now works
-under Python 3.9 and resides in `N6DataPipeline` and `N6DataSources`
-(*not* in `N6Core` where the legacy Python-2.7 stuff is kept).**
 
 The aim of this guide is to describe how to implement new *n6*
 components necessary for collecting and parsing data from some
 external security data source.
 
-
-## The guide's contents
-
-This guide consists of two parts:
+**This guide consists of two parts:**
 
 * [Collectors](collectors/index.md).
   The first part focuses on ***collectors*** which are *n6*'s data entry

docs/guides/new_source/parsers/index.md → docs/dev_guides/new_source/parsers/index.md

@@ -16,9 +16,11 @@ format), and sends them further down the data processing pipeline (by
 pushing those data -- already in their *normalized* form -- into the
 appropriate RabbitMQ exchange).
 
-**Note:** Whereas collectors may be *stateful*, **parsers shall always
-be *stateless*** (i.e., they should *neither* store any persistent state
-*nor* make use of any external mutable context, such as current time).
+!!! important
+
+    Whereas collectors may be *stateful*, **parsers shall always be
+    *stateless*** (i.e., they should *neither* store any persistent state
+    *nor* make use of any external mutable context, such as current time).
 
 
 ## Types of parsers/events
@@ -67,7 +69,7 @@ attributes [in the definition of the `RecordDict`
 class](https://github.com/CERT-Polska/n6/blob/master/N6Lib/n6lib/record_dict.py#L645).
 
 
-# The *Parsers* part's chapters
+## This part's contents
 
 * [Parser Executable Commands](command.md)
 * [Parser Base Classes](baseclasses.md)

docs/docker.md → docs/install_and_conf/docker.md

@@ -1,9 +1,12 @@
-# Docker-Based Installation Guide
+# Docker-Based Installation
 
-**TBD: this guide needs an update regarding the migration from Python
-2.7 to 3.9; in particular, the current implementation of the *n6* basic
-data pipeline now resides in `N6DataPipeline` (Python-3-only), *not* in
-`N6Core` (where the legacy Python-2 stuff is kept).**
+!!! warning "TODO note"
+
+    **This guide may need an update regarding the recent migration of
+    *n6* from Python 2.7 to 3.9.**
+
+
+## Opening remarks
 
 This short guide describes how to run, for testing and exploration, the
 latest version of *n6* -- using the *Docker* and *Docker Compose* tools.
@@ -13,48 +16,52 @@ in the easiest possible way, so that you can learn -- by monitoring and
 experimenting -- how the *n6* system works and how you can interact with
 it.
 
-> **Note:** If you wish to explore the *n6*'s ecosystem a bit more profoundly
-> you may want to familiarize yourself with the [Step-by-Step Installation
-> Guide](installation/index.md) as well.
+!!! note
 
+    If you wish to explore the *n6*'s ecosystem a bit more profoundly
+    you may want to familiarize yourself with the [Step-by-Step
+    Installation](../install_and_conf/step_by_step/index.md) guide
+    as well.
 
-## Important: what these materials *are* and what they are *not*
+!!! warning "Disclaimer: what these materials *are* and what they are *not*"
 
-This installation guide, as well as the stuff you can find in the
-[`etc/`](https://github.com/CERT-Polska/n6/tree/master/etc) and
-[`docker/`](https://github.com/CERT-Polska/n6/tree/master/docker)
-directories of the *n6* source code repository, concern setting up an
-*n6* instance just for testing, exploration and experimentation, i.e.,
-**not for production** (at least, *not* without careful security-focused
-adjustments).
+    This installation guide, as well as the stuff you can find in the
+    [`etc/`](https://github.com/CERT-Polska/n6/tree/master/etc) and
+    [`docker/`](https://github.com/CERT-Polska/n6/tree/master/docker)
+    directories of the *n6* source code repository, concern setting up an
+    *n6* instance just for testing, exploration and experimentation, i.e.,
+    **not for production** (at least, *not* without careful security-focused
+    adjustments).
 
-In other words, these materials are *not* intended to be used as a
-recipe for a secure production setup -- in particular, when it comes to
-(but not limited to) such issues as X.509 certificates (note that those
-in the [`etc/ssl/*`](https://github.com/CERT-Polska/n6/tree/master/etc/ssl)
-directories of the source code repository are purely example ones --
-they should *never* be used for anything related to production
-systems!), authentication and authorization settings (which in these
-materials are, generally, either skipped or reduced to what is necessary
-just to run the stuff), or file access permissions.
+    In other words, these materials are *not* intended to be used as a
+    recipe for a secure production setup -- in particular, when it comes to
+    (but not limited to) such issues as X.509 certificates (note that those
+    in the [`etc/ssl/*`](https://github.com/CERT-Polska/n6/tree/master/etc/ssl)
+    directories of the source code repository are purely example ones --
+    they should *never* be used for anything related to production
+    systems!), authentication and authorization settings (which in these
+    materials are, generally, either skipped or reduced to what is necessary
+    just to run the stuff), or file access permissions.
 
-It should be obvious that an experienced system administrator and/or
-security specialist should prepare and/or carefully review and adjust
-any configuration/installation/deployment of services that are to be
-made production ones, in particular if those services are to be made
-public.
+    It should be obvious that an experienced system administrator and/or
+    security specialist should prepare and/or carefully review and adjust
+    any configuration/installation/deployment of services that are to be
+    made production ones, in particular if those services are to be made
+    public.
 
 
 ## Requirements
 
-- *Docker* installed (the newest version)
-- *Docker Compose* installed (the newest version)
-- The *n6* source code repository cloned
+- *Docker* installed (a reasonably new version)
+- *Docker Compose* installed (a reasonably new version)
+- The *n6* [source code repository](https://github.com/CERT-Polska/n6) cloned
 
 
 ## Building the environment
 
-> **Note:** Make sure you are in the top-level directory of the cloned source code repository.
+!!! note
+
+    Make sure you are in the top-level directory of the cloned source code repository.
 
 To build our demonstrational _n6_ environment we use [Docker Compose](https://docs.docker.com/compose/) which binds all the services needed to run the *n6* infrastructure.
 
@@ -64,9 +71,11 @@ $ docker-compose build
 
 The result of the process are ready-to-use docker images.
 
-> **Note:** The Docker stack requires all images to be built correctly.
-> In case of errors, please do not hesitate to create an
-> [issue on our GitHub site](https://github.com/CERT-Polska/n6/issues).
+!!! note
+
+    The Docker stack requires all images to be built correctly.
+    In case of errors, please do not hesitate to create an
+    [issue on our GitHub site](https://github.com/CERT-Polska/n6/issues).
 
 If the build process has been correctly performed you should be able to
 run the following command to obtain a result similar to what is listed
@@ -103,9 +112,11 @@ By default, the stack exposes the following ports:
 * 4444 -- *n6* Admin Panel
 * 15671 -- RabbitMQ Management
 
-> **Note:** Make sure that all ports are not used by your localhost.
-> If a port is used by another service, please change it in the
-> `docker-compose.yml` file.
+!!! note
+
+    Make sure that all ports are not used by your localhost.
+    If a port is used by another service, please change it in the
+    `docker-compose.yml` file.
 
 
 ## Launching the system
@@ -118,8 +129,9 @@ $ docker-compose up
 
 Now, give Docker a few minutes to initialize.
 
-> **Note:** You can add the `-d` flag to run the stuff in the background
-> (*detached mode*).
+!!! tip
+
+    You can add the `-d` flag to run the stuff in the background (*detached mode*).
 
 
 ### First startup
@@ -275,7 +287,9 @@ Some files and directories are worth mentioning -- namely:
 
 ## Supervisor
 
-> **Note:** This setup requires running `docker-compose up`.
+!!! note
+
+    This setup requires running `docker-compose up`.
 
 Run `supervisorctl` to examine the status of all n6 components:
 

docs/guides/intelmq/config.md → docs/install_and_conf/intelmq/config.md

@@ -1,37 +1,37 @@
 # Installation and Configuration
 
-## Necessary n6 packages
+## Necessary *n6* packages
 
 Make sure you have the `N6DataPipeline` package installed:
 
 ```bash
 (env)$ ./do_setup.py N6DataPipeline
 ```
 
-## IntelMQ package
+## *IntelMQ* package
 
-Install the IntelMQ package from PyPI. For the best compatibility, install the `3.0.2` version,
-which is the last version tested.
+Install the `intelmq` package from PyPI. For the best compatibility,
+install the `3.0.2` version, which is the last version tested.
 
 ```bash
 (env)$ pip install intelmq
 (env)$ useradd -d /opt/intelmq -U -s /bin/bash intelmq
 $ sudo intelmqsetup
 ```
 
-## IntelMQ runtime configuration
+## *IntelMQ* runtime configuration
 
-n6 bot runner uses IntelMQ's runtime configuration. It may be placed
+n6 bot runner uses *IntelMQ*'s runtime configuration. It may be placed
 in '/opt/intelmq/etc/runtime.yaml' or '/etc/intelmq/runtime.yaml' etc.
 
 This YAML config's options are the IDs of the bots. These IDs will be used as arguments for
 the n6 command `n6run_bot`. You can use existing bot configurations or define new ones.
 
-n6 system for running IntelMQ bots reads the `n6config` subsection in bot ID section, which is ignored
-by the IntelMQ. This subsection provides some configuration, which may be required for some types
+n6 system for running *IntelMQ* bots reads the `n6config` subsection in bot ID section, which is ignored
+by the *IntelMQ*. This subsection provides some configuration, which may be required for some types
 of bots (like parser bots).
 
-### Example IntelMQ bot configuration
+### Example *IntelMQ* bot configuration
 
 ```yaml
 spamhaus-drop-parser:
@@ -56,17 +56,17 @@ require a `default_binding_key` option in the `n6config` section. The option's v
 that the parser will accept incoming messages from the collector, which sends messages with
 routing key `spamhaus.intelmq-collector`.
 
-## n6 pipeline configuration
+## *n6* pipeline configuration
 
-You have to place IntelMQ components somewhere in the pipeline. It can be achieved by configuring
+You have to place *IntelMQ* components somewhere in the pipeline. It can be achieved by configuring
 components that will be used in the n6 config's `pipeline` section. For example: you want to
 obtain the following order of components:
 ```
 spamhaus.intelmq-collector -> spamhaus-drop-parser -> maxmind-expert
 -> gethostbyname-1-expert -> intelmq-to-n6-adapter -> enricher
 ```
 
-The `intelmq-to-n6-adapter` is a component, which converts messages from IntelMQ format to
+The `intelmq-to-n6-adapter` is a component, which converts messages from *IntelMQ* format to
 n6 format.
 
 The `pipeline` configuration should look like:

docs/install_and_conf/intelmq/index.md

@@ -0,0 +1,13 @@
+# Running *IntelMQ* Bots in *n6* Pipeline
+
+## Opening remarks
+
+The aim of this guide is to describe how to configure and run
+*[IntelMQ](https://github.com/certtools/intelmq)* bots within the
+*n6*'s message pipeline.
+
+
+## This guide's contents
+
+* [Installation and Configuration](config.md)
+* [Running *IntelMQ* Bots and Message Format Adapters](running.md)