Skip to content

Commit

Permalink
Refactor documentation
Browse files Browse the repository at this point in the history 
Signed-off-by: tempate <[email protected]>
  • Loading branch information
@Tempate
Tempate committed Jul 8, 2024
1 parent 4de930c commit 2f895b0
Showing 1 changed file with 131 additions and 107 deletions.
238 changes: 131 additions & 107 deletions docs/rst/recording/usage/configuration.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -332,65 +332,26 @@ The recorder output file does support the following configuration settings under
- ``boolean``
- ``true``

* - Safety margin
- ``safety_margin``
- Configure safety margin (bytes) used |br|
in MCAP file size estimations.
- ``unsigned int``
- ``0``

* - Resource limits
- ``resource-limits``
- :ref:`recorder_usage_configuration_resource_limits`
- ``map``
- ``unlimited``

When DDS Recorder application is launched (or when remotely controlled, every time a ``start/pause`` command is received while in ``SUSPENDED/STOPPED`` state), a temporary file with ``filename`` name (+timestamp prefix) and ``.mcap.tmp~`` extension is created in ``path``.
This file is not readable until the application terminates, receives a ``suspend/stop/close`` command, or the file reaches its maximum size (see :ref:`Resource Limits <recorder_usage_configuration_resource_limits>`).
On such event, the temporal file is renamed to have ``.mcap`` extension in the same location, and is then ready to be processed.

.. _recorder_usage_configuration_resource_limits:

Resource Limits
"""""""""""""""

The ``resource-limits`` tag allows users to limit the size of the *DDS Recorder's* output.
The ``max-file-size`` tag specifies the maximum size of each output file and the ``max-size`` tag specifies the maximum aggregate size of all output files.
If the ``max-size`` is higher than the ``max-file-size``, the |ddsrecorder| will create multiple files with a maximum size of ``max-file-size``.
By default, however, the ``max-file-size`` is unlimited (``0B``) and the ``max-size`` is the same as the ``max-file-size``; that is, by default the |ddsrecorder| creates a single file of unlimited size.

.. warning::

If the ``max-file-size`` or the ``max-size`` are set to a value lower than the available space in the disk, the |ddsrecorder| will replace them with the available space in the disk.

To keep the |ddsrecorder| recording after reaching the ``max-size``, users can set the ``file-rotation`` tag to ``true``.
Enabling ``file-rotation`` allows the |ddsrecorder| to overwrite old files to free space for new ones.

.. note::

To keep the |ddsrecorder| from overwriting previous output files, users can set ``{"avoid_overwriting_output": true}`` as the argument (``"args"``) of the ``stop`` command (see :ref:`Control Commands <recorder_remote_controller_data_types>`).
If set, the |ddsrecorder| will not overwrite its output files, at the cost of possibly exceeding the ``max-size``.

.. note::

If an output file is moved, deleted, or renamed, the |ddsrecorder| will keep the size of the file reserved and rotate between the remaining files.

**Example of usage**

.. code-block:: yaml
resource-limits:
max-file-size: 250KB
max-size: 2MiB
file-rotation: true
Buffer size
^^^^^^^^^^^

``buffer-size`` indicates the number of samples to be stored in the process memory before the dump to disk.
This avoids disk access each time a sample is received.
By default, its value is set to ``100``.

.. _recorder_usage_configuration_cleanup_period:

Cleanup Period
^^^^^^^^^^^^^^

As explained in :ref:`Event Window <recorder_usage_configuration_event_window>`, a |ddsrecorder| in paused mode awaits for an event command to write in disk all samples received in the last ``event-window`` seconds.
To accomplish this, received samples are stored in memory until the aforementioned event is triggered and, in order to limit memory consumption, outdated (received more than ``event-window`` seconds ago) samples are removed from this buffer every ``cleanup-period`` seconds.
By default, its value is equal to twice the ``event-window``.

.. _recorder_usage_configuration_event_window:

Event Window
Expand All @@ -402,14 +363,27 @@ Thus, when an event is triggered from the remote controller, samples received in
In other words, the ``event-window`` acts as a sliding time window that allows to save the collected samples in this time window only when the remote controller event is received.
By default, its value is set to ``20`` seconds.

.. _recorder_usage_configuration_logpublishtime:
.. _recorder_usage_configuration_max_number_pending_samples:

Log Publish Time
^^^^^^^^^^^^^^^^
Maximum Number of Pending Samples
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

By default (``log-publish-time: false``) received messages are stored in the MCAP file with ``logTime`` value equals to the reception timestamp.
Additionally, the timestamp corresponding to when messages were initially published (``publishTime``) is also included in the information dumped to MCAP files.
In some applications, it may be required to use the ``publishTime`` as ``logTime``, which can be achieved by providing the ``log-publish-time: true`` configuration option.
It is possible that a |ddsrecorder| starts receiving data from a topic that it has not yet registered, i.e. a topic for which it does not know the data type.
In this case, messages are kept in an internal circular buffer until their associated type information is received, event on which they are written to disk.

However, the recorder execution might end before this event ever occurs.
Depending on configuration (see :ref:`recorder_usage_configuration_onlywithtype`), messages kept in the pending samples buffer will be stored or not on closure.
Hence, note that memory consumption would continuously grow whenever a sample with unknown type information is received.

To avoid the exhaustion of memory resources in such scenarios, a configuration option is provided which lets the user set a limit on memory usage.
The ``max-pending-samples`` parameter allows to configure the size of the aforementioned circular buffers **for each topic** that is discovered.
The default value is equal to ``5000`` samples, with ``-1`` meaning no limit, and ``0`` no pending samples.

Depending on the combination of this configuration option and the value of ``only-with-type``, the following situations may arise when a message with unknown type is received:

* If ``max-pending-samples`` is ``-1``, or if it is greater than ``0`` and the circular buffer is not full, the sample is added to the collection.
* If ``max-pending-samples`` is greater than ``0`` and the circular buffer reaches its maximum capacity, the oldest sample with same type as the received one is popped, and either written without type (``only-with-type: false``) or discarded (``only-with-type: true``).
* If ``max-pending-samples`` is ``0``, the message is written without type if ``only-with-type: false``, and discarded otherwise.

.. _recorder_usage_configuration_onlywithtype:

Expand All @@ -419,10 +393,45 @@ Only With Type
By default, all (allowed) received messages are recorded regardless of whether their associated type information has been received.
However, a user can enforce that **only** samples whose type is received are recorded by setting ``only-with-type: true``.

.. _recorder_usage_configuration_recordtypes:

Record Types
^^^^^^^^^^^^

By default, all type information received during execution is stored in a dedicated MCAP file section.
This information is then leveraged by |ddsreplayer| on playback, publishing recorded types in addition to data samples, which may be required for receiver applications relying on :term:`Dynamic Types<DynamicTypes>` (see :ref:`Replay Types <replayer_replay_configuration_replaytypes>`).
However, a user may choose to disable this feature by setting ``record-types: false``.

.. _recorder_usage_configuration_topictypeformat:

Topic type format
^^^^^^^^^^^^^^^^^

The optional ``ros2-types`` tag enables specification of the format for storing schemas.
When set to ``true``, schemas are stored in ROS 2 message format (.msg).
If set to ``false``, schemas are stored in OMG IDL format (.idl).
By default it is set to ``false``.

.. _recorder_usage_configuration_mcap:

MCAP Configuration
^^^^^^^^^^^^^^^^^^

The ``enable`` tag allows users to enable or disable whether to record data in an MCAP file.

.. _recorder_usage_configuration_logpublishtime:

Log Publish Time
""""""""""""""""

By default (``log-publish-time: false``) received messages are stored in the MCAP file with ``logTime`` value equals to the reception timestamp.
Additionally, the timestamp corresponding to when messages were initially published (``publishTime``) is also included in the information dumped to MCAP files.
In some applications, it may be required to use the ``publishTime`` as ``logTime``, which can be achieved by providing the ``log-publish-time: true`` configuration option.

.. _recorder_usage_configuration_compression:

Compression
^^^^^^^^^^^
"""""""""""

Compression settings for writing to an MCAP file can be specified under the ``compression`` configuration tag.
The supported compression options are:
Expand Down Expand Up @@ -470,24 +479,58 @@ The supported compression options are:
- ``true`` |br|
``false``

.. _recorder_usage_configuration_recordtypes:
.. _recorder_usage_configuration_resource_limits:

Record Types
^^^^^^^^^^^^
Resource Limits
"""""""""""""""

By default, all type information received during execution is stored in a dedicated MCAP file section.
This information is then leveraged by |ddsreplayer| on playback, publishing recorded types in addition to data samples, which may be required for receiver applications relying on :term:`Dynamic Types<DynamicTypes>` (see :ref:`Replay Types <replayer_replay_configuration_replaytypes>`).
However, a user may choose to disable this feature by setting ``record-types: false``.
The ``resource-limits`` tag allows users to limit the size of the *DDS Recorder's* output.
The ``max-file-size`` tag specifies the maximum size of each output file and the ``max-size`` tag specifies the maximum aggregate size of all output files.
If the ``max-size`` is higher than the ``max-file-size``, the |ddsrecorder| will create multiple files with a maximum size of ``max-file-size``.
By default, however, the ``max-file-size`` is unlimited (``0B``) and the ``max-size`` is the same as the ``max-file-size``; that is, by default the |ddsrecorder| creates a single file of unlimited size.
The ``safety-margin`` tag allows users to set a safety margin (in bytes) used when estimating the size of MCAP files.
By default, the safety margin is set to ``0``.

.. _recorder_usage_configuration_topictypeformat:
.. warning::

Topic type format
If the ``max-file-size`` or the ``max-size`` are set to a value lower than the available space in the disk, the |ddsrecorder| will replace them with the available space in the disk.

To keep the |ddsrecorder| recording after reaching the ``max-size``, users can set the ``file-rotation`` tag to ``true``.
Enabling ``file-rotation`` allows the |ddsrecorder| to overwrite old files to free space for new ones.

.. note::

To keep the |ddsrecorder| from overwriting previous output files, users can set ``{"avoid_overwriting_output": true}`` as the argument (``"args"``) of the ``stop`` command (see :ref:`Control Commands <recorder_remote_controller_data_types>`).
If set, the |ddsrecorder| will not overwrite its output files, at the cost of possibly exceeding the ``max-size``.

.. note::

If an output file is moved, deleted, or renamed, the |ddsrecorder| will keep the size of the file reserved and rotate between the remaining files.

**Example of usage**

.. code-block:: yaml
resource-limits:
max-file-size: 250KB
max-size: 2MiB
file-rotation: true
safety-margin: 10KB
.. _recorder_usage_configuration_sql:

SQL Configuration
^^^^^^^^^^^^^^^^^

The optional ``ros2-types`` tag enables specification of the format for storing schemas.
When set to ``true``, schemas are stored in ROS 2 message format (.msg).
If set to ``false``, schemas are stored in OMG IDL format (.idl).
By default it is set to ``false``.
The ``enable`` tag allows users to enable or disable whether to record data in an SQL database.

.. _recorder_usage_configuration_sql_data_format:

Data Format
"""""""""""

The ``data-format`` tag allows users to specify the format in which data is stored in the SQL database.
The data can be stored in ``cdr`` (which makes the data replayable by the |ddsreplayer|), in ``json`` (which makes the data human-readable), or in ``both`` (default).

.. _recorder_usage_configuration_remote_controller:

Expand Down Expand Up @@ -570,33 +613,6 @@ This improves the performance of the internal data communications.
This value should be set by each user depending on each system characteristics.
In case this value is not set, the default number of threads used is :code:`12`.

Maximum Number of Pending Samples
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It is possible that a |ddsrecorder| starts receiving data from a topic that it has not yet registered, i.e. a topic for which it does not know the data type.
In this case, messages are kept in an internal circular buffer until their associated type information is received, event on which they are written to disk.

However, the recorder execution might end before this event ever occurs.
Depending on configuration (see :ref:`recorder_usage_configuration_onlywithtype`), messages kept in the pending samples buffer will be stored or not on closure.
Hence, note that memory consumption would continuously grow whenever a sample with unknown type information is received.

To avoid the exhaustion of memory resources in such scenarios, a configuration option is provided which lets the user set a limit on memory usage.
The ``max-pending-samples`` parameter allows to configure the size of the aforementioned circular buffers **for each topic** that is discovered.
The default value is equal to ``5000`` samples, with ``-1`` meaning no limit, and ``0`` no pending samples.

Depending on the combination of this configuration option and the value of ``only-with-type``, the following situations may arise when a message with unknown type is received:

* If ``max-pending-samples`` is ``-1``, or if it is greater than ``0`` and the circular buffer is not full, the sample is added to the collection.
* If ``max-pending-samples`` is greater than ``0`` and the circular buffer reaches its maximum capacity, the oldest sample with same type as the received one is popped, and either written without type (``only-with-type: false``) or discarded (``only-with-type: true``).
* If ``max-pending-samples`` is ``0``, the message is written without type if ``only-with-type: false``, and discarded otherwise.

Cleanup Period
^^^^^^^^^^^^^^

As explained in :ref:`Event Window <recorder_usage_configuration_event_window>`, a |ddsrecorder| in paused mode awaits for an event command to write in disk all samples received in the last ``event-window`` seconds.
To accomplish this, received samples are stored in memory until the aforementioned event is triggered and, in order to limit memory consumption, outdated (received more than ``event-window`` seconds ago) samples are removed from this buffer every ``cleanup-period`` seconds.
By default, its value is equal to twice the ``event-window``.

.. _recorder_specs_topic_qos:

QoS
Expand Down Expand Up @@ -837,28 +853,38 @@ A complete example of all the configurations described on this page can be found
- "127.0.0.1"
recorder:
buffer-size: 50
cleanup-period: 90
event-window: 60
max-pending-samples: 10
only-with-type: false
record-types: true
ros2-types: false
output:
filename: "output"
path: "."
timestamp-format: "%Y-%m-%d_%H-%M-%S_%Z"
local-timestamp: false
safety-margin: 500
mcap:
enable: true
log-publish-time: false
resource-limits:
max-file-size: 250KB
max-size: 2MiB
file-rotation: true
safety-margin: 10KB
buffer-size: 50
event-window: 60
log-publish-time: false
only-with-type: false
compression:
algorithm: lz4
level: slowest
force: true
record-types: true
ros2-types: false
compression:
algorithm: lz4
level: slowest
force: true
sql:
enable: false
data-format: "json"
remote-controller:
enable: true
Expand All @@ -869,8 +895,6 @@ A complete example of all the configurations described on this page can be found
specs:
threads: 8
max-pending-samples: 10
cleanup-period: 90
qos:
max-rx-rate: 20
Expand Down

0 comments on commit 2f895b0

Please sign in to comment.