From 666815970d5040f94e4f5f8aa97de5756afad679 Mon Sep 17 00:00:00 2001 From: Eldar Nash Date: Sun, 20 Aug 2023 14:50:54 +0200 Subject: [PATCH 1/4] Minor cleanups on the statistics page --- content/chapter-log-statistics/_index.md | 26 ++++++++---------------- 1 file changed, 8 insertions(+), 18 deletions(-) diff --git a/content/chapter-log-statistics/_index.md b/content/chapter-log-statistics/_index.md index 01222ef5..20ab18d5 100644 --- a/content/chapter-log-statistics/_index.md +++ b/content/chapter-log-statistics/_index.md @@ -4,28 +4,18 @@ weight: 3700 --- -The {{% param "product.abbrev" %}} application collects various statistics and measures different metrics about the messages it receives and delivers. These metrics are collected into different counters, depending on the configuration of {{% param "product.abbrev" %}}. The [`stats-level()` global option]({{< relref "/chapter-global-options/reference-options/_index.md" >}}) determines exactly which statistics {{% param "product.abbrev" %}} collects. You can access these statistics and metrics using the following methods. +The {{% param "product.abbrev" %}} application collects and measures various statistics about the messages it receives and delivers. These statistics are collected into different counters, depending on the configuration of {{% param "product.abbrev" %}}. The [`stats-level()` global option]({{< relref "/chapter-global-options/reference-options/_index.md" >}}) determines exactly which statistics {{% param "product.abbrev" %}} collects. You can access these statistics and metrics using the following methods. +## Structured, selective method -## Recommended: Structured, selective methods: +Using the [`syslog-ng-ctl`]({{< relref "/chapter-quickstart/managing-and-checking-linux/_index.md#stats" >}}) query command. For further information about using `syslog-ng-ctl` commands, see The `syslog-ng` manual pages. - - Using the `monitoring()` source. +## Unstructured, bulk methods - - Using the [`syslog-ng-ctl`]({{< relref "/chapter-quickstart/managing-and-checking-linux/_index.md#stats" >}}) query command. - - For further information about using `syslog-ng-ctl` commands, see The `syslog-ng` manual pages. - - - -## Legacy: Unstructured, bulk methods: +- Using the [`internal()` source]({{< relref "/chapter-log-statistics/log-statistics-internal-source/_index.md" >}}). +- Using the [`syslog-ng-ctl stats`]({{< relref "/chapter-quickstart/managing-and-checking-linux/_index.md#stats" >}}) command. - - Using the [`internal()` source]({{< relref "/chapter-log-statistics/log-statistics-internal-source/_index.md" >}}). - - - Using the [`syslog-ng-ctl stats`]({{< relref "/chapter-quickstart/managing-and-checking-linux/_index.md#stats" >}}) command. - For further information about using `syslog-ng-ctl` commands, see The `syslog-ng` manual pages. - - Use the `socat` application: `echo STATS | socat -vv UNIX-CONNECT:/opt/syslog-ng/var/run/syslog-ng.ctl -` - - - If you have an OpenBSD-style `netcat` application installed, use the `echo STATS | nc -U /opt/syslog-ng/var/run/syslog-ng.ctl` command. Note that the `netcat` included in most Linux distributions is a GNU-style version that is not suitable to query the statistics of `syslog-ng`. - +- Use the `socat` application: `echo STATS | socat -vv UNIX-CONNECT:/opt/syslog-ng/var/run/syslog-ng.ctl -` +- If you have an OpenBSD-style `netcat` application installed, use the `echo STATS | nc -U /opt/syslog-ng/var/run/syslog-ng.ctl` command. Note that the `netcat` included in most Linux distributions is a GNU-style version that is not suitable to query the statistics of `syslog-ng`. From ba9184015db7af6d28a6693be6858c28f06b0ec5 Mon Sep 17 00:00:00 2001 From: Eldar Nash Date: Sun, 20 Aug 2023 15:04:04 +0200 Subject: [PATCH 2/4] Adds draft a chapter about Prometheus metrics --- content/chapter-log-statistics/_index.md | 4 + .../chunk/option-destination-diskbuffer.md | 19 + content/metrics/_index.md | 82 ++ content/metrics/reference-metrics/_index.md | 728 ++++++++++++++++++ 4 files changed, 833 insertions(+) create mode 100644 content/metrics/_index.md create mode 100644 content/metrics/reference-metrics/_index.md diff --git a/content/chapter-log-statistics/_index.md b/content/chapter-log-statistics/_index.md index 20ab18d5..af2b5405 100644 --- a/content/chapter-log-statistics/_index.md +++ b/content/chapter-log-statistics/_index.md @@ -4,6 +4,10 @@ weight: 3700 --- +{{% alert title="Note" color="info" %}} +This chapter describes the legacy statistics of {{% param "product.abbrev" %}}. For the Prometheus-like metrics framework, see {{% xref "/metrics/_index.md" %}}. +{{% /alert %}} + The {{% param "product.abbrev" %}} application collects and measures various statistics about the messages it receives and delivers. These statistics are collected into different counters, depending on the configuration of {{% param "product.abbrev" %}}. The [`stats-level()` global option]({{< relref "/chapter-global-options/reference-options/_index.md" >}}) determines exactly which statistics {{% param "product.abbrev" %}} collects. You can access these statistics and metrics using the following methods. ## Structured, selective method diff --git a/content/headless/chunk/option-destination-diskbuffer.md b/content/headless/chunk/option-destination-diskbuffer.md index 79c53548..3a3e3be9 100644 --- a/content/headless/chunk/option-destination-diskbuffer.md +++ b/content/headless/chunk/option-destination-diskbuffer.md @@ -100,6 +100,25 @@ Available in {{% param "product.abbrev" %}} 4.0 and later. Hazard of data loss! If you change the value of `reliable()` option when there are messages in the disk-buffer, the messages stored in the disk-buffer will be lost. {{% /alert %}} +## stats() + +| Type: | `number [minutes]` | +|--------------|-----------| +| Default: | 5 | + +Available in {{% param "product.abbrev" %}} 4.2 and later. + +*Description:* Set the frequency of generating [disk-buffer related metrics]({{< relref "/metrics/reference-metrics/_index.md#disk-buffer-metrics" >}}). Set it to zero to disable metrics. + +```shell +options { + disk-buffer( + stats( + freq(10) + ) + ); +};``` + ### truncate-size-ratio() {#diskbuf-trunkate-size-ratio} | Type: | number((between 0 and 1)) | diff --git a/content/metrics/_index.md b/content/metrics/_index.md new file mode 100644 index 00000000..32b48621 --- /dev/null +++ b/content/metrics/_index.md @@ -0,0 +1,82 @@ +--- +title: Prometheus-like metrics +weight: 3800 +type: docs +cascade: + type: docs +--- + +Starting with version 4.0, {{% param "product.abbrev" %}} provides a new metric naming scheme (labeling capability) to provide metrics and statistics about the processed events. You can query the metrics using `syslog-ng-ctl`, which can produce Prometheus-like metric exports. + + + +Metrics are identified by names and partitioned by labels, similarly to the [Prometheus data model](https://prometheus.io/docs/concepts/data_model/), in the following format: `{, } ` + +The part with the curly braces is optional, and not used for certain metrics. For example: + +```shell +syslogng_events_allocated_bytes 18446744073709550560 +syslogng_filtered_events_total{id="#anon-filter0",result="matched"} 0 +``` + +Note that: + +- The order of the labels doesn't matter. +- A label with an empty value is equivalent to a non-existing label. +- If the related queue is specific to a worker, it also has the `worker` label. +- If there is no information about the worker of the driver, it has the `abandoned=true` label. + +For example: + +```shell +syslog-ng-ctl stats prometheus +``` + +The output is similar to: + +```shell +syslogng_events_allocated_bytes 18446744073709550560 +syslogng_filtered_events_total{id="#anon-filter0",result="matched"} 0 +syslogng_filtered_events_total{id="#anon-filter0",result="not_matched"} 2 +syslogng_filtered_events_total{id="ff",result="matched"} 0 +syslogng_filtered_events_total{id="ff",result="not_matched"} 2 +syslogng_input_events_total{id="#anon-source0#0",driver_instance="-",result="processed"} 0 +syslogng_input_events_total{id="s_network#1",result="processed"} 2 +syslogng_internal_source{result="dropped"} 0 +syslogng_internal_source{result="queued"} 0 +syslogng_output_events_total{id="d_dest#0",driver_instance="tcp,127.0.0.1:5555",result="delivered"} 0 +syslogng_output_events_total{id="d_dest#0",driver_instance="tcp,127.0.0.1:5555",result="dropped"} 0 +syslogng_output_events_total{id="d_dest#0",driver_instance="tcp,127.0.0.1:5555",result="queued"} 0 +syslogng_output_events_total{id="d_dest#1",driver_instance="http,https://localhost:8080",result="delivered"} 0 +syslogng_output_events_total{id="d_dest#1",driver_instance="http,https://localhost:8080",result="dropped"} 0 +syslogng_output_events_total{id="d_dest#1",driver_instance="http,https://localhost:8080",result="queued"} 0 +syslogng_parsed_events_total{id="#anon-parser0",result="discarded"} 0 +syslogng_parsed_events_total{id="#anon-parser0",result="processed"} 0 +syslogng_scratch_buffers_bytes 0 +syslogng_scratch_buffers_total 3 +syslogng_tagged_events_total{id=".source.#anon-source0",result="processed"} 0 +syslogng_tagged_events_total{id=".source.s_network",result="processed"} 2 +``` + +## Legacy metrics + +You can access the [old statistics]({{< relref "/chapter-log-statistics/_index.md" >}}) of {{% param "product.abbrev" %}} in the new format by running: + +```shell +syslog-ng-ctl stats prometheus --with-legacy-metrics +``` + +The output should be similar to: + +```shell +syslogng_src_facility_processed{stat_instance="18"} 0 +syslogng_dst_network_memory_usage{id="d_dest#0",stat_instance="tcp,127.0.0.1:5555"} 0 +syslogng_src_host_processed{id="s_network#1",stat_instance="AnnoW"} 2 +syslogng_global_sdata_updates_processed 0 +syslogng_global_msg_clones_processed 0 +syslogng_global_internal_queue_length_processed 0 +syslogng_global_payload_reallocs_processed 4 +... +``` + +For a detailed list of available metrics, see {{% xref "/metrics/reference-metrics/_index.md" %}}. diff --git a/content/metrics/reference-metrics/_index.md b/content/metrics/reference-metrics/_index.md new file mode 100644 index 00000000..d0551614 --- /dev/null +++ b/content/metrics/reference-metrics/_index.md @@ -0,0 +1,728 @@ +--- +title: Metrics reference +--- + +This page lists the different metrics available in {{% param "product.abbrev" %}}. Note that although the metrics framework was introduced in {{% param "product.abbrev" %}} version 4.0, some of the metrics were added in later versions. + + + + + + + +## Configuration related metrics + +### last_config_file_modification_timestamp_seconds + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 0 | +| Labels: | | + +*Description:* + +*Example:* + +```json +syslogng_last_config_file_modification_timestamp_seconds 1681309877 +``` + +### last_config_reload_timestamp_seconds + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 0 | +| Labels: | | + +*Description:* + +*Example:* + +```json +syslogng_last_config_reload_timestamp_seconds 1681309903 +``` + +### last_successful_config_reload_timestamp_seconds + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 0 | +| Labels: | | + +*Description:* + +*Example:* + +```json +syslogng_last_successful_config_reload_timestamp_seconds 1681309758 +``` + +## Disk-buffer metrics + +Disk-buffer metrics are available for abandoned disk-buffer files as well. Metrics for abandoned disk-buffer files always include the `abandoned="true"` label. The `reliable` label shows whether the disk-buffer `reliable()` option is enabled or not. + +### capacity_bytes + +| | | +| -------- | ----------------------- | +| Type: | | +| Level: | 1 | +| Labels: | `driver_id`, `driver_instance`, `path`, `reliable` | + +*Description:* The theoretical maximal useful size of the disk-buffer. This is always smaller, than `disk-buf-size()`, as there is some reserved space for metadata. The actual full disk-buffer file can be larger than this, as syslog-ng allows to write over this limit once, at the end of the file. + +*Example:* + +```json +syslogng_disk_queue_capacity_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 104853504 +``` + +### dir_available_bytes + +| | | +| -------- | ----------------------- | +| Type: | | +| Level: | 1 | +| Labels: | `dir` | + +*Description:* Metric for monitoring the available space in disk-buffer directories (set with the `dir()` option of the disk buffer). + +*Example:* + +```json +syslogng_disk_queue_dir_available_bytes{dir="/var/syslog-ng"} 870109413376 +``` + +### disk_allocated_bytes + +| | | +| -------- | ----------------------- | +| Type: | | +| Level: | 1 | +| Labels: | `driver_id`, `driver_instance`, `path`, `reliable` | + +*Description:* The current size of the disk-buffer file on the disk. Note that the disk-buffer file size does not strictly correlate with the number of messages, as it is a ring buffer implementation, and also {{% param "product.abbrev" %}} optimizes the truncation of the file for performance reasons. + +*Example:* + +```json +syslogng_disk_queue_disk_allocated_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 17284 + +``` + +### disk_usage_bytes + +| | | +| -------- | ----------------------- | +| Type: | | +| Level: | 1 | +| Labels: | `driver_id`, `driver_instance`, `path`, `reliable` | + +*Description:* The serialized size of the queued messages in the disk-buffer file. This counter is useful for calculating the disk usage percentage (disk_usage_bytes / capacity_bytes) or the remaining available space (capacity_bytes - disk_usage_bytes). + +*Example:* + +```json +syslogng_disk_queue_disk_usage_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 13188 +``` + +### events + +| | | +| -------- | ----------------------- | +| Type: | | +| Level: | 1 | +| Labels: | `driver_instance`, `id`, `path`, `reliable` | + +*Description:* + +*Example:* + +```json +syslogng_disk_queue_events{driver_instance="tcp,localhost:1235",id="d_network_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.qf",reliable="false"} 101 +``` + +### memory_usage_bytes + +| | | +| -------- | ----------------------- | +| Type: | | +| Level: | 1 | +| Labels: | `driver_instance`, `id`, `path`, `reliable`, `worker` | + +*Description:* + +*Example:* + +```json +syslogng_disk_queue_memory_usage_bytes{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true",worker="0"} 3136 +``` + +## Incoming and outgoing event metrics + +Available for the `file()`, `http()`, `network()`, `syslog()` drivers. + +### input_event_bytes_total + +| | | +| -------- | ----------------------- | +| Type: | | +| Level: | | +| Labels: | `driver_instance`, `id` | + +*Description:* Shows the serialized message size (without protocol-specific header/framing/similar) in bytes for the incoming messages. + +*Example:* + +```json +syslogng_input_event_bytes_total{id="s_network#0",driver_instance="tcp,127.0.0.1"} 1925529600 +``` + +### output_event_bytes_total + +| | | +| -------- | ----------------------- | +| Type: | | +| Level: | | +| Labels: | `driver_instance`, `id` | + +*Description:* Shows the serialized message size (without protocol-specific header/framing/similar) in bytes for the outgoing messages. + +*Example:* + +```json +syslogng_output_event_bytes_total{id="d_network#0",driver_instance="tcp,127.0.0.1:5555"} 565215232 +syslogng_output_event_bytes_total{id="d_http#0",driver_instance="http,http://127.0.0.1:8080/"} 1024 +``` + +## `internal()` metrics + +### internal_events_total + +| | | +| -------- | ----------------------- | +| Type: | L | +| Level: | 0 | +| Labels: | `result=queued`, `result=dropped`, `result=processed` | + +*Description:* + +*Example:* + +```json + +``` + +### internal_events_queue_capacity + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 0 | +| Labels: | | + +*Description:* + +*Example:* + +```json + +``` + +## `kubernetes()` metrics + +### input_event_bytes_total + +| | | +| -------- | ----------------------- | +| Type: | | +| Level: | | +| Labels: | `cluster`, `driver`, `id`, `namespace`, `pod` | + +*Description:* + +*Example:* + +```json +syslogng_input_event_bytes_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1682517834-7797487dcc-49hqc"} 1859 +``` + +### input_events_total + +| | | +| -------- | ----------------------- | +| Type: | | +| Level: | | +| Labels: | `cluster`, `driver`, `id`, `namespace`, `pod` | + +*Description:* + +*Example:* + +```json +syslogng_input_events_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1682517834-7797487dcc-49hqc"} 25 +``` + +## Memory queue metrics + +### events + +| | | +| -------- | ----------------------- | +| Type: | | +| Level: | | +| Labels: | `driver_instance`, `id` | + +*Description:* + +*Example:* + +```json +syslogng_memory_queue_events{driver_instance="tcp,localhost:1234",id="d_network#0"} 29 +``` + +### memory_usage_bytes + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | `driver_instance`, `id`, `worker` | + +*Description:* + +*Example:* + +```json +syslogng_memory_queue_memory_usage_bytes{driver_instance="http,http://localhost:1236",id="d_http#0",worker="0"} 5896 +``` + +## Message delay related metrics + +### output_message_delay_sample_age_seconds + +| | | +| -------- | ----------------------- | +| Type: | | +| Level: | | +| Labels: | | + +*Description:* Shows when the delay metric was last sampled. It contains the age of the sample relative to the current time. This is a relative timestamp, so the accuracy of clock on the host running the {{% param "product.ose" %}} doesn't matter. The actual delay is stored in `output_message_delay_sample_seconds` + +*Example:* + +```json +syslogng_output_message_delay_sample_seconds{driver="http",url="https://opensearch-cluster-master:9200/_bulk",id="#anon-destination0#0",worker="2"} 841.38900000000001 +``` + +### output_message_delay_sample_seconds + +| | | +| -------- | ----------------------- | +| Type: | | +| Level: | | +| Labels: | | + +*Description:* Shows the delay between the time when {{% param "product.ose" %}} received the message and when {{% param "product.ose" %}} sent the message to its destination. This is a sampled metric, not an average: it shows the delay associated with a specific message that was recently sent out. It is assumed that messages sitting in the queue in close proximity have very similar delay values, so {{% param "product.ose" %}} samples this delay once every second. + +*Example:* + +```json + +``` + +## TCP connection metrics + +These metrics are available for the `network()` and `syslog()` drivers. + +### socket_connections + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | `id`, `driver_instance`, `direction` | + +*Description:* + +*Example:* + +```json +syslogng_socket_connections{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 3 +``` + +### socket_max_connections + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 0 | +| Labels: | `id`, `driver_instance`, `direction` | + +*Description:* + +*Example:* + +```json +syslogng_socket_max_connections{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 10 +``` + +### socket_receive_dropped_packets_total + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | `id`, `driver_instance`, `direction` | + +*Description:* + +*Example:* + +```json +syslogng_socket_receive_dropped_packets_total{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 619173 +``` + +### socket_receive_buffer_max_bytes + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | `id`, `driver_instance`, `direction` | + +*Description:* + +*Example:* + +```json +syslogng_socket_receive_buffer_max_bytes{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 268435456 +``` + +### socket_receive_buffer_used_bytes + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | `id`, `driver_instance`, `direction` | + +*Description:* + +*Example:* + +```json +syslogng_socket_receive_buffer_used_bytes{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 0 +``` + +### socket_rejected_connections_total + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | `id`, `driver_instance`, `direction` | + +*Description:* + +*Example:* + +```json +syslogng_socket_rejected_connections_total{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 96 +``` + +## Unsorted + +### classified_events_total + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 0 | +| Labels: | `app`, `host`, `program`, `source` | + +*Description:* + +*Example:* + +```json + +``` + +### events_allocated_bytes + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | | + +*Description:* + +*Example:* + +```json +syslogng_events_allocated_bytes 11639868 +``` + +### filtered_events_total + +| | | +| -------- | ----------------------- | +| Type: | L | +| Level: | 1 | +| Labels: | `id`, `result=matched`, `result=not_matched` | + +*Description:* + +*Example:* + +```json +syslogng_filtered_events_total{id="#anon-filter0",result="matched"} 2369 +syslogng_filtered_events_total{id="#anon-filter0",result="not_matched"} 0 +``` + +### input_events_total + +| | | +| -------- | ----------------------- | +| Type: | L | +| Level: | 1 | +| Labels: | `driver_instance`, `id` | + +*Description:* + +*Example:* + +```json +syslogng_input_events_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1692522870-6966655876-c57dx"} 1367 +``` + +### input_event_bytes_total + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | `driver_instance`, `id` | + +*Description:* + +*Example:* + +```json +syslogng_input_event_bytes_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="kube-system",pod="kube-controller-manager-minikube"} 36993 +``` + +### io_worker_latency_seconds + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | | + +*Description:* + +*Example:* + +```json +syslogng_io_worker_latency_seconds 7.4839999999999995e-05 +``` + +### mainloop_io_worker_roundtrip_latency_seconds + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | | + +*Description:* + +*Example:* + +```json +syslogng_mainloop_io_worker_roundtrip_latency_seconds 0.00012654699999999999 +``` + +### output_events_bytes_total + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | `driver_instance`, `id` | + +*Description:* + +*Example:* + +```json +``` + +### output_events_total + +| | | +| -------- | ----------------------- | +| Type: | L | +| Level: | 1 | +| Labels: | `driver_instance`, `id`, `result=queued`, `result=dropped`, `result=suppressed`, `result=delivered` | + +*Description:* + +*Example:* + +```json +syslogng_output_events_total{driver="http",url="https://opensearch-cluster-master:9200/_bulk",id="#anon-destination0#0",result="dropped"} 0 +syslogng_output_events_total{driver="http",url="https://opensearch-cluster-master:9200/_bulk",id="#anon-destination0#0",result="queued"} 2369 +syslogng_output_events_total{driver="http",url="https://opensearch-cluster-master:9200/_bulk",id="#anon-destination0#0",result="delivered"} 0 +``` + +### parsed_events_total + +| | | +| -------- | ----------------------- | +| Type: | L | +| Level: | 1 | +| Labels: | `id`, `result=discarded`, `result=processed` | + +*Description:* + +*Example:* + +```json +``` + +### route_egress_total + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | `id` | + +*Description:* + +*Example:* + +```json +``` + +### route_ingress_total + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | `id` | + +*Description:* + +*Example:* + +```json +``` + +### scratch_buffers_bytes + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 0 | +| Labels: | | + +*Description:* + +*Example:* + +```json +syslogng_scratch_buffers_bytes 0 +``` + +### scratch_buffers_count + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 0 | +| Labels: | | + +*Description:* + +*Example:* + +```json +syslogng_scratch_buffers_count 206 +``` + +### tagged_events_total + +| | | +| -------- | ----------------------- | +| Type: | L | +| Level: | 3 | +| Labels: | `id` | + +*Description:* + +*Example:* + +```json + +``` + +### tls_handshake_errors_total + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | `direction`, `driver_instance`, `id`, `transport=syslog` | + +*Description:* + +*Example:* + +```json + +``` + +### transport_syslog_framing_errors_total + +| | | +| -------- | ----------------------- | +| Type: | S | +| Level: | 1 | +| Labels: | `driver_instance`, `id` | + +*Description:* + +*Example:* + +```json + +``` \ No newline at end of file From cb96ebe5aed8c447206cb3fefaf9d4362c6e10fc Mon Sep 17 00:00:00 2001 From: Eldar Nash Date: Sun, 20 Aug 2023 15:24:26 +0200 Subject: [PATCH 3/4] Mention metrics in syslog-ng-ctl manpage --- content/app-man-syslog-ng/syslog-ng-ctl.1.md | 50 ++++++++++++++++++-- 1 file changed, 46 insertions(+), 4 deletions(-) diff --git a/content/app-man-syslog-ng/syslog-ng-ctl.1.md b/content/app-man-syslog-ng/syslog-ng-ctl.1.md index 8fbc5a82..3efc998a 100644 --- a/content/app-man-syslog-ng/syslog-ng-ctl.1.md +++ b/content/app-man-syslog-ng/syslog-ng-ctl.1.md @@ -190,7 +190,12 @@ The `syslog-ng-ctl query get` command has the following options: `stats [options]` -Use the `stats` command to display statistics about the processed messages. For details about the displayed statistics, see [The {{% param "product.ose" %}} documentation](https://axoflow.com/). The `stats` command has the following options: +Use the `stats` command to display statistics and metrics about the processed messages. + +- For details about the available Prometheus-like metrics, see {{% xref "/metrics/reference-metrics/_index.md" %}}. You can retrieve these metrics with the [`syslog-ng-ctl stats prometheus` command](#syslog-ng-ctl-stats-prometheus). +- For details about legacy statistics, see {{% xref "/chapter-log-statistics/_index.md" %}}. + +The `stats` command has the following options: - `--control=` or `-c` @@ -202,7 +207,7 @@ Use the `stats` command to display statistics about the processed messages. For - `--remove-orphans` - Safely removes all counters that are not referenced by any syslog-ng stat producer objects. + Safely removes all counters that are not referenced by any statistics producer objects. The flag can be used to prune dynamic and static counters manually. This is useful, for example, when a templated file destination produces a lot of stats: @@ -216,8 +221,7 @@ Use the `stats` command to display statistics about the processed messages. For The `stats-lifetime()` can be used to do the same automatically and periodically, but currently stats-lifetime() removes only dynamic counters that have a timestamp field set. {{% /alert %}} - -### Example +### Example for statistics ```shell syslog-ng-ctl stats @@ -255,6 +259,44 @@ center;;queued;a;processed;0 destination;df_facility_dot_err;;a;processed;0 ``` +For details about legacy statistics, see {{% xref "/chapter-log-statistics/_index.md" %}}. + + + +### Example for statistics + +```shell +syslog-ng-ctl stats prometheus +``` + +The output is similar to: + +```shell +syslogng_events_allocated_bytes 18446744073709550560 +syslogng_filtered_events_total{id="#anon-filter0",result="matched"} 0 +syslogng_filtered_events_total{id="#anon-filter0",result="not_matched"} 2 +syslogng_filtered_events_total{id="ff",result="matched"} 0 +syslogng_filtered_events_total{id="ff",result="not_matched"} 2 +syslogng_input_events_total{id="#anon-source0#0",driver_instance="-",result="processed"} 0 +syslogng_input_events_total{id="s_network#1",result="processed"} 2 +syslogng_internal_source{result="dropped"} 0 +syslogng_internal_source{result="queued"} 0 +syslogng_output_events_total{id="d_dest#0",driver_instance="tcp,127.0.0.1:5555",result="delivered"} 0 +syslogng_output_events_total{id="d_dest#0",driver_instance="tcp,127.0.0.1:5555",result="dropped"} 0 +syslogng_output_events_total{id="d_dest#0",driver_instance="tcp,127.0.0.1:5555",result="queued"} 0 +syslogng_output_events_total{id="d_dest#1",driver_instance="http,https://localhost:8080",result="delivered"} 0 +syslogng_output_events_total{id="d_dest#1",driver_instance="http,https://localhost:8080",result="dropped"} 0 +syslogng_output_events_total{id="d_dest#1",driver_instance="http,https://localhost:8080",result="queued"} 0 +syslogng_parsed_events_total{id="#anon-parser0",result="discarded"} 0 +syslogng_parsed_events_total{id="#anon-parser0",result="processed"} 0 +syslogng_scratch_buffers_bytes 0 +syslogng_scratch_buffers_total 3 +syslogng_tagged_events_total{id=".source.#anon-source0",result="processed"} 0 +syslogng_tagged_events_total{id=".source.s_network",result="processed"} 2 +``` + +For details about the available Prometheus-like metrics, see {{% xref "/metrics/reference-metrics/_index.md" %}}. + ## Handling password-protected private keys From d6ed81f0be270c1cdf46e027663a09311f242e66 Mon Sep 17 00:00:00 2001 From: Eldar Nash Date: Tue, 17 Oct 2023 15:49:43 +0200 Subject: [PATCH 4/4] Minor fixes and notes --- content/metrics/reference-metrics/_index.md | 45 ++++----------------- 1 file changed, 8 insertions(+), 37 deletions(-) diff --git a/content/metrics/reference-metrics/_index.md b/content/metrics/reference-metrics/_index.md index d0551614..ae3d6960 100644 --- a/content/metrics/reference-metrics/_index.md +++ b/content/metrics/reference-metrics/_index.md @@ -5,8 +5,7 @@ title: Metrics reference This page lists the different metrics available in {{% param "product.abbrev" %}}. Note that although the metrics framework was introduced in {{% param "product.abbrev" %}} version 4.0, some of the metrics were added in later versions. @@ -183,7 +182,13 @@ syslogng_disk_queue_memory_usage_bytes{driver_instance="http,http://localhost:12 ## Incoming and outgoing event metrics -Available for the `file()`, `http()`, `network()`, `syslog()` drivers. +Available for the `file()`, `http()`, `kubernetes()`, `network()`, `otel()`, `syslog()` drivers. + +Note that metrics of the `kubernetes()` source (`driver="kubernetes"`) have some special labels: `cluster`, `namespace`, `pod`, for example: + +```json +syslogng_input_event_bytes_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1682517834-7797487dcc-49hqc"} 1859 +``` ### input_event_bytes_total @@ -252,40 +257,6 @@ syslogng_output_event_bytes_total{id="d_http#0",driver_instance="http,http://127 ``` -## `kubernetes()` metrics - -### input_event_bytes_total - -| | | -| -------- | ----------------------- | -| Type: | | -| Level: | | -| Labels: | `cluster`, `driver`, `id`, `namespace`, `pod` | - -*Description:* - -*Example:* - -```json -syslogng_input_event_bytes_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1682517834-7797487dcc-49hqc"} 1859 -``` - -### input_events_total - -| | | -| -------- | ----------------------- | -| Type: | | -| Level: | | -| Labels: | `cluster`, `driver`, `id`, `namespace`, `pod` | - -*Description:* - -*Example:* - -```json -syslogng_input_events_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1682517834-7797487dcc-49hqc"} 25 -``` - ## Memory queue metrics ### events