[System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

tetianakravchenko · 2023-05-08T14:25:32Z

What does this PR do?

Add/adjust dimensions fields:

⛔ (when adding core.id fields all documents are dropped, need to double-check, will add in different PR) core - add standard fields + core.id
- Verification of the count of documents (before & after TSDB enablement) in Discover Interface
- For every field annotated with dimension:true, there exists time_series_dimension: true annotation in the index
✅ cpu - adjust list of fields according to [ECS] [TSDB] Centralisation of Dimension Fields #5193 (comment).
- Verification of the count of documents (before & after TSDB enablement) in Discover Interface ✅
- For every field annotated with dimension:true, there exists time_series_dimension: true annotation in the index ✅
✅ diskio - add standard fields + system.diskio.name
- Verification of the count of documents (before & after TSDB enablement) in Discover Interface ✅
- For every field annotated with dimension:true, there exists time_series_dimension: true annotation in the index ✅
✅ (retest) filesystem - add standard fields + system.filesystem.device_name and system.filesystem.mount_point
- Verification of the count of documents (before & after TSDB enablement) in Discover Interface ✅
- For every field annotated with dimension:true, there exists time_series_dimension: true annotation in the index ✅-
✅ fsstat - add standard fields
- Verification of the count of documents (before & after TSDB enablement) in Discover Interface ✅
- For every field annotated with dimension:true, there exists time_series_dimension: true annotation in the index ✅
✅ load - adjust list of fields according to [ECS] [TSDB] Centralisation of Dimension Fields #5193 (comment).
- Verification of the count of documents (before & after TSDB enablement) in Discover Interface ✅
- For every field annotated with dimension:true, there exists time_series_dimension: true annotation in the index ✅
✅ memory - adjust list of fields according to [ECS] [TSDB] Centralisation of Dimension Fields #5193 (comment).
- Verification of the count of documents (before & after TSDB enablement) in Discover Interface ✅
- For every field annotated with dimension:true, there exists time_series_dimension: true annotation in the index ✅
⛔ network - add standard fields + system.network.name - will be added in separate PR
- Verification of the count of documents (before & after TSDB enablement) in Discover Interface ✅
- For every field annotated with dimension:true, there exists time_series_dimension: true annotation in the index ✅
⛔ process - add standard fields + process.pid- will be added in separate PR
- Verification of the count of documents (before & after TSDB enablement) in Discover Interface
- For every field annotated with dimension:true, there exists time_series_dimension: true annotation in the index
✅ process_summary - add standard fields
- Verification of the count of documents (before & after TSDB enablement) in Discover Interface ✅
- For every field annotated with dimension:true, there exists time_series_dimension: true annotation in the index ✅
✅ socket_summary - add standard fields
- Verification of the count of documents (before & after TSDB enablement) in Discover Interface ✅
- For every field annotated with dimension:true, there exists time_series_dimension: true annotation in the index ✅
✅ uptime - add standard fields
- Verification of the count of documents (before & after TSDB enablement) in Discover Interface ✅
- For every field annotated with dimension:true, there exists time_series_dimension: true annotation in the index ✅

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

Signed-off-by: Tetiana Kravchenko <[email protected]>

elasticmachine · 2023-05-08T14:36:05Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-05-31T09:22:26.110+0000
Duration: 15 min 4 sec

Test stats 🧪

Test	Results
Failed	0
Passed	145
Skipped	0
Total	145

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

Signed-off-by: Tetiana Kravchenko <[email protected]>

elasticmachine · 2023-05-08T15:10:44Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`3/3`)	💚
Files	100.0% (`4/4`)	💚
Classes	100.0% (`4/4`)	💚
Methods	60.759% (`48/79`)	❕
Lines	99.536% (`2790/2803`)	❕
Conditionals	100.0% (`0/0`)	💚

Signed-off-by: Tetiana Kravchenko <[email protected]>

agithomas

system.process.cmdline should be considered as a dimension field.

agithomas · 2023-05-09T13:29:28Z

system.process.cmdline should be considered as a dimension field.

Please ignore the above review comment.

lalit-satapathy · 2023-05-10T13:35:26Z

CC: @ruflin current system package migration PR.

tetianakravchenko · 2023-05-10T13:40:50Z

packages/system/data_stream/process/fields/agent.yml

@@ -202,6 +209,13 @@
  description: Process metrics.
  type: group
  fields:
+    - name: pid


NOTE: it implies that new time series will be created when restarting a process, the same for k8s environment since we have this setting in manifest: https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-managed-kubernetes.yaml#L29

lalit-satapathy · 2023-05-10T14:03:14Z

packages/system/data_stream/process/fields/agent.yml

@@ -202,6 +209,13 @@
  description: Process metrics.
  type: group
  fields:
+    - name: pid


Is this a field, for which mapping was completely missing before?

@lalit-satapathy yes. this field is defined in ecs https://www.elastic.co/guide/en/ecs/current/ecs-process.html
note: there are other process.* fields in document, I've added here only the relevant for this PR field
document example with process.* fields:

ruflin · 2023-05-11T07:47:12Z

packages/system/docs/README.md

@@ -1357,7 +1358,7 @@ This data should be available without elevated permissions.
 | host.os.full.text | Multi-field of `host.os.full`. | match_only_text |  |  |
 | host.os.kernel | Operating system kernel version as a raw string. | keyword |  |  |
 | host.os.name | Operating system name, without the version. | keyword |  |  |
-| host.os.name.text | Multi-field of `host.os.name`. | match_only_text |  |  |
+| host.os.name.text | Multi-field of `host.os.name`. | text |  |  |


Where is this change coming from?

I didn't touch any ecs fields in this PR, I thought that this is coming from the ecs definition, when running elastic-package build
but, you are right - it should still be match_only_text from the ecs fields definition - https://github.com/elastic/ecs/blob/8.0/experimental/generated/beats/fields.ecs.yml#L3857-L3866

it appeared that this field is defined in fields folder twice - in agent.yml

integrations/packages/system/data_stream/diskio/fields/agent.yml

Lines 159 to 169 in 72dfc88

- name: os.name

level: extended

type: keyword

ignore_above: 1024

multi_fields:

- name: text

type: text

norms: false

default_field: false

description: Operating system name, without the version.

example: Mac OS X

and in ecs.yml -

integrations/packages/system/data_stream/diskio/fields/ecs.yml

Lines 19 to 20 in 72dfc88

- external: ecs

name: host.os.name

I will remove the custom definition

done - b762ecc

there are lots of duplicated fields, to not mix up fields clean up and tsdb dimensions pr - I am not going to introduce more fields removal here. @elastic/elastic-agent-data-plane could you please take care of it?

I think that elastic-package check could verify that there are no duplicated fields to avoid such situations, wdyt? should it be created as a feature request for elastic-package?

FYI: elastic/elastic-package#1287, there is already available a check for the duplicated fields, it is only needed to update the format_version

ruflin · 2023-05-11T07:47:52Z

packages/system/docs/README.md

@@ -1733,7 +1737,7 @@ This data should be available without elevated permissions.
 | host.mac | Host mac addresses. | keyword |  |  |
 | host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword |  |  |
 | host.network.in.bytes | The number of bytes received on all network interfaces by the host in a given period of time. | scaled_float | byte | counter |
-| host.network.in.packets | The number of packets received on all network interfaces by the host in a given period of time. | long |  |  |
+| host.network.in.packets | The number of packets received on all network interfaces by the host in a given period of time. | scaled_float |  | counter |


This looks off. Why is this suddenly a scaled float but at the same time a counter which I assume is always a long ?

the same here: the same field defined twice: once in fields.yml:

integrations/packages/system/data_stream/network/fields/fields.yml

Lines 56 to 62 in 72dfc88

- name: network.in.bytes

type: scaled_float

format: bytes

unit: byte

metric_type: counter

description: |

The number of bytes received on all network interfaces by the host in a given period of time.

another one in agent.yml:

integrations/packages/system/data_stream/network/fields/agent.yml

Lines 206 to 210 in 72dfc88

- name: network.in.bytes

type: long

format: bytes

description: >

The number of bytes received on all network interfaces by the host in a given period of time.

I am not sure what is a correct 😕 Could be that the files evaluation sequence changed that caused this

@elastic/elastic-agent-data-plane I need some help here, what is the correct value for this field?

I'm tempted to say the one in fields.yml is wrong. Two questions:

Can you check what is loaded today if you install the package? Hopefully long.

Can you check what we have in metricbeat for the same?

If we load long today and it is also the one used in Metricbeat, lets fix the fields.yml file. Otherwise lets continue the discussion.

Can you check what is loaded today if you install the package? Hopefully long.

long

Can you check what we have in metricbeat for the same?

also long - https://github.com/elastic/beats/blob/main/metricbeat/module/system/network/_meta/fields.yml#L30-L33

I've reverted network changes for this PR, will create an another one

Signed-off-by: Tetiana Kravchenko <[email protected]>

ruflin · 2023-05-31T08:38:37Z

Could we take out the process and network changes to get this in quickly and follow up with the other two separately?

Signed-off-by: Tetiana Kravchenko <[email protected]>

tetianakravchenko · 2023-05-31T09:25:47Z

Could we take out the process and network changes to get this in quickly and follow up with the other two separately?

@ruflin done, process and network data_stream changes were reverted

ruflin

LGTM

As this PR does mostly add dimensions + some cleanups, this should be low risk.

ruflin · 2023-05-31T09:45:11Z

packages/system/docs/README.md

@@ -1350,7 +1351,7 @@ This data should be available without elevated permissions.
 | host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword |  |  |
 | host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword |  |  |
 | host.ip | Host ip addresses. | ip |  |  |
-| host.mac | Host MAC addresses. The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen. | keyword |  |  |


Not sure where this change is coming from but I assume pulled in from some ECS changes.

the same issue as above - there are duplication one in ecs file, another in

integrations/packages/system/data_stream/diskio/fields/agent.yml

Lines 134 to 138 in 1314197

- name: mac

level: core

type: keyword

ignore_above: 1024

description: Host mac addresses.

Lets ignore it for now.

ruflin

LGTM

As this PR does mostly add dimensions + some cleanups, this should be low risk.

ruflin

LGTM

As this PR does mostly add dimensions + some cleanups, this should be low risk.

agithomas · 2023-05-31T09:49:24Z

Referring to #5193 (comment), please include agent.id as the dimension field.

agithomas · 2023-05-31T09:52:33Z

Referring to #5193 (comment), please include agent.id as the dimension field.

I see it is already handled. Please ignore this

agithomas

LGTM!

tetianakravchenko · 2023-05-31T10:04:26Z

@elastic/elastic-agent-data-plane could you please review this PR? I can't merge it since Code owner review required

lalit-satapathy · 2023-05-31T11:07:18Z

@tetianakravchenko,

Can we make this beta and update the stack.version as in this PR: #6129

lalit-satapathy · 2023-05-31T11:10:48Z

@tetianakravchenko,

Please confirm that system dashboards loading fine (without any errors) in the TSDB mode?

tetianakravchenko · 2023-05-31T11:20:08Z

@tetianakravchenko,
Can we make this beta and update the stack.version as in this PR: #6129

@lalit-satapathy before doing it - this PR also should be merged #6395, there are some missing metric_type metadata fields.

after merging this and #6395, tsdb can be enabled for all except: core, network, process datastream

tetianakravchenko · 2023-05-31T12:42:37Z

@tetianakravchenko,
Please confirm that system dashboards loading fine (without any errors) in the TSDB mode?

@lalit-satapathy yes, dashboards and inventory page works (note: tested on linux system only, so the windows dashboard have some errors regarding missing metrics)

packages/system/data_stream/diskio/fields/agent.yml

elasticmachine · 2023-05-31T23:45:59Z

Package system - 1.30.0 containing this change is available at https://epr.elastic.co/search?package=system

…except core data_streams (elastic#6118) * add dimensions to system package, metrics data_streams only Signed-off-by: Tetiana Kravchenko <[email protected]> * revert k8s package Signed-off-by: Tetiana Kravchenko <[email protected]> * revert k8s changes; add extra dimensions for diskio and network Signed-off-by: Tetiana Kravchenko <[email protected]> * filesystem: add mount_point and device_name Signed-off-by: Tetiana Kravchenko <[email protected]> * move core and process datastream to different PR Signed-off-by: Tetiana Kravchenko <[email protected]> * update PR number Signed-off-by: Tetiana Kravchenko <[email protected]> * move agent.id field to ecs.yaml Signed-off-by: Tetiana Kravchenko <[email protected]> * run elastic-package build Signed-off-by: Tetiana Kravchenko <[email protected]> * run elastic-package build Signed-off-by: Tetiana Kravchenko <[email protected]> * elastic-package check Signed-off-by: Tetiana Kravchenko <[email protected]> * add dimensions to process data_stream Signed-off-by: Tetiana Kravchenko <[email protected]> * clean up some duplicated fields Signed-off-by: Tetiana Kravchenko <[email protected]> * revert network data_stream changes Signed-off-by: Tetiana Kravchenko <[email protected]> * rever process data_stream changes Signed-off-by: Tetiana Kravchenko <[email protected]> * adjust the changelog description Signed-off-by: Tetiana Kravchenko <[email protected]> * revert process data_stream changes Signed-off-by: Tetiana Kravchenko <[email protected]> --------- Signed-off-by: Tetiana Kravchenko <[email protected]>

…except core data_streams (#6118) * add dimensions to system package, metrics data_streams only Signed-off-by: Tetiana Kravchenko <[email protected]> * revert k8s package Signed-off-by: Tetiana Kravchenko <[email protected]> * revert k8s changes; add extra dimensions for diskio and network Signed-off-by: Tetiana Kravchenko <[email protected]> * filesystem: add mount_point and device_name Signed-off-by: Tetiana Kravchenko <[email protected]> * move core and process datastream to different PR Signed-off-by: Tetiana Kravchenko <[email protected]> * update PR number Signed-off-by: Tetiana Kravchenko <[email protected]> * move agent.id field to ecs.yaml Signed-off-by: Tetiana Kravchenko <[email protected]> * run elastic-package build Signed-off-by: Tetiana Kravchenko <[email protected]> * run elastic-package build Signed-off-by: Tetiana Kravchenko <[email protected]> * elastic-package check Signed-off-by: Tetiana Kravchenko <[email protected]> * add dimensions to process data_stream Signed-off-by: Tetiana Kravchenko <[email protected]> * clean up some duplicated fields Signed-off-by: Tetiana Kravchenko <[email protected]> * revert network data_stream changes Signed-off-by: Tetiana Kravchenko <[email protected]> * rever process data_stream changes Signed-off-by: Tetiana Kravchenko <[email protected]> * adjust the changelog description Signed-off-by: Tetiana Kravchenko <[email protected]> * revert process data_stream changes Signed-off-by: Tetiana Kravchenko <[email protected]> --------- Signed-off-by: Tetiana Kravchenko <[email protected]>

tetianakravchenko added 5 commits May 8, 2023 11:00

add dimensions to system package, metrics data_streams only

e4bb378

Signed-off-by: Tetiana Kravchenko <[email protected]>

revert k8s package

db3cc62

Signed-off-by: Tetiana Kravchenko <[email protected]>

revert k8s changes; add extra dimensions for diskio and network

2f7b372

Signed-off-by: Tetiana Kravchenko <[email protected]>

filesystem: add mount_point and device_name

bc990cc

Signed-off-by: Tetiana Kravchenko <[email protected]>

move core and process datastream to different PR

9bc7b24

Signed-off-by: Tetiana Kravchenko <[email protected]>

tetianakravchenko marked this pull request as ready for review May 8, 2023 14:25

tetianakravchenko requested a review from a team as a code owner May 8, 2023 14:25

tetianakravchenko requested review from ycombinator, faec, agithomas, ritalwar and constanca-m May 8, 2023 14:25

update PR number

3b85782

Signed-off-by: Tetiana Kravchenko <[email protected]>

tetianakravchenko mentioned this pull request May 8, 2023

Enable TSDB for System integration #5848

Closed

11 tasks

tetianakravchenko added 4 commits May 8, 2023 16:41

move agent.id field to ecs.yaml

ef0e5be

Signed-off-by: Tetiana Kravchenko <[email protected]>

run elastic-package build

2a3ebfb

Signed-off-by: Tetiana Kravchenko <[email protected]>

run elastic-package build

952e665

Signed-off-by: Tetiana Kravchenko <[email protected]>

elastic-package check

f8cd9f1

Signed-off-by: Tetiana Kravchenko <[email protected]>

add dimensions to process data_stream

72dfc88

Signed-off-by: Tetiana Kravchenko <[email protected]>

tetianakravchenko changed the title ~~[System] Add dimensions to system package metrics data_streams only, except core and process data_streams~~ [System] Add dimensions to system package metrics data_streams only, except core data_streams May 9, 2023

agithomas reviewed May 9, 2023

View reviewed changes

tetianakravchenko commented May 10, 2023

View reviewed changes

lalit-satapathy reviewed May 10, 2023

View reviewed changes

ruflin reviewed May 11, 2023

View reviewed changes

clean up some duplicated fields

b762ecc

Signed-off-by: Tetiana Kravchenko <[email protected]>

Merge branch 'main' into system-tsdb

9979f69

tetianakravchenko added 4 commits May 31, 2023 11:07

revert network data_stream changes

24348cd

Signed-off-by: Tetiana Kravchenko <[email protected]>

rever process data_stream changes

ad39244

Signed-off-by: Tetiana Kravchenko <[email protected]>

adjust the changelog description

fbd3e49

Signed-off-by: Tetiana Kravchenko <[email protected]>

revert process data_stream changes

1314197

Signed-off-by: Tetiana Kravchenko <[email protected]>

ruflin approved these changes May 31, 2023

View reviewed changes

agithomas approved these changes May 31, 2023

View reviewed changes

cmacknz requested a review from leehinman May 31, 2023 12:42

leehinman requested changes May 31, 2023

View reviewed changes

packages/system/data_stream/diskio/fields/agent.yml Show resolved Hide resolved

This was referenced May 31, 2023

[System] [Network] Add dimensions and clean up duplicated fields definition #6405

Merged

Check fields duplication elastic/elastic-package#1287

Closed

[System][Process] Add dimensions metadata; remove duplicated fields #6407

Merged

leehinman approved these changes May 31, 2023

View reviewed changes

tetianakravchenko mentioned this pull request May 31, 2023

[System] remove duplicated fields #6412

Open

tetianakravchenko merged commit bbed053 into elastic:main May 31, 2023

andrewkroh added the Integration:system System label Jul 22, 2024

	- name: os.name
	level: extended
	type: keyword
	ignore_above: 1024
	multi_fields:
	- name: text
	type: text
	norms: false
	default_field: false
	description: Operating system name, without the version.
	example: Mac OS X

	- name: network.in.bytes
	type: scaled_float
	format: bytes
	unit: byte
	metric_type: counter
	description: \|
	The number of bytes received on all network interfaces by the host in a given period of time.

	- name: network.in.bytes
	type: long
	format: bytes
	description: >
	The number of bytes received on all network interfaces by the host in a given period of time.

	- name: mac
	level: core
	type: keyword
	ignore_above: 1024
	description: Host mac addresses.

[System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

[System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

Conversation

tetianakravchenko commented May 8, 2023 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented May 8, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented May 8, 2023 • edited Loading

🌐 Coverage report

agithomas left a comment

Choose a reason for hiding this comment

agithomas commented May 9, 2023

lalit-satapathy commented May 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented May 31, 2023

tetianakravchenko commented May 31, 2023

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

agithomas commented May 31, 2023

agithomas commented May 31, 2023

agithomas left a comment

Choose a reason for hiding this comment

tetianakravchenko commented May 31, 2023

lalit-satapathy commented May 31, 2023

lalit-satapathy commented May 31, 2023

tetianakravchenko commented May 31, 2023 • edited Loading

tetianakravchenko commented May 31, 2023

elasticmachine commented May 31, 2023

tetianakravchenko commented May 8, 2023 •

edited

Loading

elasticmachine commented May 8, 2023 •

edited

Loading

elasticmachine commented May 8, 2023 •

edited

Loading

tetianakravchenko commented May 31, 2023 •

edited

Loading