Error handling and resolution for user/custom dashboards no longer working because of TSDB unsupported aggregations

[tetianakravchenko]

As a part of TSDB migration - elastic/integrations#5233, some metrics were set to type counter, this type does not support all available aggregations. OOTB dashboards were adjusted accordingly, but for custom dashboards should be provided more helpful error messages.

Example:
For fields with the type: counter, avg aggregation is not supported (as other aggregation, except max and last_value)

Error message is not really descriptive, would be helpful to add clarification, that supported aggregations for this type are max or last_value

[mlunadia]

There are two milestones we need to guide users to achieve:
1. Error awareness: Better error handling for users who have dashboards with TSDB unsupported aggs
2. Error resolution: Surfacing options to those aggregations (where available)
eg. users could be directed in the error message to a document explaining in detail and potential options they can use to replace unsupported aggregations.

[lalit-satapathy]

Earlier issue for this here and the scenario reproducing the same.

[elasticmachine]

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

[lalit-satapathy]

Couple of recent offline threads where custom dashboards got broken, because a field is now a TSDB counter. Need a compressive document to assist the user migration process for the custom dashboards.

1. From @lucabelluccini : k8 package

reported a customer complaining why they cannot perform an avg aggregation anymore on the field kubernetes.pod.network.rx.bytes. They get:

Field [kubernetes.pod.network.rx.bytes] of type [long] is not supported for aggregation [avg]

This is associated with the K8s switch to TSDB GA.

2. From @Danouchka : System package
   Hello, Since 8.8.1, I have the following error
   Field [XXX] of type [long] is not supported for aggregation [avg]. Is it a bug ?

CC: @mlunadia @giladgal

[timductive]

@martijnvg will work on adding documentation and work with docs team on where it lives.
@dej611 can update the error message and link to the doc when its ready.

[lalit-satapathy]

Link to ES issue: "Enable all aggregations on counter fields"- elastic/elasticsearch#97882

[lalit-satapathy]

Capturing below a brief summary of the behaviour of custom dashboards in Kibana while using un-supported counter aggregations. This is fixed as part of elastic/elasticsearch#97974

• Fixed 8.9.1 onwards
• Developing new dashboards with un-supported counter aggregations is still an error in Kibana.
• Upgrading stack, while using a custom dashboard with un-supported counter aggregations, is now supported.
  • Is there any warning in this case?

@dej611 @salvatore-campagna @martijnvg, Please update if this is accurate.

[dej611]

Developing new dashboards with un-supported counter aggregations is still an error in Kibana.

A user should not be able to create a visualization with un-supported counter aggregations within Lens, the editor should prevent to build and save such configuration. If some user manually builds the visualization via manually write the SO, then the other "relaxed" check will kick in.

Upgrading stack, while using a custom dashboard with un-supported counter aggregations, is now supported.
Is there any warning in this case?

Yes, there's a warning showing up in this case but the visualization will be rendered.

[lalit-satapathy]

Thanks @dej611 for the clarifications.

@harnish-elastic, Please check with one custom dashboard for a TSDB enabled package, to verify that both the behaviours captured above are working.

[harnish-elastic]

@lalit-satapathy and @dej611, I have performed the above-mentioned scenarios on the cloud elastic. However, I am not able to get the warning in 8.9.1 kibana! Please refer to the below steps that I have performed!

Summary:

• Create the deployment of the Kibana version 8.4.0 (< 8.7.0) from cloud.elastic.co
• Start the data collection for the integration packages(Apache and ActiveMQ) via Elastic-Agent
• Create the Custom dashboard including the counter metric_type fields using un-supported aggregation(Average/Sum)
• Go to the deployment and upgrade the current stack to 8.9.1
• Upgrade the integration policy and package version to the latest(including TSDB enablement) or delete the existing policy, add the new policy, and restart the data collection!
• Observe the behavior of the Custom dashboard

Output:

• Not able to see any warnings on the visualization
• I am able to create a fresh dashboard with the counter metric_type fields using un-supported aggregation(Average/Sum) in 8.9.1 for the same integration package in the stack without any warning/error.

Screenshots:

Let me know if there is anything else I can perform to re-produce the warning!

[dej611]

The backport had some bumps and it seems that it didn't make it to 8.9.1 but rather in 8.9.2: #163515

[lalit-satapathy]

@harnish-elastic, Please confirm that it is working 8.9.2

[harnish-elastic]

@harnish-elastic, Please confirm that it is working 8.9.2

@lalit-satapathy I have performed the same steps mentioned in this comment. However, I still cannot see the warning message in the visualizations. Let me know if there is anything else I can perform to re-produce the warning!

[lalit-satapathy]

What about 8.10?

[harnish-elastic]

What about 8.10?

The current latest deployment in the elastic cloud is 8.9.2! So I cannot perform this scenario. Let me know if there are other ways to test on 8.10.0

[stratoula]

Hey, can we close this issue?

[markov00]

I'm going to close this issue, please reopen it if the problem persist in 8.14 or higher