TSDB: Discover fields UI does not change icons counter/gauge field after package upgrade to TSDB.

[lalit-satapathy]

Kibana version: 8.8.0-snapshot

install method: via elastic-package

Description of the problem including expected versus actual behaviour:
This is a typical usage where a customer will upgrade a package from non-tsdb index mode to tsdb index mode. In the TSDB mode, Kiubana UI does not seem to respect new field mapping of counter/gauges.

Steps

• Ran nginx non-tsdb version 1.1.0 for 30 mins.

• Upgraded to TSDB beta version- 1.12.0-beta after that.

• New index got created.

• New index has all the time_series field mapping.

• Started getting data to new index.

• Verified that the TSDB field mappings are correct.

GET /.ds-metrics-nginx.stubstatus-default-2023.05.10-000002/_mapping/field/nginx.stubstatus.accepts
{
  ".ds-metrics-nginx.stubstatus-default-2023.05.10-000002": {
    "mappings": {
      "nginx.stubstatus.accepts": {
        "full_name": "nginx.stubstatus.accepts",
        "mapping": {
          "accepts": {
            "type": "long",
            "time_series_metric": "counter"
          }
        }
      }
    }
  }
}

• However, not getting the discover UI showing counters/gauges properly. This indicates, the new field mapping was not being used by Kibana (check the icons next to the fields). It also did not seem to give similar errors when wrong aggregation function was used on a counter.

Not sure, where exactly it got broken.

• When counter/gauze mappings are used by Kibana, it comes like this:

CC: @mlunadia @ruflin

[ruflin]

I'm wondering if the data views are not refreshed. Checking the UI, not sure if there is still a way to manually refresh a data view? @kpollich Is Fleet refreshing the data view in any way on upgrade?

[lalit-satapathy]

install method: via elastic-package

Same issue with elastic cloud 8.8.0-snapshot run. Summary: Field mapping not reflecting the behaviour in Kibana discover after package upgrade to TSDB.

[kpollich]

Is Fleet refreshing the data view in any way on upgrade?

As far as I can tell, no we don't update/refresh data views after a package upgrade. Maybe that's the root cause fix here? Should we just opt to always refresh all integration data views following an upgrade?

[drewdaemon]

@lalit-satapathy what happens if you perform a "hard" browser refresh to clear the cache? Wondering if this is a duplicate of the field caching problem seen in #156556 and #149093.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[lalit-satapathy]

@lalit-satapathy what happens if you perform a "hard" browser refresh to clear the cache? Wondering if this is a duplicate of the field caching problem seen in #156556 and #149093.

I didnot see much update after refreshing the cache. @ritalwar Please confirm your findings on this.

[jughosta]

If a page refresh does not make the icons appear correctly, then would be great to check a response from ES field caps API: for these fields it should start returning timeSeriesMetric value if this data stream is in time_series mode.

[ritalwar]

I observed similar behaviour in MSSQL.
After enabling TSDB:

But was expecting:

[ritalwar]

But opening Kibana in a new browser window seemed to fix the issue.

[agithomas]

Observed that, after enabling the TSDB enabled package (influxdb) and continuing on the same browser tab, opened the Influxdb dashboard and observed almost every dashboard having errors.

The same dashboard was opened in a different tab , no errors were seen. When I did the hard refresh on the initially opened tab, the error goes off. This issue appear to be similar to what is mentioned here #149093 .

[kertal]

Closing this the described issue is closed with #160195 , and the Dashboard related issue is tracked in #149093