Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TSDB: Discover fields UI does not change icons counter/gauge field after package upgrade to TSDB. #157239

Closed
lalit-satapathy opened this issue May 10, 2023 · 11 comments
Assignees
Labels
bug Fixes for quality problems that affect the customer experience Feature:Data Views Data Views code and UI - index patterns before 8.0 impact:needs-assessment Product and/or Engineering needs to evaluate the impact of the change. loe:needs-research This issue requires some research before it can be worked on or estimated Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Data, DataViews)

Comments

@lalit-satapathy
Copy link

Kibana version: 8.8.0-snapshot

install method: via elastic-package

Description of the problem including expected versus actual behaviour:
This is a typical usage where a customer will upgrade a package from non-tsdb index mode to tsdb index mode. In the TSDB mode, Kiubana UI does not seem to respect new field mapping of counter/gauges.

Steps

  • Ran nginx non-tsdb version 1.1.0 for 30 mins.

  • Upgraded to TSDB beta version- 1.12.0-beta after that.

  • New index got created.

Screenshot 2023-05-10 at 11 52 35 AM
  • New index has all the time_series field mapping.
Screenshot 2023-05-10 at 11 53 13 AM
  • Started getting data to new index.

  • Verified that the TSDB field mappings are correct.

GET /.ds-metrics-nginx.stubstatus-default-2023.05.10-000002/_mapping/field/nginx.stubstatus.accepts
{
  ".ds-metrics-nginx.stubstatus-default-2023.05.10-000002": {
    "mappings": {
      "nginx.stubstatus.accepts": {
        "full_name": "nginx.stubstatus.accepts",
        "mapping": {
          "accepts": {
            "type": "long",
            "time_series_metric": "counter"
          }
        }
      }
    }
  }
}
  • However, not getting the discover UI showing counters/gauges properly. This indicates, the new field mapping was not being used by Kibana (check the icons next to the fields). It also did not seem to give similar errors when wrong aggregation function was used on a counter.
Screenshot 2023-05-10 at 11 51 58 AM

Not sure, where exactly it got broken.

  • When counter/gauze mappings are used by Kibana, it comes like this:
Screenshot 2023-05-10 at 12 51 46 PM

CC: @mlunadia @ruflin

@lalit-satapathy lalit-satapathy added the bug Fixes for quality problems that affect the customer experience label May 10, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label May 10, 2023
@ruflin
Copy link
Member

ruflin commented May 10, 2023

I'm wondering if the data views are not refreshed. Checking the UI, not sure if there is still a way to manually refresh a data view? @kpollich Is Fleet refreshing the data view in any way on upgrade?

@lalit-satapathy
Copy link
Author

install method: via elastic-package

Same issue with elastic cloud 8.8.0-snapshot run. Summary: Field mapping not reflecting the behaviour in Kibana discover after package upgrade to TSDB.

@kpollich
Copy link
Member

Is Fleet refreshing the data view in any way on upgrade?

As far as I can tell, no we don't update/refresh data views after a package upgrade. Maybe that's the root cause fix here? Should we just opt to always refresh all integration data views following an upgrade?

@drewdaemon
Copy link
Contributor

drewdaemon commented May 10, 2023

@lalit-satapathy what happens if you perform a "hard" browser refresh to clear the cache? Wondering if this is a duplicate of the field caching problem seen in #156556 and #149093.

@drewdaemon drewdaemon added Feature:Data Views Data Views code and UI - index patterns before 8.0 Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Data, DataViews) labels May 10, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@lalit-satapathy
Copy link
Author

@lalit-satapathy what happens if you perform a "hard" browser refresh to clear the cache? Wondering if this is a duplicate of the field caching problem seen in #156556 and #149093.

I didnot see much update after refreshing the cache. @ritalwar Please confirm your findings on this.

@jughosta
Copy link
Contributor

If a page refresh does not make the icons appear correctly, then would be great to check a response from ES field caps API: for these fields it should start returning timeSeriesMetric value if this data stream is in time_series mode.

@ritalwar
Copy link
Contributor

I observed similar behaviour in MSSQL.
After enabling TSDB:
image

But was expecting:

image

@ritalwar
Copy link
Contributor

ritalwar commented May 11, 2023

But opening Kibana in a new browser window seemed to fix the issue.

image

@agithomas
Copy link

Observed that, after enabling the TSDB enabled package (influxdb) and continuing on the same browser tab, opened the Influxdb dashboard and observed almost every dashboard having errors.

Screenshot 2023-05-18 at 10 31 17 AM

The same dashboard was opened in a different tab , no errors were seen. When I did the hard refresh on the initially opened tab, the error goes off. This issue appear to be similar to what is mentioned here #149093 .

Screenshot 2023-05-18 at 10 34 11 AM

@lukasolson lukasolson added loe:needs-research This issue requires some research before it can be worked on or estimated impact:needs-assessment Product and/or Engineering needs to evaluate the impact of the change. labels May 25, 2023
jughosta added a commit that referenced this issue Jul 4, 2023
- Closes #156556
- Probably also addresses
#157239

## Summary

This PR allows to refresh fields list when revisiting Discover page.

## For testing

1. Open discover
2. In a separate tab open Dev Tools and create documents which include
new field names
3. Press "Refresh" in the unified search bar on Discover and notice that
new fields appear under Unmapped section
4. Navigate to another Kibana app (for example Home page)
5. Navigate back to Discover and notice that new fields are now under
Available section and have a proper icon.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Jul 4, 2023
- Closes elastic#156556
- Probably also addresses
elastic#157239

## Summary

This PR allows to refresh fields list when revisiting Discover page.

## For testing

1. Open discover
2. In a separate tab open Dev Tools and create documents which include
new field names
3. Press "Refresh" in the unified search bar on Discover and notice that
new fields appear under Unmapped section
4. Navigate to another Kibana app (for example Home page)
5. Navigate back to Discover and notice that new fields are now under
Available section and have a proper icon.

(cherry picked from commit 4819efa)
@kertal
Copy link
Member

kertal commented Jul 4, 2023

Closing this the described issue is closed with #160195 , and the Dashboard related issue is tracked in #149093

@kertal kertal closed this as completed Jul 4, 2023
kibanamachine added a commit that referenced this issue Jul 4, 2023
…1154)

# Backport

This will backport the following commits from `main` to `8.9`:
- [[Discover] Refresh fields when entering Discover
(#160195)](#160195)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Julia
Rechkunova","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-07-04T08:19:30Z","message":"[Discover]
Refresh fields when entering Discover (#160195)\n\n- Closes
https://github.com/elastic/kibana/issues/156556\r\n- Probably also
addresses\r\nhttps://github.com//issues/157239\r\n\r\n##
Summary\r\n\r\nThis PR allows to refresh fields list when revisiting
Discover page.\r\n\r\n## For testing\r\n\r\n1. Open discover\r\n2. In a
separate tab open Dev Tools and create documents which include\r\nnew
field names\r\n3. Press \"Refresh\" in the unified search bar on
Discover and notice that\r\nnew fields appear under Unmapped
section\r\n4. Navigate to another Kibana app (for example Home
page)\r\n5. Navigate back to Discover and notice that new fields are now
under\r\nAvailable section and have a proper
icon.","sha":"4819efa8eb16529db807043d36a1214070e10c79","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:DataDiscovery","backport:prev-minor","v8.10.0"],"number":160195,"url":"https://github.com/elastic/kibana/pull/160195","mergeCommit":{"message":"[Discover]
Refresh fields when entering Discover (#160195)\n\n- Closes
https://github.com/elastic/kibana/issues/156556\r\n- Probably also
addresses\r\nhttps://github.com//issues/157239\r\n\r\n##
Summary\r\n\r\nThis PR allows to refresh fields list when revisiting
Discover page.\r\n\r\n## For testing\r\n\r\n1. Open discover\r\n2. In a
separate tab open Dev Tools and create documents which include\r\nnew
field names\r\n3. Press \"Refresh\" in the unified search bar on
Discover and notice that\r\nnew fields appear under Unmapped
section\r\n4. Navigate to another Kibana app (for example Home
page)\r\n5. Navigate back to Discover and notice that new fields are now
under\r\nAvailable section and have a proper
icon.","sha":"4819efa8eb16529db807043d36a1214070e10c79"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/160195","number":160195,"mergeCommit":{"message":"[Discover]
Refresh fields when entering Discover (#160195)\n\n- Closes
https://github.com/elastic/kibana/issues/156556\r\n- Probably also
addresses\r\nhttps://github.com//issues/157239\r\n\r\n##
Summary\r\n\r\nThis PR allows to refresh fields list when revisiting
Discover page.\r\n\r\n## For testing\r\n\r\n1. Open discover\r\n2. In a
separate tab open Dev Tools and create documents which include\r\nnew
field names\r\n3. Press \"Refresh\" in the unified search bar on
Discover and notice that\r\nnew fields appear under Unmapped
section\r\n4. Navigate to another Kibana app (for example Home
page)\r\n5. Navigate back to Discover and notice that new fields are now
under\r\nAvailable section and have a proper
icon.","sha":"4819efa8eb16529db807043d36a1214070e10c79"}}]}] BACKPORT-->

Co-authored-by: Julia Rechkunova <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience Feature:Data Views Data Views code and UI - index patterns before 8.0 impact:needs-assessment Product and/or Engineering needs to evaluate the impact of the change. loe:needs-research This issue requires some research before it can be worked on or estimated Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Data, DataViews)
Projects
None yet
Development

No branches or pull requests