Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discover - Unmapped fields seems to be cached #156556

Closed
P1llus opened this issue May 3, 2023 · 2 comments · Fixed by #160195
Closed

Discover - Unmapped fields seems to be cached #156556

P1llus opened this issue May 3, 2023 · 2 comments · Fixed by #160195
Assignees
Labels
bug Fixes for quality problems that affect the customer experience :DataDiscovery/fix-it-week Feature:Data Views Data Views code and UI - index patterns before 8.0 impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. loe:needs-research This issue requires some research before it can be worked on or estimated Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Data, DataViews)

Comments

@P1llus
Copy link
Member

P1llus commented May 3, 2023

Stack Version
Stack Version: 8.78

Browser version:
Chrome: Version 112.0.5615.138

Browser OS version:
Windows 11

Original install method (e.g. download page, yum, from source, etc.):
elastic-package (integration development tooling)

Describe the bug:

In the left side of Discover, there is a list of fields, like "available fields", "unmapped fields" etc.

If a new field mapping is introduced in a index/datastream while your current Kibana session is live, and you do not do a hard refresh (CMD + Shift + R) then Kibana still thinks the newly introduced field is an unmapped field, not allowing you to use it for visualizations or in KQL queries, even though the field itself is corretly mapped on the index template.

Steps to reproduce:

  1. Spin up a fresh 8.7.1 version of the stack.
  2. Enable an integration + some datastream.
  3. Browse around in Discover.
  4. Enable another datastream (This updates the datastream field mapping and rolls it over).
  5. View discover again, new fields introduced is shown as unmapped until a hard refresh

Expected behavior:

The list of unmapped fields should not be cached, but rather retrived once the page is revisited later.

Screenshots (if relevant):
image

Provide logs and/or server output (if relevant):

Output from api/index_patterns/_fields_for_wildcard?

{"name":"rpc.cred.machinename","type":"string","esTypes":["keyword"],"searchable":true,"aggregatable":true,"readFromDocValues":true,"metadata_field":false}

Index mapping from GET /logs-network_traffic.nfs-default/_mapping

"rpc": {
          "properties": {
            "auth_flavor": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "cred": {
              "properties": {
                "gid": {
                  "type": "long"
                },
                "gids": {
                  "type": "long"
                },
                "machinename": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "stamp": {
                  "type": "long"
                },
                "uid": {
                  "type": "long"
                }
              }
            },
            "status": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "xid": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },

Any additional context:

@P1llus P1llus added the bug Fixes for quality problems that affect the customer experience label May 3, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label May 3, 2023
@drewdaemon drewdaemon added Feature:Data Views Data Views code and UI - index patterns before 8.0 Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Data, DataViews) labels May 3, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@botelastic botelastic bot removed the needs-team Issues missing a team label label May 3, 2023
@jughosta
Copy link
Contributor

jughosta commented May 3, 2023

Probably due to the fact that data view fields are not fetched on each search request.
Similar issue #149093

@jughosta jughosta self-assigned this Jun 21, 2023
@davismcphee davismcphee added loe:needs-research This issue requires some research before it can be worked on or estimated impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. labels Jun 22, 2023
jughosta added a commit that referenced this issue Jul 4, 2023
- Closes #156556
- Probably also addresses
#157239

## Summary

This PR allows to refresh fields list when revisiting Discover page.

## For testing

1. Open discover
2. In a separate tab open Dev Tools and create documents which include
new field names
3. Press "Refresh" in the unified search bar on Discover and notice that
new fields appear under Unmapped section
4. Navigate to another Kibana app (for example Home page)
5. Navigate back to Discover and notice that new fields are now under
Available section and have a proper icon.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Jul 4, 2023
- Closes elastic#156556
- Probably also addresses
elastic#157239

## Summary

This PR allows to refresh fields list when revisiting Discover page.

## For testing

1. Open discover
2. In a separate tab open Dev Tools and create documents which include
new field names
3. Press "Refresh" in the unified search bar on Discover and notice that
new fields appear under Unmapped section
4. Navigate to another Kibana app (for example Home page)
5. Navigate back to Discover and notice that new fields are now under
Available section and have a proper icon.

(cherry picked from commit 4819efa)
kibanamachine referenced this issue Jul 4, 2023
…1154)

# Backport

This will backport the following commits from `main` to `8.9`:
- [[Discover] Refresh fields when entering Discover
(#160195)](#160195)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Julia
Rechkunova","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-07-04T08:19:30Z","message":"[Discover]
Refresh fields when entering Discover (#160195)\n\n- Closes
https://github.com/elastic/kibana/issues/156556\r\n- Probably also
addresses\r\nhttps://github.com//issues/157239\r\n\r\n##
Summary\r\n\r\nThis PR allows to refresh fields list when revisiting
Discover page.\r\n\r\n## For testing\r\n\r\n1. Open discover\r\n2. In a
separate tab open Dev Tools and create documents which include\r\nnew
field names\r\n3. Press \"Refresh\" in the unified search bar on
Discover and notice that\r\nnew fields appear under Unmapped
section\r\n4. Navigate to another Kibana app (for example Home
page)\r\n5. Navigate back to Discover and notice that new fields are now
under\r\nAvailable section and have a proper
icon.","sha":"4819efa8eb16529db807043d36a1214070e10c79","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:DataDiscovery","backport:prev-minor","v8.10.0"],"number":160195,"url":"https://github.com/elastic/kibana/pull/160195","mergeCommit":{"message":"[Discover]
Refresh fields when entering Discover (#160195)\n\n- Closes
https://github.com/elastic/kibana/issues/156556\r\n- Probably also
addresses\r\nhttps://github.com//issues/157239\r\n\r\n##
Summary\r\n\r\nThis PR allows to refresh fields list when revisiting
Discover page.\r\n\r\n## For testing\r\n\r\n1. Open discover\r\n2. In a
separate tab open Dev Tools and create documents which include\r\nnew
field names\r\n3. Press \"Refresh\" in the unified search bar on
Discover and notice that\r\nnew fields appear under Unmapped
section\r\n4. Navigate to another Kibana app (for example Home
page)\r\n5. Navigate back to Discover and notice that new fields are now
under\r\nAvailable section and have a proper
icon.","sha":"4819efa8eb16529db807043d36a1214070e10c79"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/160195","number":160195,"mergeCommit":{"message":"[Discover]
Refresh fields when entering Discover (#160195)\n\n- Closes
https://github.com/elastic/kibana/issues/156556\r\n- Probably also
addresses\r\nhttps://github.com//issues/157239\r\n\r\n##
Summary\r\n\r\nThis PR allows to refresh fields list when revisiting
Discover page.\r\n\r\n## For testing\r\n\r\n1. Open discover\r\n2. In a
separate tab open Dev Tools and create documents which include\r\nnew
field names\r\n3. Press \"Refresh\" in the unified search bar on
Discover and notice that\r\nnew fields appear under Unmapped
section\r\n4. Navigate to another Kibana app (for example Home
page)\r\n5. Navigate back to Discover and notice that new fields are now
under\r\nAvailable section and have a proper
icon.","sha":"4819efa8eb16529db807043d36a1214070e10c79"}}]}] BACKPORT-->

Co-authored-by: Julia Rechkunova <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience :DataDiscovery/fix-it-week Feature:Data Views Data Views code and UI - index patterns before 8.0 impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. loe:needs-research This issue requires some research before it can be worked on or estimated Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Data, DataViews)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants