[Security Assistant] Knowledge base switch to use semantic_text
#197007
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…bana into fix/kb-semantic-text
patrykkopycinski
added
the
ci:project-deploy-security
patrykkopycinski
added
ci:project-redeploy
This was referenced Oct 29, 2024
...ssistant/impl/knowledge_base/knowledge_base_settings_management/use_knowledge_base_table.tsx
Show resolved
Hide resolved
x-pack/packages/kbn-elastic-assistant/impl/assistant/index.test.tsx
Outdated
Show resolved
Hide resolved
patrykkopycinski
added
the
ci:project-deploy-security
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]Async chunks
History
|
| @@ -6,6 +6,8 @@ | |||
| */ | |||
| import { FieldMap } from '@kbn/data-stream-adapter'; | |||
|
|
|||
| export const ASSISTANT_ELSER_INFERENCE_ID = 'elastic-security-ai-assistant-elser2'; | |||
There was a problem hiding this comment.
the endpoint id is derived from this const
| (node) => node.routing_state.routing_state === 'started' | ||
| ); | ||
| const isReadyServerless = (stats: MlTrainedModelStats) => | ||
| (stats.deployment_stats?.nodes as unknown as MlTrainedModelDeploymentNodesStats[])?.some( |
There was a problem hiding this comment.
No changes requested in this pr, however this check doesn't include the model id used by the inference endpoint, which means it's possible for this check to return true, even though there are no longer any nodes capable of performing inference for the assistant.
andrew-goldstein
approved these changes
Oct 30, 2024
There was a problem hiding this comment.
Thank you @patrykkopycinski for this fix!
✅ Desk tested locally
LGTM 🚀
|
Starting backport for target branches: 8.16, 8.x |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 30, 2024
…astic#197007) ## Summary - `text_expansion` is deprecated, use `semantic_text` instead - fix KB index entry form field options - explicitly create inference endpoint on KB setup if `assistantKnowledgeBaseByDefault` is true - when upgrade from v1 update KB ingest pipeline and remove unnecessary processor, but keep the pipeline for the backward compatibility - switch to use `doc` update for KB entries due to the limitations od `semantic_text` https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script - split loading Security labs content into smaller chunks --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Pedro Jaramillo <[email protected]> (cherry picked from commit a8c54f2)
💔 Some backports could not be created
Note: Successful backport PRs will be merged automatically after passing CI. Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
patrykkopycinski
added a commit
to patrykkopycinski/kibana
that referenced
this pull request
Oct 30, 2024
…astic#197007) ## Summary - `text_expansion` is deprecated, use `semantic_text` instead - fix KB index entry form field options - explicitly create inference endpoint on KB setup if `assistantKnowledgeBaseByDefault` is true - when upgrade from v1 update KB ingest pipeline and remove unnecessary processor, but keep the pipeline for the backward compatibility - switch to use `doc` update for KB entries due to the limitations od `semantic_text` https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script - split loading Security labs content into smaller chunks --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Pedro Jaramillo <[email protected]> (cherry picked from commit a8c54f2) # Conflicts: # packages/kbn-data-stream-adapter/src/field_maps/types.ts
kibanamachine
added a commit
that referenced
this pull request
Oct 30, 2024
…c_text` (#197007) (#198437) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Assistant] Knowledge base switch to use `semantic_text` (#197007)](#197007) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Patryk Kopyciński","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-30T19:20:45Z","message":"[Security Assistant] Knowledge base switch to use `semantic_text` (#197007)\n\n## Summary\n\n- `text_expansion` is deprecated, use `semantic_text` instead\n- fix KB index entry form field options \n- explicitly create inference endpoint on KB setup if\n`assistantKnowledgeBaseByDefault` is true\n- when upgrade from v1 update KB ingest pipeline and remove unnecessary\nprocessor, but keep the pipeline for the backward compatibility\n- switch to use `doc` update for KB entries due to the limitations od\n`semantic_text`\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script\n- split loading Security labs content into smaller chunks\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>\nCo-authored-by: Pedro Jaramillo <[email protected]>","sha":"a8c54f2ea4f482285eda15c72c56da35d76a6719","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","v9.0.0","Feature:Security Assistant","ci:project-deploy-security","Team:Security Generative AI","v8.16.0","backport:version","v8.17.0"],"title":"[Security Assistant] Knowledge base switch to use `semantic_text`","number":197007,"url":"https://github.com/elastic/kibana/pull/197007","mergeCommit":{"message":"[Security Assistant] Knowledge base switch to use `semantic_text` (#197007)\n\n## Summary\n\n- `text_expansion` is deprecated, use `semantic_text` instead\n- fix KB index entry form field options \n- explicitly create inference endpoint on KB setup if\n`assistantKnowledgeBaseByDefault` is true\n- when upgrade from v1 update KB ingest pipeline and remove unnecessary\nprocessor, but keep the pipeline for the backward compatibility\n- switch to use `doc` update for KB entries due to the limitations od\n`semantic_text`\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script\n- split loading Security labs content into smaller chunks\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>\nCo-authored-by: Pedro Jaramillo <[email protected]>","sha":"a8c54f2ea4f482285eda15c72c56da35d76a6719"}},"sourceBranch":"main","suggestedTargetBranches":["8.16","8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197007","number":197007,"mergeCommit":{"message":"[Security Assistant] Knowledge base switch to use `semantic_text` (#197007)\n\n## Summary\n\n- `text_expansion` is deprecated, use `semantic_text` instead\n- fix KB index entry form field options \n- explicitly create inference endpoint on KB setup if\n`assistantKnowledgeBaseByDefault` is true\n- when upgrade from v1 update KB ingest pipeline and remove unnecessary\nprocessor, but keep the pipeline for the backward compatibility\n- switch to use `doc` update for KB entries due to the limitations od\n`semantic_text`\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script\n- split loading Security labs content into smaller chunks\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>\nCo-authored-by: Pedro Jaramillo <[email protected]>","sha":"a8c54f2ea4f482285eda15c72c56da35d76a6719"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Patryk Kopyciński <[email protected]>
stephmilovic
pushed a commit
to stephmilovic/kibana
that referenced
this pull request
Oct 30, 2024
…astic#197007) ## Summary - `text_expansion` is deprecated, use `semantic_text` instead - fix KB index entry form field options - explicitly create inference endpoint on KB setup if `assistantKnowledgeBaseByDefault` is true - when upgrade from v1 update KB ingest pipeline and remove unnecessary processor, but keep the pipeline for the backward compatibility - switch to use `doc` update for KB entries due to the limitations od `semantic_text` https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script - split loading Security labs content into smaller chunks --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Pedro Jaramillo <[email protected]> (cherry picked from commit a8c54f2) # Conflicts: # packages/kbn-data-stream-adapter/src/field_maps/types.ts
patrykkopycinski
added a commit
that referenced
this pull request
Oct 30, 2024
…xt` (#197007) (#198439) # Backport This will backport the following commits from `main` to `8.16`: - [[Security Assistant] Knowledge base switch to use `semantic_text` (#197007)](#197007) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Patryk Kopyciński","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-30T19:20:45Z","message":"[Security Assistant] Knowledge base switch to use `semantic_text` (#197007)\n\n## Summary\n\n- `text_expansion` is deprecated, use `semantic_text` instead\n- fix KB index entry form field options \n- explicitly create inference endpoint on KB setup if\n`assistantKnowledgeBaseByDefault` is true\n- when upgrade from v1 update KB ingest pipeline and remove unnecessary\nprocessor, but keep the pipeline for the backward compatibility\n- switch to use `doc` update for KB entries due to the limitations od\n`semantic_text`\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script\n- split loading Security labs content into smaller chunks\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>\nCo-authored-by: Pedro Jaramillo <[email protected]>","sha":"a8c54f2ea4f482285eda15c72c56da35d76a6719","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","v9.0.0","Feature:Security Assistant","ci:project-deploy-security","Team:Security Generative AI","v8.16.0","backport:version","v8.17.0"],"number":197007,"url":"https://github.com/elastic/kibana/pull/197007","mergeCommit":{"message":"[Security Assistant] Knowledge base switch to use `semantic_text` (#197007)\n\n## Summary\n\n- `text_expansion` is deprecated, use `semantic_text` instead\n- fix KB index entry form field options \n- explicitly create inference endpoint on KB setup if\n`assistantKnowledgeBaseByDefault` is true\n- when upgrade from v1 update KB ingest pipeline and remove unnecessary\nprocessor, but keep the pipeline for the backward compatibility\n- switch to use `doc` update for KB entries due to the limitations od\n`semantic_text`\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script\n- split loading Security labs content into smaller chunks\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>\nCo-authored-by: Pedro Jaramillo <[email protected]>","sha":"a8c54f2ea4f482285eda15c72c56da35d76a6719"}},"sourceBranch":"main","suggestedTargetBranches":["8.16"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197007","number":197007,"mergeCommit":{"message":"[Security Assistant] Knowledge base switch to use `semantic_text` (#197007)\n\n## Summary\n\n- `text_expansion` is deprecated, use `semantic_text` instead\n- fix KB index entry form field options \n- explicitly create inference endpoint on KB setup if\n`assistantKnowledgeBaseByDefault` is true\n- when upgrade from v1 update KB ingest pipeline and remove unnecessary\nprocessor, but keep the pipeline for the backward compatibility\n- switch to use `doc` update for KB entries due to the limitations od\n`semantic_text`\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script\n- split loading Security labs content into smaller chunks\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>\nCo-authored-by: Pedro Jaramillo <[email protected]>","sha":"a8c54f2ea4f482285eda15c72c56da35d76a6719"}},{"branch":"8.16","label":"v8.16.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.x","label":"v8.17.0","labelRegex":"^v8.17.0$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/198437","number":198437,"state":"OPEN"}]}] BACKPORT-->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
text_expansionis deprecated, usesemantic_textinsteadassistantKnowledgeBaseByDefaultis truedocupdate for KB entries due to the limitations odsemantic_texthttps://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script