Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Assistant] Knowledge base switch to use semantic_text #197007

Merged
merged 55 commits into from
Oct 30, 2024

Conversation

patrykkopycinski
Copy link
Contributor

@patrykkopycinski patrykkopycinski commented Oct 21, 2024

Summary

  • text_expansion is deprecated, use semantic_text instead
  • fix KB index entry form field options
  • explicitly create inference endpoint on KB setup if assistantKnowledgeBaseByDefault is true
  • when upgrade from v1 update KB ingest pipeline and remove unnecessary processor, but keep the pipeline for the backward compatibility
  • switch to use doc update for KB entries due to the limitations od semantic_text https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script
  • split loading Security labs content into smaller chunks

@patrykkopycinski patrykkopycinski added the ci:project-deploy-security Create a Security Serverless Project label Oct 23, 2024
@patrykkopycinski patrykkopycinski added ci:project-redeploy Always create a new Cloud project and removed ci:project-deploy-security Create a Security Serverless Project labels Oct 23, 2024
Copy link
Contributor

@kapral18 kapral18 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@patrykkopycinski patrykkopycinski added the ci:project-deploy-security Create a Security Serverless Project label Oct 29, 2024
@elasticmachine
Copy link
Contributor

elasticmachine commented Oct 30, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #47 / Actions and Triggers app Maintenance Windows Maintenance window update form should update a maintenance window

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 21.0MB 21.0MB +1.2KB

History

cc @patrykkopycinski

@@ -6,6 +6,8 @@
*/
import { FieldMap } from '@kbn/data-stream-adapter';

export const ASSISTANT_ELSER_INFERENCE_ID = 'elastic-security-ai-assistant-elser2';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the endpoint id is derived from this const

(node) => node.routing_state.routing_state === 'started'
);
const isReadyServerless = (stats: MlTrainedModelStats) =>
(stats.deployment_stats?.nodes as unknown as MlTrainedModelDeploymentNodesStats[])?.some(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No changes requested in this pr, however this check doesn't include the model id used by the inference endpoint, which means it's possible for this check to return true, even though there are no longer any nodes capable of performing inference for the assistant.

Copy link
Contributor

@andrew-goldstein andrew-goldstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @patrykkopycinski for this fix!
✅ Desk tested locally
LGTM 🚀

@patrykkopycinski patrykkopycinski merged commit a8c54f2 into elastic:main Oct 30, 2024
41 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.16, 8.x

https://github.com/elastic/kibana/actions/runs/11600189542

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Oct 30, 2024
…astic#197007)

## Summary

- `text_expansion` is deprecated, use `semantic_text` instead
- fix KB index entry form field options
- explicitly create inference endpoint on KB setup if
`assistantKnowledgeBaseByDefault` is true
- when upgrade from v1 update KB ingest pipeline and remove unnecessary
processor, but keep the pipeline for the backward compatibility
- switch to use `doc` update for KB entries due to the limitations od
`semantic_text`
https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script
- split loading Security labs content into smaller chunks

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Pedro Jaramillo <[email protected]>
(cherry picked from commit a8c54f2)
@kibanamachine
Copy link
Contributor

💔 Some backports could not be created

Status Branch Result
8.16 Backport failed because of merge conflicts
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 197007

Questions ?

Please refer to the Backport tool documentation

@patrykkopycinski
Copy link
Contributor Author

💚 All backports created successfully

Status Branch Result
8.16

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

patrykkopycinski added a commit to patrykkopycinski/kibana that referenced this pull request Oct 30, 2024
…astic#197007)

## Summary

- `text_expansion` is deprecated, use `semantic_text` instead
- fix KB index entry form field options
- explicitly create inference endpoint on KB setup if
`assistantKnowledgeBaseByDefault` is true
- when upgrade from v1 update KB ingest pipeline and remove unnecessary
processor, but keep the pipeline for the backward compatibility
- switch to use `doc` update for KB entries due to the limitations od
`semantic_text`
https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script
- split loading Security labs content into smaller chunks

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Pedro Jaramillo <[email protected]>
(cherry picked from commit a8c54f2)

# Conflicts:
#	packages/kbn-data-stream-adapter/src/field_maps/types.ts
kibanamachine added a commit that referenced this pull request Oct 30, 2024
…c_text&#x60; (#197007) (#198437)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Assistant] Knowledge base switch to use
&#x60;semantic_text&#x60;
(#197007)](#197007)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Patryk
Kopyciński","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-30T19:20:45Z","message":"[Security
Assistant] Knowledge base switch to use `semantic_text` (#197007)\n\n##
Summary\n\n- `text_expansion` is deprecated, use `semantic_text`
instead\n- fix KB index entry form field options \n- explicitly create
inference endpoint on KB setup if\n`assistantKnowledgeBaseByDefault` is
true\n- when upgrade from v1 update KB ingest pipeline and remove
unnecessary\nprocessor, but keep the pipeline for the backward
compatibility\n- switch to use `doc` update for KB entries due to the
limitations
od\n`semantic_text`\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script\n-
split loading Security labs content into smaller
chunks\n\n---------\n\nCo-authored-by: kibanamachine
<[email protected]>\nCo-authored-by: Pedro
Jaramillo
<[email protected]>","sha":"a8c54f2ea4f482285eda15c72c56da35d76a6719","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","v9.0.0","Feature:Security
Assistant","ci:project-deploy-security","Team:Security Generative
AI","v8.16.0","backport:version","v8.17.0"],"title":"[Security
Assistant] Knowledge base switch to use
`semantic_text`","number":197007,"url":"https://github.com/elastic/kibana/pull/197007","mergeCommit":{"message":"[Security
Assistant] Knowledge base switch to use `semantic_text` (#197007)\n\n##
Summary\n\n- `text_expansion` is deprecated, use `semantic_text`
instead\n- fix KB index entry form field options \n- explicitly create
inference endpoint on KB setup if\n`assistantKnowledgeBaseByDefault` is
true\n- when upgrade from v1 update KB ingest pipeline and remove
unnecessary\nprocessor, but keep the pipeline for the backward
compatibility\n- switch to use `doc` update for KB entries due to the
limitations
od\n`semantic_text`\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script\n-
split loading Security labs content into smaller
chunks\n\n---------\n\nCo-authored-by: kibanamachine
<[email protected]>\nCo-authored-by: Pedro
Jaramillo
<[email protected]>","sha":"a8c54f2ea4f482285eda15c72c56da35d76a6719"}},"sourceBranch":"main","suggestedTargetBranches":["8.16","8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197007","number":197007,"mergeCommit":{"message":"[Security
Assistant] Knowledge base switch to use `semantic_text` (#197007)\n\n##
Summary\n\n- `text_expansion` is deprecated, use `semantic_text`
instead\n- fix KB index entry form field options \n- explicitly create
inference endpoint on KB setup if\n`assistantKnowledgeBaseByDefault` is
true\n- when upgrade from v1 update KB ingest pipeline and remove
unnecessary\nprocessor, but keep the pipeline for the backward
compatibility\n- switch to use `doc` update for KB entries due to the
limitations
od\n`semantic_text`\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script\n-
split loading Security labs content into smaller
chunks\n\n---------\n\nCo-authored-by: kibanamachine
<[email protected]>\nCo-authored-by: Pedro
Jaramillo
<[email protected]>","sha":"a8c54f2ea4f482285eda15c72c56da35d76a6719"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Patryk Kopyciński <[email protected]>
stephmilovic pushed a commit to stephmilovic/kibana that referenced this pull request Oct 30, 2024
…astic#197007)

## Summary

- `text_expansion` is deprecated, use `semantic_text` instead
- fix KB index entry form field options
- explicitly create inference endpoint on KB setup if
`assistantKnowledgeBaseByDefault` is true
- when upgrade from v1 update KB ingest pipeline and remove unnecessary
processor, but keep the pipeline for the backward compatibility
- switch to use `doc` update for KB entries due to the limitations od
`semantic_text`
https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script
- split loading Security labs content into smaller chunks

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Pedro Jaramillo <[email protected]>
(cherry picked from commit a8c54f2)

# Conflicts:
#	packages/kbn-data-stream-adapter/src/field_maps/types.ts
patrykkopycinski added a commit that referenced this pull request Oct 30, 2024
…xt` (#197007) (#198439)

# Backport

This will backport the following commits from `main` to `8.16`:
- [[Security Assistant] Knowledge base switch to use `semantic_text`
(#197007)](#197007)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Patryk
Kopyciński","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-30T19:20:45Z","message":"[Security
Assistant] Knowledge base switch to use `semantic_text` (#197007)\n\n##
Summary\n\n- `text_expansion` is deprecated, use `semantic_text`
instead\n- fix KB index entry form field options \n- explicitly create
inference endpoint on KB setup if\n`assistantKnowledgeBaseByDefault` is
true\n- when upgrade from v1 update KB ingest pipeline and remove
unnecessary\nprocessor, but keep the pipeline for the backward
compatibility\n- switch to use `doc` update for KB entries due to the
limitations
od\n`semantic_text`\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script\n-
split loading Security labs content into smaller
chunks\n\n---------\n\nCo-authored-by: kibanamachine
<[email protected]>\nCo-authored-by: Pedro
Jaramillo
<[email protected]>","sha":"a8c54f2ea4f482285eda15c72c56da35d76a6719","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","v9.0.0","Feature:Security
Assistant","ci:project-deploy-security","Team:Security Generative
AI","v8.16.0","backport:version","v8.17.0"],"number":197007,"url":"https://github.com/elastic/kibana/pull/197007","mergeCommit":{"message":"[Security
Assistant] Knowledge base switch to use `semantic_text` (#197007)\n\n##
Summary\n\n- `text_expansion` is deprecated, use `semantic_text`
instead\n- fix KB index entry form field options \n- explicitly create
inference endpoint on KB setup if\n`assistantKnowledgeBaseByDefault` is
true\n- when upgrade from v1 update KB ingest pipeline and remove
unnecessary\nprocessor, but keep the pipeline for the backward
compatibility\n- switch to use `doc` update for KB entries due to the
limitations
od\n`semantic_text`\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script\n-
split loading Security labs content into smaller
chunks\n\n---------\n\nCo-authored-by: kibanamachine
<[email protected]>\nCo-authored-by: Pedro
Jaramillo
<[email protected]>","sha":"a8c54f2ea4f482285eda15c72c56da35d76a6719"}},"sourceBranch":"main","suggestedTargetBranches":["8.16"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197007","number":197007,"mergeCommit":{"message":"[Security
Assistant] Knowledge base switch to use `semantic_text` (#197007)\n\n##
Summary\n\n- `text_expansion` is deprecated, use `semantic_text`
instead\n- fix KB index entry form field options \n- explicitly create
inference endpoint on KB setup if\n`assistantKnowledgeBaseByDefault` is
true\n- when upgrade from v1 update KB ingest pipeline and remove
unnecessary\nprocessor, but keep the pipeline for the backward
compatibility\n- switch to use `doc` update for KB entries due to the
limitations
od\n`semantic_text`\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-text.html#update-script\n-
split loading Security labs content into smaller
chunks\n\n---------\n\nCo-authored-by: kibanamachine
<[email protected]>\nCo-authored-by: Pedro
Jaramillo
<[email protected]>","sha":"a8c54f2ea4f482285eda15c72c56da35d76a6719"}},{"branch":"8.16","label":"v8.16.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.x","label":"v8.17.0","labelRegex":"^v8.17.0$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/198437","number":198437,"state":"OPEN"}]}]
BACKPORT-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:version Backport to applied version labels bug Fixes for quality problems that affect the customer experience ci:project-deploy-security Create a Security Serverless Project Feature:Security Assistant Security Assistant release_note:skip Skip the PR/issue when compiling release notes Team:Security Generative AI Security Generative AI v8.16.0 v8.17.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants