[Security solution] assistantKnowledgeBaseByDefault flag removed

[stephmilovic]

Summary

Still WIP

Removes the assistantKnowledgeBaseByDefault flag and all of its code paths.

This PR will be backported to8.x and main. In order to not risk many code changes in 8.16 and enable us to rollback, 8.16 will simply have a hardcoded assistantKnowledgeBaseByDefault value of true. That is implemented in #198178, which needs to merge before this PR.

Since #198178 will set assistantKnowledgeBaseByDefault to true by default, this PR is not the one that enables the flag by default. This simply cleans up all the code paths that are no longer used.

PR merge order

1. [Security solution] Knowledge base tour video update merged!
2. [Security Assistant] Knowledge base switch to use semantic_text
3. [Security AI Assistant] Fixed license issue for Knowledge Base resources initialization
4. [Security solution] assistantKnowledgeBaseByDefault flag defaults to true
5. [Security solution] assistantKnowledgeBaseByDefault flag removed (THIS PR)

[elasticmachine]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: af04dbc

Failed CI Steps

• FTR Configs #86
• FTR Configs #86
• Check Types
• Serverless AI Assistant - Security Solution Cypress Tests #1
• AI Assistant - Security Solution Cypress Tests #1

Test Failures

• [job] [logs] Serverless AI Assistant - Security Solution Cypress Tests / AI Assistant Conversations Changing conversations Correctly creates and titles new conversations, and allows title updates Correctly creates and titles new conversations, and allows title updates
• [job] [logs] AI Assistant - Security Solution Cypress Tests / AI Assistant Conversations Changing conversations Correctly creates and titles new conversations, and allows title updates Correctly creates and titles new conversations, and allows title updates
• [job] [logs] Serverless AI Assistant - Security Solution Cypress Tests / AI Assistant Conversations Changing conversations Properly switches back and forth between conversations Properly switches back and forth between conversations
• [job] [logs] AI Assistant - Security Solution Cypress Tests / AI Assistant Conversations Changing conversations Properly switches back and forth between conversations Properly switches back and forth between conversations
• [job] [logs] FTR Configs #86 / GenAI - Knowledge Base Entries APIs @ess Basic Security AI Assistant Knowledge Base Entries Create Entries Admin User should create a new document entry for the current user
• [job] [logs] FTR Configs #86 / GenAI - Knowledge Base Entries APIs @ess Basic Security AI Assistant Knowledge Base Entries Create Entries Admin User should create a new document entry for the current user

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	21.0MB	21.0MB	-3.0KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	88.4KB	88.3KB	-35.0B

History