Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make generated assets a non-terminal state #992

Merged
merged 28 commits into from
Dec 8, 2024

Conversation

sabaimran
Copy link
Member

Currently, Khoj has terminal states with respect to what assets it outputs. We limit it to image, text, and excalidraw diagram. This limitation is unnecessary and provides undue constraints for creating more dynamic, diverse experiences. For instance, we may want the chat view to morph for document editing or generation, in which case having limited output modes would be a detriment.

This change allows us to emit generated assets and then continue on to more text generation in final response. It forces text response for all messages. It adds a new stream event, GENERATED_ASSETS, which holds content that the AI is emitting in response to the query.

@sabaimran sabaimran added the maintain Maintain code, documentation or project label Nov 30, 2024
Copy link

gitguardian bot commented Nov 30, 2024

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
14541143 Triggered Generic Password c5329d7 .github/workflows/run_evals.yml View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@sabaimran sabaimran requested a review from debanjum November 30, 2024 21:22
src/interface/obsidian/src/chat_view.ts Show resolved Hide resolved
src/interface/obsidian/src/chat_view.ts Show resolved Hide resolved
src/interface/obsidian/src/chat_view.ts Show resolved Hide resolved
src/interface/obsidian/src/chat_view.ts Show resolved Hide resolved
src/interface/obsidian/src/chat_view.ts Show resolved Hide resolved
src/interface/obsidian/styles.css Show resolved Hide resolved
…tions

- Added it to the Django migrations so that it auto-triggers when someone updates their server and starts it up again for the first time. This will require that they update their clients as well in order to view/consume image content.
- Remove server-side references in the code that allow to parse the text-to-image intent as it will no longer be necessary, given the chat logs will be migrated
Copy link
Member

@debanjum debanjum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments. Looking forward to give Khoj more flexibility in its responses!

src/interface/obsidian/src/chat_view.ts Outdated Show resolved Hide resolved
src/khoj/processor/conversation/prompts.py Show resolved Hide resolved
src/khoj/processor/conversation/prompts.py Outdated Show resolved Hide resolved
src/khoj/processor/conversation/prompts.py Show resolved Hide resolved
src/khoj/processor/conversation/utils.py Outdated Show resolved Hide resolved
src/khoj/processor/conversation/utils.py Show resolved Hide resolved
src/khoj/processor/conversation/utils.py Outdated Show resolved Hide resolved
src/khoj/processor/conversation/prompts.py Outdated Show resolved Hide resolved
@sabaimran sabaimran requested a review from debanjum December 8, 2024 22:18
@sabaimran sabaimran merged commit e3789ae into master Dec 8, 2024
11 checks passed
@sabaimran sabaimran deleted the features/allow-multi-outputs-in-chat branch December 8, 2024 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintain Maintain code, documentation or project
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants