Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RAG guide for docs #2525

Merged
merged 59 commits into from
Apr 8, 2024
Merged

RAG guide for docs #2525

merged 59 commits into from
Apr 8, 2024

Conversation

strickvl
Copy link
Contributor

Describe changes

I implemented/fixed _ to achieve _.

Pre-requisites

Please ensure you have done the following:

  • I have read the CONTRIBUTING.md document.
  • If my change requires a change to docs, I have updated the documentation accordingly.
  • I have added tests to cover my changes.
  • I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
  • If my changes require changes to the dashboard, these changes are communicated/requested.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Other (add details above)

@strickvl strickvl added the documentation Improvements or additions to documentation label Mar 13, 2024
@strickvl strickvl requested a review from htahir1 March 13, 2024 16:01
Copy link
Contributor

coderabbitai bot commented Mar 13, 2024

Important

Auto Review Skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@github-actions github-actions bot added the internal To filter out internal PRs and issues label Mar 13, 2024
@strickvl strickvl marked this pull request as draft March 18, 2024 12:48
@strickvl strickvl changed the title RAG guide for docs WIP: RAG guide for docs Mar 18, 2024
Copy link
Contributor

github-actions bot commented Apr 4, 2024

Images automagically compressed by Calibre's image-actions

Compression reduced images by 33.2%, saving 250.69 KB.

Filename Before After Improvement Visual comparison
docs/book/.gitbook/assets/rag-and-zenml.png 93.22 KB 58.99 KB -36.7% View diff
docs/book/.gitbook/assets/rag-overview.png 71.82 KB 48.42 KB -32.6% View diff
docs/book/.gitbook/assets/rag-process-whole.png 82.52 KB 54.37 KB -34.1% View diff
docs/book/.gitbook/assets/rag-stage-1.png 83.37 KB 57.23 KB -31.3% View diff
docs/book/.gitbook/assets/rag-stage-2.png 82.86 KB 55.11 KB -33.5% View diff
docs/book/.gitbook/assets/rag-stage-3.png 81.71 KB 54.94 KB -32.8% View diff
docs/book/.gitbook/assets/rag-stage-4.png 82.40 KB 54.55 KB -33.8% View diff
docs/book/.gitbook/assets/rag-when.png 177.20 KB 120.82 KB -31.8% View diff

241 images did not require optimisation.

Update required: Update image-actions configuration to the latest version before 1/1/21. See README for instructions.

@strickvl strickvl requested a review from htahir1 April 8, 2024 05:59
@strickvl strickvl changed the title WIP: RAG guide for docs RAG guide for docs Apr 8, 2024
Copy link
Contributor

github-actions bot commented Apr 8, 2024

Images automagically compressed by Calibre's image-actions

Compression reduced images by 5.1%, saving 59.46 KB.

Filename Before After Improvement Visual comparison
docs/book/.gitbook/assets/rag-overview.png 160.64 KB 158.83 KB -1.1% View diff
docs/book/.gitbook/assets/rag-process-whole.png 136.60 KB 133.47 KB -2.3% View diff
docs/book/.gitbook/assets/rag-stage-1.png 134.48 KB 130.10 KB -3.3% View diff
docs/book/.gitbook/assets/rag-stage-2.png 134.17 KB 127.08 KB -5.3% View diff
docs/book/.gitbook/assets/rag-stage-3.png 134.13 KB 127.29 KB -5.1% View diff
docs/book/.gitbook/assets/rag-stage-4.png 136.98 KB 129.05 KB -5.8% View diff
docs/book/.gitbook/assets/rag-when.png 322.52 KB 294.24 KB -8.8% View diff

242 images did not require optimisation.

Update required: Update image-actions configuration to the latest version before 1/1/21. See README for instructions.

Copy link
Contributor

@htahir1 htahir1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

This image is pixelated here: https://zenml-io.gitbook.io/alextestingground/Z1RNwxFe5UgX8V1GrZoR/user-guide/llmops-guide/rag-with-zenml/storing-embeddings-in-a-vector-database

apart from that I love it!

My only criticism is that the front is too 'text-heavy', a lot of words to read before the action shows up. Might make sense to have a small example to run at the very start to make developers excited?

Generally, I would merge and backport this first and then do the improvements

@strickvl
Copy link
Contributor Author

strickvl commented Apr 8, 2024

@htahir1 Pixellation is on purpose, to remove reference to my supabase org / db name etc. I will merge this in now, backport and then see what I can do about making it less text-heavy.

Note there's lots of reference to the project itself, which would probably want to be merged in before I do the full backport merge. There's an open PR here zenml-io/zenml-projects#97

@strickvl
Copy link
Contributor Author

strickvl commented Apr 8, 2024

Oh I guess you maybe mean that the image could be higher quality... I can bump that a bit for sure...

Copy link
Contributor

github-actions bot commented Apr 8, 2024

Images automagically compressed by Calibre's image-actions

Compression reduced images by 23.6%, saving 68.39 KB.

Filename Before After Improvement Visual comparison
docs/book/.gitbook/assets/supabase-editor-interface.png 290.26 KB 221.87 KB -23.6% View diff

248 images did not require optimisation.

Update required: Update image-actions configuration to the latest version before 1/1/21. See README for instructions.

@strickvl strickvl merged commit 39ef6dc into develop Apr 8, 2024
5 checks passed
@strickvl strickvl deleted the doc/llm-rag-guide branch April 8, 2024 12:00
strickvl added a commit that referenced this pull request Apr 9, 2024
* inital update for toc

* add intro / overview page

* update toc and add placeholder pages

* rag page draft

* undo changes

* more undo

* even more

* revert gitignore

* add scarf and fix typo

* update docs with nested llmops guide sections

* split RAG guide into sections

* add titles

* update docs further

* write first few sections

* data ingestion docs

* add data ingestion section

* remove extra requirements file

* add embeddings generation docs

* fix image URL

* fix image

* tweak for embeddings generation docs

* update toc

* no need for double header

* fix image url

* actually fix image url

* add zenml artifact load to the doc

* update embeddings section

* update embeddings storage section

* complete embeddings storage section

* finish RAG pipeline walkthrough draft

* Optimised images with calibre/image-actions

* make link html

* add link and fix missing text

* add initial set of draft images

* Optimised images with calibre/image-actions

* fix link

* make it less wordy

* remove unused parts of the guide

* update TOC

* add step operator docs link

* fix inference image link

* add specific code references

* Update code examples with links to specific files

* remove infelicity

* update images

* remove one illustration

* Optimised images with calibre/image-actions

* use higher quality image

* Optimised images with calibre/image-actions

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
(cherry picked from commit 39ef6dc)
strickvl added a commit that referenced this pull request Apr 9, 2024
* inital update for toc

* add intro / overview page

* update toc and add placeholder pages

* rag page draft

* undo changes

* more undo

* even more

* revert gitignore

* add scarf and fix typo

* update docs with nested llmops guide sections

* split RAG guide into sections

* add titles

* update docs further

* write first few sections

* data ingestion docs

* add data ingestion section

* remove extra requirements file

* add embeddings generation docs

* fix image URL

* fix image

* tweak for embeddings generation docs

* update toc

* no need for double header

* fix image url

* actually fix image url

* add zenml artifact load to the doc

* update embeddings section

* update embeddings storage section

* complete embeddings storage section

* finish RAG pipeline walkthrough draft

* Optimised images with calibre/image-actions

* make link html

* add link and fix missing text

* add initial set of draft images

* Optimised images with calibre/image-actions

* fix link

* make it less wordy

* remove unused parts of the guide

* update TOC

* add step operator docs link

* fix inference image link

* add specific code references

* Update code examples with links to specific files

* remove infelicity

* update images

* remove one illustration

* Optimised images with calibre/image-actions

* use higher quality image

* Optimised images with calibre/image-actions

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
(cherry picked from commit 39ef6dc)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation internal To filter out internal PRs and issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants