feat!: implement improved fulltext search #671
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When MC failed, this would restart the container forever, even when the Alexandria dev env was stopped
winged
force-pushed
the
cleanup_search
branch
from
f6d10e6
to
c819a7d
Compare
winged
force-pushed
the
cleanup_search
branch
from
c819a7d
to
bdf7198
Compare
winged
commented
Oct 22, 2024
Yelinz
reviewed
Oct 22, 2024
winged
force-pushed
the
cleanup_search
branch
from
637db77
to
e36f534
Compare
|
Status update: I let some custom file/document factories run overnight, created over 400k files and then used the FTS in this branch to search. Using the new commandline utility, I was able to run some tests - general query duration is between 0.05 and 0.2 seconds. |
This makes the search endpoint resemble a different data type (not "files" anymore!) Also, the search behaves now rather differently: * Search results are ordered by search rank * Search results get the search context (a fragment of the text) as part of the response * File and document are referenced as related fields (and includes are available) This should provide a highly performant, useful search that does what it should[tm] BREAKING CHANGE: This changes the structure and type of the search endpoint's data.
This helps selecting files when we're searching (or otherwise looking for files) and only want the newest version
This runs the search as it were run through the search endpoint. Note this is mainly to be used for performance testing. No auth support and no visibility support exists currently. If you enable visibility / auth, it will likely break or not return anything.
winged
force-pushed
the
cleanup_search
branch
from
e36f534
to
e96aff8
Compare
Yelinz
reviewed
Oct 24, 2024
| ) | ||
| queryset = queryset.annotate( | ||
| search_rank=SearchRank(F("content_vector"), search_query), | ||
| search_context=SearchHeadline(F("content_text"), search_query), |
There was a problem hiding this comment.
If the search matches with the file name but not content, then there will be no context. But that should be ok.
Yelinz
approved these changes
Oct 24, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This makes the search endpoint resemble a different data type (not "files"
anymore!) Also, the search behaves now rather differently:
as part of the response
This should provide a highly performant, useful search that does what it should[tm]
BREAKING CHANGE: This changes the structure and type of the search
endpoint's data.
Drive-By: chore: do not restart minio config container
When MC failed, this would restart the container forever, even when the Alexandria dev env was stopped
Drive-By: feat(filters): add "only_newest" filter for files
This helps selecting files when we're searching (or otherwise looking
for files) and only want the newest version
Drive-By: feat(cmdline): search utility
This runs the search as it were run through the search endpoint.
Note this is mainly to be used for performance testing. No auth
support and no visibility support exists currently. If you enable
visibility / auth, it will likely break or not return anything.