feat: add es8 indexer&retriever #41
Open
+2,186
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
license-eye has totally checked 179 files.
| Valid | Invalid | Ignored | Fixed |
|---|---|---|---|
| 157 | 22 | 0 | 0 |
Click to see the invalid file list
- components/indexer/es8/field_mapping/consts.go
- components/indexer/es8/field_mapping/field_mapping.go
- components/indexer/es8/internal/consts.go
- components/indexer/es8/consts.go
- components/indexer/es8/indexer.go
- components/indexer/es8/indexer_test.go
- components/indexer/es8/utils.go
- components/retriever/es8/field_mapping/consts.go
- components/retriever/es8/field_mapping/mapping.go
- components/retriever/es8/internal/consts.go
- components/retriever/es8/search_mode/approximate.go
- components/retriever/es8/search_mode/approximate_test.go
- components/retriever/es8/search_mode/dense_vector_similarity.go
- components/retriever/es8/search_mode/dense_vector_similarity_test.go
- components/retriever/es8/search_mode/exact_match.go
- components/retriever/es8/search_mode/interface.go
- components/retriever/es8/search_mode/raw_string.go
- components/retriever/es8/search_mode/sparse_vector_text_expansion.go
- components/retriever/es8/search_mode/sparse_vector_text_expansion_test.go
- components/retriever/es8/search_mode/utils.go
- components/retriever/es8/consts.go
- components/retriever/es8/retriever.go
components/retriever/es8/search_mode/sparse_vector_text_expansion_test.go
Show resolved
Hide resolved
|
go-test-coverage report: |
There was a problem hiding this comment.
license-eye has totally checked 174 files.
| Valid | Invalid | Ignored | Fixed |
|---|---|---|---|
| 164 | 10 | 0 | 0 |
Click to see the invalid file list
- components/retriever/es8/search_mode/approximate.go
- components/retriever/es8/search_mode/approximate_test.go
- components/retriever/es8/search_mode/dense_vector_similarity.go
- components/retriever/es8/search_mode/dense_vector_similarity_test.go
- components/retriever/es8/search_mode/exact_match.go
- components/retriever/es8/search_mode/raw_string.go
- components/retriever/es8/search_mode/sparse_vector_text_expansion.go
- components/retriever/es8/search_mode/sparse_vector_text_expansion_test.go
- components/retriever/es8/search_mode/utils.go
- components/retriever/es8/option.go
BytePender
commented
Jan 13, 2025
BytePender
commented
Jan 13, 2025
| // QueryFieldName the name of query field, required when using Hybrid | ||
| QueryFieldName string | ||
| // VectorFieldName the name of the vector field to search against, required | ||
| VectorFieldName string |
There was a problem hiding this comment.
QueryFieldName 和 VectorFieldName 分别怎么理解？
There was a problem hiding this comment.
QueryFieldName 代表 query 对应的 field 名称
VectorFieldName 代表 query 通过 embedding 生成的 vector 对应的 field 名称
N3kox
force-pushed
the
feat/es8_retriever
branch
4 times, most recently
from
ad8060a to
1b7e6dd
Compare
There was a problem hiding this comment.
license-eye has totally checked 180 files.
| Valid | Invalid | Ignored | Fixed |
|---|---|---|---|
| 179 | 1 | 0 | 0 |
Click to see the invalid file list
- components/retriever/es8/search_mode/raw_string_test.go
N3kox
force-pushed
the
feat/es8_retriever
branch
from
1b7e6dd to
da0e449
Compare
BytePender
commented
Jan 13, 2025
components/indexer/es8/utils.go
Outdated
|
|
||
| var chunks [][]T | ||
| for size < len(slice) { | ||
| slice, chunks = slice[size:], append(chunks, slice[0:size:size]) |
There was a problem hiding this comment.
slice[0:size:size] 这里 slice 切片时，指定 cap 的用意是啥
BytePender
commented
Jan 13, 2025
components/indexer/es8/indexer.go
Outdated
| if err = bi.Add(ctx, item); err != nil { | ||
| return nil, err | ||
| } | ||
| } |
There was a problem hiding this comment.
这个地方有 bi.Add 进行批量添加， 是不是少一个 执行的步骤？
There was a problem hiding this comment.
BulkIndexer 内部会定期 flush，下面的 Close 方法也会 flush
BytePender
commented
Jan 13, 2025
components/indexer/es8/indexer.go
Outdated
| tuples := make([]tuple[string, int], 0, len(fields)) | ||
| texts := make([]string, 0, len(fields)) | ||
| for k, text := range needEmbeddingFields { | ||
| tuples = append(tuples, tuple[string, int]{k, len(texts)}) |
There was a problem hiding this comment.
tuple 这个语义应该是 pair
There was a problem hiding this comment.
texts 变量中， 长度为啥使用 fields， 而不是 needEmbeddingFields？
There was a problem hiding this comment.
len(texts) 表达的是这个 tuple 对应在 texts 中的位置？
BytePender
commented
Jan 13, 2025
components/indexer/es8/indexer.go
Outdated
| } | ||
|
|
||
| for _, t := range tuples { | ||
| fields[t.A] = vectors[t.B] |
There was a problem hiding this comment.
原来 fields 中的 value 应该是啥， 是不是不应该更改 fields 这个 map[string]any
BytePender
commented
Jan 13, 2025
components/indexer/es8/indexer.go
Outdated
| } | ||
| } | ||
|
|
||
| b, err := json.Marshal(fields) |
There was a problem hiding this comment.
看是否需要把 json 换乘 sonic
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.