feat: add SearchAfterMixin for ES search_after capability #4536
Conversation
Ali-D-Akbar
force-pushed
the
aakbar/PROD-4233
branch
from
2262eb3 to
a70ebb7
Compare
Ali-D-Akbar
force-pushed
the
aakbar/PROD-4233
branch
4 times, most recently
from
2b92b48 to
d93f032
Compare
| mock_document = MagicMock() | ||
| mock_search = MagicMock() | ||
| mock_document.search.return_value = mock_search | ||
|
|
||
| mock_search.query.return_value = mock_search | ||
| mock_search.sort.return_value = mock_search | ||
| mock_search.extra.return_value = mock_search | ||
|
|
||
| mock_result1 = MagicMock() | ||
| mock_result1.pk = 1 | ||
| mock_result1.meta.sort = ["sort1"] | ||
|
|
||
| mock_result2 = MagicMock() | ||
| mock_result2.pk = 2 | ||
| mock_result2.meta.sort = ["sort2"] | ||
|
|
||
| mock_search.execute.side_effect = [ | ||
| [mock_result1, mock_result2], | ||
| [], | ||
| ] |
There was a problem hiding this comment.
This is not the right way to test. This is mocking everything, including ES search after behavior. All the ES related tests in course-discovery hit test ES instance. The search after mixin tests should aim for that too.
There was a problem hiding this comment.
Updated the tests to use ElasticsearchTestMixin instead.
| break | ||
|
|
||
| all_ids.update(ids) | ||
| search_after = results[-1].meta.sort |
There was a problem hiding this comment.
add safety check here, like
search_after = results[-1].meta.sort if results[-1] else NoneAlso, have you tried it locally? By default, results[-1].meta.sort is a Elastic.AttrList, which is not json serializable. Using results[-1].meta.sort.copy() will convert it into a simple list
There was a problem hiding this comment.
Nice catch for the search_after check.
I've tested this locally, I'm using this search_after value just to be sent back to the ES query execution.
Ali-D-Akbar
force-pushed
the
aakbar/PROD-4233
branch
from
d93f032 to
7a8f3b7
Compare
| identified_course_ids.update( | ||
| i.key | ||
| for i in CourseRun.search( | ||
| query, | ||
| partner=ESDSLQ('term', partner=partner.short_code), | ||
| identifiers=ESDSLQ('terms', **{'key.raw': course_run_ids}) | ||
| ).source(['key']) | ||
| ) | ||
| if course_uuids: | ||
| course_uuids = [UUID(course_uuid) for course_uuid in course_uuids.split(',')] | ||
| specified_course_ids += course_uuids | ||
|
|
||
| log.info(f"Specified course ids: {specified_course_ids}") | ||
| identified_course_ids.update( | ||
| Course.search( | ||
| query, | ||
| partner=ESDSLQ('term', partner=partner.short_code), | ||
| identifiers=ESDSLQ('terms', **{'uuid': course_uuids}) | ||
| ).values_list('uuid', flat=True) |
There was a problem hiding this comment.
Aside from this code that updates the querying logic for ES (so rather than fetching all the products AND THEN filter it, we're fetching only the necessary products), this is just a copy of https://github.com/openedx/course-discovery/blob/a948a741b1839e845e07c762bab6ee51622daea5/course_discovery/apps/api/v1/views/catalog_queries.py
There was a problem hiding this comment.
Is this change not compatible with v1?
There was a problem hiding this comment.
It can be, yes.
There was a problem hiding this comment.
Requires changing the pkSearchableMixin a bit as well if we were to keep a single CatalogQueryContainsViewSet.
Ali-D-Akbar
force-pushed
the
aakbar/PROD-4233
branch
from
7a8f3b7 to
de8f6a3
Compare
PROD-4233
Adds a new
SearchAfterMixinto be added in place ofPkSearchableMixinthat allows using search_after. Once this mixin is used, it will bypass the default search limit of 10k by making multiple calls to ES in case we have more than 10k records in an index.Previously, we faced an issue regarding the search limit, resulting in less records to be returned. We increased the
MAX_RESULT_WINDOWbefore but a better way is to use search_after capability for an optimal and flexible result.Testing Instructions:
update_indexlocally in Discovery shell.PkSearchableMixinwithSearchAfterMixinin Course model./api/v1/catalog/query_contains/endpoint and add a sample query like this: http://localhost:18381/api/v1/catalog/query_contains/?course_uuids=2de67490-f748-4efd-8532-b445f7ecc6f9,f9f1e100-668a-4fd5-a966-a127de1f69de&query=org:edXYou can set
ELASTICSEARCH_DSL_QUERYSET_PAGINATIONto your specific value in order to test the behavior. While the end result will be the same but this can affect the number of times the search_after mechanism is called.