Broken continuation results with ordered results with recent v3.31.2 change #3639
Assignees
Labels
Comments
|
@neildsh could you take a look? |
|
Hello @IsaacBoy thanks for reporting this issue. Would you mind attaching the following information:
Thanks! |
sure. And then from one using the continuation token with the filter property set The index policy is: |
|
neildsh Those diag logs were from a test environment that has a much smaller subset of data. Here are the logs from the actual prod environment where the issue is occurring. First call (no continuation token): and from the second call that uses the continuation token these returning 1 record: |
|
Thanks for sharing these. We will look into why the backend was not able to push the filters down to the composite index. |
|
Hello @IsaacBoy , thanks for reporting this issue and for sharing the diagnostics. We will change the backend and the continuation token format to accept values so that we can remove these filters from the query. This work is currently scheduled for next month. |
|
@neildsh Was this ever completed? |
|
Hi @IsaacBoy, The backend side of changes is done. We'll update the SDK once the backend changes are deployed. Thanks. |
|
Hi @IsaacBoy, I apologize for the delay. We found an issue with our backend implementation and had to make a change in our backend. The backend changes are currently rolling out and there are few more regions to go. I'm hoping to check-in the SDK changes in a week or two if everything goes as planned. Thanks. |
@neildsh @balaperu Hey its been a about 5 months. Hows this fix going? Thank you! |
|
Hi @IsaacBoy, really sorry about the delay. There is one thing or the other that keeps blocking this PR. The PR below has the fix. But I'm not able to merge it in, as our integration tests uses the public emulator which hasn't been updated with the required backend changes. Without the emulator changes, the tests are failing in the official run. I have been working with the emulator team to get them to release a new emulator. The current ETA for the emulator release is later this month and I'll be able to check it in after that. |
|
@balaperu No problem. Thanks for the update |
@balaperu Is this the version you were waiting on? |
|
The release notes doesn't seem to indicate the version that I was waiting for, so I checked with our emulator team. Looks like the release notes got prematurely merged and the emulator is getting released only later this week. |
|
The emulator got released yesterday. I'll resurface the PR and work on getting it checked in. |
|
The change has been checked in as part of this pull request - #3774 |
@balaperu do you happen to know if this fix was included in 3.39.0? |
|
Yes, I think it should be part of this release. The PR was marked as internal which is why it is not listed in the changelog.
From: Isaac ***@***.***>
Sent: Wednesday, April 10, 2024 6:10 PM
To: Azure/azure-cosmos-dotnet-v3 ***@***.***>
Cc: Bala Perumalswamy ***@***.***>; Mention ***@***.***>
Subject: Re: [Azure/azure-cosmos-dotnet-v3] Broken continuation results with ordered results with recent v3.31.2 change (Issue #3639)
The change has been checked in as part of this pull request - #3774<#3774>
@balaperu<https://github.com/balaperu> do you happen to know if this fix was included in 3.39.0?
https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/changelog.md
-
Reply to this email directly, view it on GitHub<#3639 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFH3FTJL4GPH2DQGADWUM4TY4XPFZAVCNFSM6AAAAAATW2OK7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBYG4ZDKNRRGQ>.
You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have confirmed the issue is comming from with the change in v3.31.2. It does not happen in v3.31.1.
When fetching ordered results that cause a continuation token to be returned, the first set of results are returned as normal, but with subsequent fetches, an "odd" filter query is returned with the continuation token, which then when passed into the next query execution causes only 1 record to be returned per call (and very poor performance). So for example, when you have a dataset of 1 million records, and you query with a max page size of 1000, you'd get 1000 records, and then after that 1 record per call. Again, its definitely related to the filter being a part of the continuation token because without it (as it works with the version of the sdk previous to this) the sdk behaves as expected (1000 or max page size results returned per call to cosmos)
SDK Version: v3.31.2
The situation seems to be possibly circumstantial under unknown specific conditions, and I'm not sure which are relevant. Once it happens it consistently happens. The query I'm performing is something like
where AuditDate is a number and DocumentType is a string. The query is cross-partition. The first query returns a continuation result like this:
but the subseuent ones return one with the following
filterpropertywhich only returns one result.
I do have a composite index setup that the query is utilizing, confirmed through IndexMetrics
With the continuation token without a filter:
With the continuation token with one"
The text was updated successfully, but these errors were encountered: