Pagination for QueryTableChanges

[charlenelyu-db]

• Part 1: Pagination for QueryTable at snapshot
  • Part 2: Pagination for QueryTable from starting_version
    • Part 3: Pagination for QueryTableChanges
      • Part 4: Client change

---

Support pagination for QueryTableChanges in reference server.

This is part 3 for issue #351.

[chakankardb]

Question: It seems first we will generate all the changes, and then apply pagination? So pagination will not save any processing on the server side right? Is the same true for DS server in UC as well? @linzhou-db

[charlenelyu-db]

We're saving some progress: the start will be equal to the startingVersion in the page token, so we won't unpack delta logs for versions that are already processed in previous pages.

However, we still generate all changes until the endingVersion each time. I guess this part can be optimized in UC. Two options I can think of:

1. we could push down actionListener to getChanges(), so we read logs and process actions together and it'll return in the middle
2. we don't change the interface of getChanges(), we call it to unpack one version at a time and process version by version.

WDYT?

[chakankardb]

I think both options should work; the first one is similar to what we do for normal query, so may be preferable for consistency reason.
In the future, we will likely process json files in parallel as well (similar to checkpoint parquet).

Let's open a ticket for now to address this in UC.

[linzhou-db]

let's have a separate discussion about how to handle this in UC.

[chakankardb]

Perhaps wait for Lin's review as well.