provide a flowFromPaginatedQuery on ApolloClient to get all pages from a paginated query

[yogurtearl]

Use case

Would like get to have function to get a Flow of all pages from a paginated query.

Describe the solution you'd like

Add something like the flowFromPaginatedQuery function below.

Provide extractCursorFromResponse that describe how to get the cursor out of the response.
And provide queryFactory that describes how to construct a new query given a cursor.

fun <T> T?.optional() = Optional.presentIfNotNull(this)

suspend fun <TQueryData : Query.Data> ApolloClient.flowFromPaginatedQuery(
    extractCursorFromResponse: (ApolloResponse<TQueryData>) -> String?,
    queryFactory: (cursor: Optional<String>) -> Query<TQueryData>,
): Flow<TQueryData> = flow {
    var response: ApolloResponse<TQueryData>? = null
    do {
        response = query(
            queryFactory(response?.let { extractCursorFromResponse(it) }.optional())
        ).execute()
        response.data?.let { emit(it) }
    } while (response?.let { extractCursorFromResponse(it) } != null)
}

Using the example from the tutorial you would do something like this:

val launchFlow: Flow<LaunchListQuery.Data> =  
  apolloClient.flowFromPaginatedQuery( { it.data?.launches?.cursor } ) { cursor -> 
      LaunchListQuery(cursor.optional())
  }

[BoD]

Hi!

This could be convenient but could also be a bit dangerous as this will potentially make a lot of network requests in a loop if there are many pages. For this maybe we could add some kind of a limit parameter, but overall this also feels a bit too specific to reside in the library and since it is only a few lines I think it's OK to keep it in your project?

On a related note, we are looking at improving pagination scenarios related to the Normalized Cache, if you're interested you can monitor issue #3807.

[yogurtearl]

take(..) on the flow would limit the number of requests made.

this would make at most 3 requests.

val listFlow: Flow<LaunchListQuery.Data> =  
  apolloClient.flowFromPaginatedQuery( { it.data?.launches?.cursor } ) { cursor -> 
      LaunchListQuery(cursor.optional())
  }

// only makes 3 requests
val firstThreePages = listFlow.take(3)

Regarding

make a lot of network requests in a loop if there are many pages

that is a risk of any code that loops over multiple pages. :)

Also, the flow won't trigger another request until the "collection" of the previous items is complete.

e.g. this will make at most 1 request per second, because each collect takes at least one second.

listFlow.collect {
     // process the data  
     delay(1000)
}

maybe flowFromPaginatedQuery could reside in a separate apollo-flow artifact?

[BoD]

Hi! Sorry for the long delay! I think for now a separate artifact would be overkill but we could add this to the main artifact (apollo-runtime) as an @ApolloExperimental method. Do you want to open a PR for it?

[martinbonnin]

Hi 👋 We've decided to not implement this for now as it can be done at the callsite. If anyone feels strongly this should be included by default, feel free to send a pull request.