bug: BigQuery backend table.to_pyarrow_batches is not honoring the chunk_size parameter

[ruiyang2015]

What happened?

for following code for bigquery:

c = ibis.bigquery.connect(...)
t = c.table('some table')
for y in t.to_pyarrow_batches(chunk_size=1_000_000):  # <- change this parameter does not take effect
  print(y.num_rows)

for our case, the record returned is always 4k rows instead of larger set we expect.
tried same code for Duckdb/Snowflake, both can return proper sized pyarrow table

What version of ibis are you using?

9.0.0

What backend(s) are you using, if any?

BigQuery

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[cpcloud]

It's possible that we're not encoding the chunk size in the right way. I recall there being some complexity around how paged results are related to chunks.

Maybe @tswast knows: is it possible to get back an exact chunk size (modulo the last chunk which will be <= the requested chunk size)?