Read Parquet metadata via suffix requests

[kylebarron]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

In high-latency environments, like in WebAssembly in users' browsers, minimizing the number of sequential requests can be a significant performance improvement. Now that #5222 has been merged, object_store now has the ability to fetch a suffix byte range of files. It would be great to be able to integrate this with parquet to reduce the number of individual requests required.

Describe the solution you'd like

It seems that parquet::arrow::async_reader::MetadataLoader::load can be refactored to use an initial suffix request instead of needing to know the file size.

Or, perhaps, there should be a new MetadataLoader::load_suffix method, so that implementations can choose to use load if they already know the file size, and use load_suffix if they don't.

Related to this, the ParquetObjectReader::new API requires an ObjectMeta, which requires knowing the file size. It would be great to be able to construct ParquetObjectReader with only the store and the object_store::path::Path. (I'm trying to construct ParquetObjectReader with a fake file length, passing my own ArrowReaderMetadata #5583, but I haven't figured out if that works yet)

Describe alternatives you've considered

Make an extra HEAD request instead of using suffix requests.

Additional context

[alamb]

Or, perhaps, there should be a new MetadataLoader::load_suffix method, so that implementations can choose to use load if they already know the file size, and use load_suffix if they don't.

It seems to me that load_suffix would always be preferred (as it requires less information), except for when the underlying store doesn't support suffix requests.

[kylebarron]

I think the API of AsyncFileReader might have to change? Right now the signature of get_bytes takes a range: core::ops::range::Range<usize>, and presumably that would have to change to something like object_store::GetRange for the concept of a suffix range?

If it does need a breaking API change, does that mean we shouldn't start a PR until 52.2.0 is tagged and released?

[alamb]

If it does need a breaking API change, does that mean we shouldn't start a PR until 52.2.0 is tagged and released?

It is up to you, but I think it would be better to get the PR up and ready (and we can merge it when main opens for 53.0.0 work

[alamb]

Potentially related: #6002

[alamb]

We now have a 53.0.0-dev branch where we are merging PRs with breaking API changes, in case you are still interested in submitting a PR