Any plan to support filter pushdown?

[storm-dance]

Hi:
Like apache/arrow#37559, any plan to support filter pushdown or selection pushdown?

[tustvold]

Various forms of filter pushdown are supported

See https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.RowSelection.html and https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.RowFilter.html

These are low level interfaces, if you want a higher-level interface I'd recommend looking into DataFusion which utilises these interfaces and exposes a dataframe style API or SQL

[storm-dance]

Thanks for your reply. RowSelection based on consecutive ranges is well supported now.
However, performance is poor when deal with discrete rowSelection of delete files in the lakehouse scenario(such as iceberg).

[tustvold]

I'm not sure what your question is, we support filter pushdown, but parquet being a columnar format has tradeoffs involved. If you want deletes to not impact reads you will need to rewrite the files

[storm-dance]

Sorry, I didn't make myself clear.
The point I was making about poor performance refers to the row-level function calls (skip_records | read_records) brought by discrete selection. Perhaps it would be better to batch read all the data and then apply the filter.

[tustvold]

Yes the docs for https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.RowFilter.html discuss this

[jp0317]

The point I was making about poor performance refers to the row-level function calls (skip_records | read_records) brought by discrete selection.

i think this is also an issue in parquet c++ implementation, where skip/read with selective inputs can incur large overhead

[storm-dance]

Yes, there is already initial design plan with BMI. However, the progress is a little slow.

[tustvold]

I can't see any concrete benchmark numbers on apache/arrow#37559, IMO this should be the starting point for such discussions and the lack of such numbers might be why implementation has stalled.

Given the nature of parquet's encodings I am skeptical that it will ever be able to support disjoint selections efficiently, I would not expect BMI to yield major performance wins relative to the inherent complexity. For that you would need encodings that provide O(1) random access. Parquet only provides this when plain encoding non-nullable fixed width primitives. There are formats that provide this more generally, e.g. arrow IPC, however, this comes at the costs of significantly worse space efficiency.

[storm-dance]

Yes. But according to my understanding, the main purpose of using BMI is not to access only the required data, but to merge the fully read and filter data into one materialization. BMI can effectively reduce the branch overhead of whether to need to materialize data in the decode algorithm.

[tustvold]

I think there is a slight misunderstanding as to where the bottlenecks in parquet decoding lie and the relevance of the BMI instructions to speeding it up. The BMI instructions provide an efficient way to iterate consecutive ranges in a bitmask, arrow-rs has a number of specialized iterators for doing just this based on the compiler's ability to automatically use these instructions.

Modern parquet decoders, like arrow-rs and arrow-cpp, rely on decoding large numbers of values together to make use of SIMD instructions, and this in turn means that decode must be driven by such range based iteration. As such arrow-rs, and arrow-cpp which was based off the approach used by arrow-rs, already use range-based skipping and not bitmask based selection.

As such it is unclear to me what relevance BMI has to this process, we already have the range-based iteration we require, bitmasks would only be slower.

FWIW #1248 contains some details of how BMI and similar mechanisms are used to accelerate arrow-rs' filter kernels, in particular it is worth noting the BMI style iteration is only advantageous for very selective bitmasks - https://github.com/apache/arrow-rs/pull/1248/files#diff-473b09d0e22e5556a2a84cf20e96dd6e3a412e5a63e57d4b4f192b2f825c4318R35