GH-40592: [C++][Parquet] Implement SizeStatistics #40594

wgtmac · 2024-03-16T15:53:08Z

Rationale for this change

Parquet format 2.10.0 has introduced SizeStatistics. parquet-mr has also implemented this: apache/parquet-java#1177. Now it is time for parquet-cpp to pick the ball.

What changes are included in this PR?

Implement reading and writing size statistics for parquet-cpp.

Are these changes tested?

Yes, a bunch of test cases have been added.

Are there any user-facing changes?

Yes, now parquet users are able to read and write size statistics.

GitHub Issue: [C++][Parquet] Implement SizeStatistics #40592

cpp/src/parquet/size_statistics.h

emkornfield · 2024-05-02T23:17:14Z

cpp/src/parquet/column_writer.cc

@@ -1631,6 +1694,12 @@ class TypedColumnWriterImpl : public ColumnWriterImpl, public TypedColumnWriter<
      page_statistics_->UpdateSpaced(values, valid_bits, valid_bits_offset,
                                     num_spaced_values, num_values, num_nulls);
    }
+    if constexpr (std::is_same_v<T, ByteArray>) {
+      if (page_size_stats_builder_ != nullptr) {
+        page_size_stats_builder_->WriteValuesSpaced(values, valid_bits, valid_bits_offset,


I wonder if we could somehow gather this at a lower level (based on buffer size of written values instead of having to handle Spaced values separately)

I have thought about this. We have different interfaces to write values of BYTE_ARRAY type:

dense ByteArray values

spaced ByteArray values

arrow::Array of String, Binary, and their large variants

dictionary-encoded arrow::Array

These interfaces then directly put values into encoders. So here is the last chance to catch BYTE_ARRAY values before encoding.

emkornfield · 2024-05-02T23:19:46Z

cpp/src/parquet/column_writer.cc

+      page_statistics_->Update(*referenced_dictionary, /*update_counts=*/false);
+    }
+    if (page_size_stats_builder_) {
+      page_size_stats_builder_->WriteValues(*referenced_dictionary);


is this write, why are we writing values in the dictionary for page size stats? Maybe a comment or a a name value?

Yes, this is write. The binary values are passed in the form of an arrow::DictionaryArray and encoded indices in an arrow::Int32Array. Here we need to restore the referenced values in the dictionary array to precisely build page stats and size stats.

cpp/src/parquet/page_index.cc

cpp/src/parquet/size_statistics.h

emkornfield · 2024-05-02T23:27:06Z

cpp/src/parquet/size_statistics.h

+  /// \param[in] valid_bits pointer to bitmap representing if values are non-null.
+  /// \param[in] valid_bits_offset offset into valid_bits where the slice of data begins.
+  /// \param[in] num_spaced_values length of values in values/valid_bits to inspect.
+  void WriteValuesSpaced(const ByteArray* values, const uint8_t* valid_bits,


as commented above, I wonder if it is possible to not interwrine values spaced (and the Array option below) into this interface.

The logic is anyway required. Perhaps we can provide only the dense interface here and move the logic of dealing with nulls & arrow array to the caller?

cpp/src/parquet/size_statistics.cc

cpp/src/parquet/size_statistics.h

emkornfield

A few high level questions/suggestions.

wgtmac · 2024-07-10T15:35:42Z

Finally this PR is complete on my side. Please take a look when you have time. Thanks! @emkornfield @pitrou @mapleFU

pitrou

Sorry for the delay @wgtmac . This is a first partial review, I'll go over the rest once these comments are answered or addressed :-)

pitrou · 2024-07-22T15:11:33Z

cpp/src/parquet/size_statistics.h

+  /// \param size_statistics pointer to the thrift SizeStatistics structure.
+  /// \param descr column descriptor for the column.
+  /// \returns SizeStatistics object. Its lifetime is not bound to the input.
+  static std::unique_ptr<SizeStatistics> Make(const void* size_statistics,


If you're using the pimpl idiom, then you should just return a SizeStatistics here, since all the implementation is already inside a std::unique_ptr.

Conversely, you could also remove the pimpl idiom and return a subclass here instead. This is better if you want to be able to pass an optionally null pointer, or store a shared_ptr at some pointer.

I was following the pimpl idiom of class FileMetaData:

arrow/cpp/src/parquet/metadata.h

Lines 275 to 279 in 14384ac

/// \brief Create a FileMetaData from a serialized thrift message.

static std::shared_ptr<FileMetaData> Make(

const void* serialized_metadata, uint32_t* inout_metadata_len,

const ReaderProperties& properties = default_reader_properties(),

std::shared_ptr<InternalFileDecryptor> file_decryptor = NULLPTR);

Returning a SizeStatistics instead of std::unique_ptr<SizeStatistics> make it impossible to store it in a smart pointer, which is on the contrary of the convention in this codebase.

Returning a subclass requires implementing virtual functions, which will be called frequently at every batch. This is something I want to avoid.

cpp/src/parquet/size_statistics.h

pitrou · 2024-07-22T15:36:42Z

cpp/src/parquet/size_statistics.h

+  /// \brief Add repeated repetition level to the histogram.
+  /// \param num_levels number of repetition levels to add.
+  /// \param rep_level repeated repetition level value.
+  void AddRepetitionLevel(int64_t num_levels, int16_t rep_level);
+
+  /// \brief Add repeated definition level to the histogram.
+  /// \param num_levels number of definition levels to add.
+  /// \param def_level repeated definition level value.
+  void AddDefinitionLevel(int64_t num_levels, int16_t def_level);


Are these two really useful?

Ah, sorry! The name misled me. Can't we name them AddDefinitionLevels and AddRepetitionLevels? Otherwise, these looks like they are adding a single level.

Also, not sure why they're taking the explicit rep_level and def_level values. AFAICT, these are only useful to append levels equal to 0.

You're right. They are just used to append level value 0. It might look more strange if I special case a new function like AppendDefLevelZero(num_levels). And it is convenient to be used in the unit test so I am inclined to keep them.

pitrou · 2024-07-22T15:38:49Z

cpp/src/parquet/size_statistics.h

+  void AddValuesSpaced(const ByteArray* values, const uint8_t* valid_bits,
+                       int64_t valid_bits_offset, int64_t num_spaced_values);
+
+  /// \brief Add dense BYTE_ARRAY values.
+  /// \param values pointer to values of BYTE_ARRAY type.
+  /// \param num_values  length of values.
+  void AddValues(const ByteArray* values, int64_t num_values);
+
+  /// \brief Add BYTE_ARRAY values in the arrow array.
+  void AddValues(const ::arrow::Array& values);


Wouldn't it be more logical for the BYTE_ARRAY encoders to accumulate the unencoded_byte_array_data_bytes, instead of visiting the input data again here?

There are two cases where BYTE_ARRAY encoders do not work:

When dictionary encoding is enabled.

When the input data is in a arrow::DictionaryArray.

cpp/src/parquet/properties.h

cpp/src/parquet/column_writer.cc

emkornfield · 2024-08-06T05:13:09Z

going to do another pass through, CI failure looks like a formatting issue.

emkornfield · 2024-08-06T05:27:15Z

cpp/src/parquet/page_index.cc

+        /// Finalize unencoded_byte_array_data_bytes and make sure page sizes match.
+        if (offset_index_.page_locations.size() ==
+            offset_index_.unencoded_byte_array_data_bytes.size()) {
+          offset_index_.__isset.unencoded_byte_array_data_bytes = true;


The check above is short hand if anything isn't provided? we only expect two states they always match or they never match once page is added?

Yes, it should always match if size stats is enabled. Otherwise, we should expect the list is empty.

cpp/src/parquet/column_writer.cc

emkornfield

LGTM, I'm OK with this as long as @pitrou is thank you for driving this.

cpp/src/parquet/column_page.h

cpp/src/parquet/column_writer.cc

cpp/src/parquet/size_statistics.cc

wgtmac · 2024-08-07T15:29:23Z

@emkornfield @mapleFU Thanks for the feedback! I haven't addressed all comments from @pitrou yet. Will let you know once ready for review again.

wgtmac · 2024-08-22T06:53:38Z

This is ready for review again. Thanks in advance! @emkornfield @pitrou @mapleFU

wgtmac marked this pull request as draft March 16, 2024 15:53

github-actions bot added Component: Parquet Component: C++ awaiting review Awaiting review labels Mar 16, 2024

felipecrv reviewed Mar 17, 2024

View reviewed changes

cpp/src/parquet/size_statistics.h Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Mar 17, 2024

wgtmac force-pushed the size_stats branch from 044c265 to b573d0c Compare March 19, 2024 06:04

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 19, 2024

wgtmac force-pushed the size_stats branch from b573d0c to c776001 Compare March 26, 2024 05:57

wgtmac commented Apr 2, 2024

View reviewed changes

cpp/src/parquet/size_statistics.h Show resolved Hide resolved

wgtmac force-pushed the size_stats branch from c776001 to b762bf7 Compare April 5, 2024 15:38

wgtmac marked this pull request as ready for review April 5, 2024 15:39

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Apr 10, 2024

emkornfield reviewed May 2, 2024

View reviewed changes

cpp/src/parquet/page_index.cc Outdated Show resolved Hide resolved

emkornfield reviewed May 2, 2024

View reviewed changes

cpp/src/parquet/size_statistics.h Show resolved Hide resolved

emkornfield reviewed May 2, 2024

View reviewed changes

cpp/src/parquet/size_statistics.cc Show resolved Hide resolved

emkornfield reviewed May 2, 2024

View reviewed changes

cpp/src/parquet/size_statistics.cc Show resolved Hide resolved

emkornfield reviewed May 2, 2024

View reviewed changes

cpp/src/parquet/size_statistics.h Outdated Show resolved Hide resolved

emkornfield requested changes May 2, 2024

View reviewed changes

wgtmac force-pushed the size_stats branch from bb2e9cf to 82f4b34 Compare June 23, 2024 15:30

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jun 23, 2024

wgtmac force-pushed the size_stats branch from 095bd6a to a03caef Compare July 1, 2024 02:15

wgtmac force-pushed the size_stats branch from a03caef to 3c7cf12 Compare July 10, 2024 03:15

wgtmac force-pushed the size_stats branch 2 times, most recently from 8661324 to 90caf32 Compare July 10, 2024 15:32

pitrou requested changes Jul 22, 2024

View reviewed changes

emkornfield reviewed Aug 6, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Aug 6, 2024

emkornfield reviewed Aug 6, 2024

View reviewed changes

cpp/src/parquet/column_writer.cc Outdated Show resolved Hide resolved

emkornfield reviewed Aug 6, 2024

View reviewed changes

mapleFU reviewed Aug 6, 2024

View reviewed changes

cpp/src/parquet/column_page.h Show resolved Hide resolved

cpp/src/parquet/column_writer.cc Outdated Show resolved Hide resolved

cpp/src/parquet/size_statistics.cc Outdated Show resolved Hide resolved

cpp/src/parquet/size_statistics.cc Outdated Show resolved Hide resolved

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 21, 2024

wgtmac force-pushed the size_stats branch 2 times, most recently from 0449426 to a83ed41 Compare August 22, 2024 02:28

wgtmac mentioned this pull request Aug 22, 2024

[CI][Python][Parquet] Failed to open local file column_chunk_key_value_metadata.parquet #43785

Closed

wgtmac added 6 commits September 20, 2024 16:13

apacheGH-40592: [C++][Parquet] Implement SizeStatistics

abf6967

add basic test

1d3fdc5

rename builder functions

14fd0f8

add tests

465063a

apply modern c++ style

d4fbf92

address a couple of feedback

a1e80eb

wgtmac force-pushed the size_stats branch from a83ed41 to a1e80eb Compare September 20, 2024 08:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-40592: [C++][Parquet] Implement SizeStatistics #40594

GH-40592: [C++][Parquet] Implement SizeStatistics #40594

wgtmac commented Mar 16, 2024 •

edited

Loading

emkornfield May 2, 2024

wgtmac Jun 22, 2024

emkornfield May 2, 2024

wgtmac Jun 22, 2024

emkornfield May 2, 2024

wgtmac Jun 22, 2024

emkornfield left a comment

wgtmac commented Jul 10, 2024

pitrou left a comment •

edited

Loading

pitrou Jul 22, 2024

pitrou Jul 22, 2024

wgtmac Jul 24, 2024

pitrou Jul 22, 2024

pitrou Jul 22, 2024

pitrou Jul 22, 2024

wgtmac Aug 21, 2024

pitrou Jul 22, 2024

wgtmac Jul 24, 2024

emkornfield commented Aug 6, 2024

emkornfield Aug 6, 2024

wgtmac Aug 21, 2024

emkornfield left a comment

wgtmac commented Aug 7, 2024

wgtmac commented Aug 22, 2024

	/// \brief Create a FileMetaData from a serialized thrift message.
	static std::shared_ptr<FileMetaData> Make(
	const void* serialized_metadata, uint32_t* inout_metadata_len,
	const ReaderProperties& properties = default_reader_properties(),
	std::shared_ptr<InternalFileDecryptor> file_decryptor = NULLPTR);

GH-40592: [C++][Parquet] Implement SizeStatistics #40594

Are you sure you want to change the base?

GH-40592: [C++][Parquet] Implement SizeStatistics #40594

Conversation

wgtmac commented Mar 16, 2024 • edited Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emkornfield left a comment

Choose a reason for hiding this comment

wgtmac commented Jul 10, 2024

pitrou left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emkornfield commented Aug 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emkornfield left a comment

Choose a reason for hiding this comment

wgtmac commented Aug 7, 2024

wgtmac commented Aug 22, 2024

wgtmac commented Mar 16, 2024 •

edited

Loading

pitrou left a comment •

edited

Loading