Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parquet Statistics - deprecate has_* APIs and add _opt functions that return Option<T> #6216

Merged
merged 24 commits into from
Aug 15, 2024

Commits on Aug 9, 2024

  1. update public api Statistics::min to return an option.

    I first re-named the existing method to `min_unchecked` and made it
    internal to the crate.
    
    I then added a `pub min(&self) -> Opiton<&T>` method.
    
    I figure we can first change the public API before deciding what to do
    about internal usage.
    
    Ref: apache#6093
    Michael-J-Ward committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    fd2ac15 View commit details
    Browse the repository at this point in the history
  2. update public api Statistics::max to return an option.

    I first re-named the existing method to `max_unchecked` and made it
    internal to the crate.
    
    I then added a `pub max(&self) -> Opiton<&T>` method.
    
    I figure we can first change the public API before deciding what to do
    about internal usage.
    
    Ref: apache#6093
    Michael-J-Ward committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    18b467b View commit details
    Browse the repository at this point in the history
  3. cargo fmt

    Michael-J-Ward committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    7ffac1a View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    87c375a View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    6950994 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    c65a773 View commit details
    Browse the repository at this point in the history
  7. make Statistics::null_count return Option<u64>

    This removes ambiguity around whether the between all values are non-null or just that the null count stat is missing
    
    Ref: apache#6215
    Michael-J-Ward committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    89ea21d View commit details
    Browse the repository at this point in the history
  8. update expected metadata memory size tests

    Changing null_count from u64 to Option<u64> increases the memory size and layout of the metadata.
    
    I included these tests as a separate commit to call extra attention to it.
    Michael-J-Ward committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    f2785f8 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    9a4921f View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. Apply suggestions from code review

    Co-authored-by: Andrew Lamb <[email protected]>
    Michael-J-Ward and alamb committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    7e082a7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ddd8498 View commit details
    Browse the repository at this point in the history
  3. rename new optional ValueStatistics::max to max_opt

    Per PR review, we will deprecate the old API instead of introducing a brekaing change.
    
    Ref: apache#6216 (review)
    Michael-J-Ward committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    e92321b View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e4db777 View commit details
    Browse the repository at this point in the history
  5. add Statistics:{min,max}_bytes_opt

    This adds the API and migrates all of the test usage.
    The old APIs will be deprecated next.
    Michael-J-Ward committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    bd83a38 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. Configuration menu
    Copy the full SHA
    4e2909b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2f3e21e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    050ba89 View commit details
    Browse the repository at this point in the history
  4. remove has_min_max_set check from make_decimal_stats_iterator

    The check is unnecessary now that the stats funcs return Option<T> when unset.
    Michael-J-Ward committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    fc0ed20 View commit details
    Browse the repository at this point in the history
  5. deprecate has_min_max_set

    An internal version was also created because it is used so extensively in testing.
    Michael-J-Ward committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    2366942 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    82b40b8 View commit details
    Browse the repository at this point in the history
  7. remove redundant test assertions of stats._internal_has_min_max_set

    This removes the assertion from any test that subsequently unwraps both
    min_opt and max_opt.
    Michael-J-Ward committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    3efc631 View commit details
    Browse the repository at this point in the history
  8. replace negated test assertions of stats._internal_has_mix_max_set wi…

    …th assertions on min_opt and max_opt
    
    This removes all use of Statistics::_internal_has_min_max_set from the code base, and so it is also removed.
    Michael-J-Ward committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    a23cf8b View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. Configuration menu
    Copy the full SHA
    7d4e650 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    48a35d3 View commit details
    Browse the repository at this point in the history