Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix collection_ops_tests for Spark 4.0 [databricks] #11414

Merged
merged 10 commits into from
Oct 12, 2024

Commits on Aug 30, 2024

  1. Fix collection_ops_tests for Spark 4.0.

    Fixes NVIDIA#11011.
    
    This commit fixes the failures in `collection_ops_tests` on Spark 4.0.
    
    On all versions of Spark, when a Sequence is collected with rows that exceed MAX_INT,
    an exception is thrown indicating that the collected Sequence/array is
    larger than permissible. The different versions of Spark vary in the
    contents of the exception message.
    
    On Spark 4, one sees that the error message now contains more
    information than all prior versions, including:
    1. The name of the op causing the error
    2. The errant sequence size
    
    This commit introduces a shim to make this new information available in
    the exception.
    
    Note that this shim does not fit cleanly in RapidsErrorUtils, because
    there are differences within major Spark versions. For instance, Spark
    3.4.0-1 have a different message as compared to 3.4.2 and 3.4.3.
    Likewise, the differences in 3.5.0, 3.5.1, 3.5.2.
    
    Signed-off-by: MithunR <[email protected]>
    mythrocks committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    b8bd960 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. Fixed formatting error.

    mythrocks committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    ee2eb81 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. Configuration menu
    Copy the full SHA
    1037b69 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. Configuration menu
    Copy the full SHA
    3bcf04f View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2024

  1. Review comments.

    This moves the construction of the long-sequence error strings into
    RapidsErrorUtils.  The process involved introducing many new RapidsErrorUtils
    classes, and using mix-ins of concrete implementations for the error-string
    construction.
    mythrocks committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    2427bf3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    cc4ae45 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. Configuration menu
    Copy the full SHA
    9798684 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0bfac65 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. Configuration menu
    Copy the full SHA
    fe704af View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Configuration menu
    Copy the full SHA
    4ae9e68 View commit details
    Browse the repository at this point in the history