Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark 4: Fix parquet_test.py. #11519

Open
wants to merge 1 commit into
base: branch-24.12
Choose a base branch
from
Open

Spark 4: Fix parquet_test.py. #11519

wants to merge 1 commit into from

Commits on Sep 27, 2024

  1. Spark 4: Fix parquet_test.py. 

    Fixes NVIDIA#11015.
Contributes to NVIDIA#11004.

This commit addresses the tests that fail in parquet_test.py, when
run on Spark 4.

1. Some of the tests were failing as a result of NVIDIA#5114.  Those tests
have been disabled, at least until we get around to supporting
aggregations with ANSI mode enabled.

2. `test_parquet_check_schema_compatibility` fails on Spark 4 regardless
of ANSI mode, because it tests implicit type promotions where the read
schema includes wider columns than the write schema.  This will require
new code.  The test is disabled until NVIDIA#11512 is addressed.

3. `test_parquet_int32_downcast` had an erroneous setup phase that fails
   in ANSI mode.  This has been corrected. The test was refactored to
run in ANSI and non-ANSI mode.

Signed-off-by: MithunR <[email protected]>
    @mythrocks
    mythrocks committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    b712c2d View commit details
    Browse the repository at this point in the history