HPCC-31860 Test suite for the Parquet plugin #18980

ilhan2316 · 2024-08-09T17:33:32Z

Type of change:

This change is a bug fix (non-breaking change which fixes an issue).
This change is a new feature (non-breaking change which adds functionality).
This change improves the code (refactor or other change that does not change the functionality)
This change fixes warnings (the fix does not alter the functionality or the generated code)
This change is a breaking change (fix or feature that will cause existing behavior to change).
This change alters the query API (existing queries will have to be recompiled)

Checklist:

Smoketest:

Send notifications about my Pull Request position in Smoketest queue.
Test my draft Pull Request.

Testing:

jackdelv

@ilhan2316 I took a quick first look and had some questions and comments.

There were a few repeated style issues that I only pointed out a single example of, but make sure to check over all the files for things like ending newlines, correct copyright headers, etc.

Also, don't forget to remove any changes not associated with this PR.

Back to you.

testing/parquet/parquet_partition.ecl

testing/parquet/parquet_types.ecl

testing/regress/ecl/parquet_compress.ecl

testing/regress/ecl/parquet_types.ecl

testing/regress/ecl/parquet_write.ecl

testing/regress/ecl/parquet_schema.ecl

jackdelv

@ilhan2316 I think it is looking good. Don't be put off by the number of comments. Feel free to ask any questions that may come up.

Once you address these changes please take a look through the rest of the files to see if my comments apply in any other places. Once you are done I will look through the rest of the files.

esp/src/dgrid

plugins/parquet/parquetembed.cpp

plugins/parquet/parquetembed.hpp

testing/regress/ecl/parquet_compress.ecl

testing/regress/ecl/parquet_partition.ecl

testing/regress/ecl/parquet_schema.ecl

testing/regress/ecl/parquet_size.ecl

vcpkg

jackdelv

@ilhan2316 A few more comments, but I think it is close.

testing/regress/ecl/key/parquet_partition.xml

testing/regress/ecl/parquet_compress.ecl

testing/regress/ecl/parquet_overwrite.ecl

testing/regress/ecl/parquet_partition.ecl

testing/regress/ecl/parquet_size.ecl

testing/regress/ecl/parquet_types.ecl

jackdelv

@ilhan2316 I just took a look at your ECL files and have a few comments.

testing/regress/ecl/parquet_compress.ecl

testing/regress/ecl/parquet_partition.ecl

testing/regress/ecl/parquet_size.ecl

jackdelv · 2024-08-22T14:26:29Z

testing/regress/ecl/parquet_string.ecl

+
+outputDataset := ParquetIO.Read(recordLayout, filePath);
+
+compareDatasets := IF(COUNT(importedDataset) = COUNT(outputDataset),


Comment still stands.

jackdelv · 2024-08-22T14:26:59Z

testing/regress/ecl/parquet_types.ecl

+############################################################################## */
+
+//class=parquet
+//Cover's data type's supported by ECL and arrow


Comment still stands.

testing/regress/ecl/parquet_types.ecl

testing/regress/ecl/parquet_write.ecl

testing/regress/ecl/upload.ecl

jackdelv · 2024-09-16T15:55:07Z

@ilhan2316 This will need to be rebased to the latest version of 9.8.x to pull in the changes from this JIRA. It contains changes to data type sizes and could cause some of your key files to change.

Signed-off-by: Ilhan Gelle <[email protected]>

jackdelv

@ilhan2316 Don't be put off by the number of comments it is getting close. Feel free to leave questions on GitHub and I can add examples and clarification.

Some files used in your test cases are missing from the PR as well as files that are included but not referenced in your ECL. I have included the list below. Also, make sure you check back to the PR to see the results of the smoketests. Some of the test cases are failing.

Files I didn't see a reference to in the ECL:
diverse.parquet
edgecase1.parquet
intial.parquet
integertest.parquet
medium.parquet
small.parquet
time3.parquet
updated.parquet
Files missing that were referenced:
single.parquet
multi_1_of_3.parquet
multi_2_of_3.parquet
multi_3_of_3.parquet
Missing key files:
parquet_types.xml
parquet_schema.xml

testing/regress/ecl/key/parquet_empty.xml

jackdelv · 2024-09-25T16:06:10Z

testing/regress/ecl/parquet_partition.ecl

+############################################################################## */
+
+//class=parquet
+//fail


This is still marked fail. Try running it again as the fix should be in the latest version of 9.8.x.

jackdelv · 2024-09-25T16:08:28Z

testing/regress/ecl/parquet_partition.ecl

+ParquetIO.HivePartition.Write(
+    smallData,
+    rowSize,                            // Number of rows per file
+    '/var/lib/HPCCSystems/mydropzone/hive_partitioned/',


Use Std.File.GetDefaultDropZone() + '/regress/parquet/hive_partitioned/' to avoid cluttering up the dropzone root directory and assuming the platform is installed in root.

For this file, I have fixed it in terms of the partitioning I believe, but this is the current error that I am still trying to figure out : assert(!"Unknown copy source type") failed - file: hqlcpp.cpp, line 11772

That doesn't look like an issue with the plugin. Perhaps your ECL is still incorrect. I noted a few issues in my comments.

jackdelv · 2024-09-25T16:08:43Z

testing/regress/ecl/parquet_partition.ecl

+ParquetIO.DirectoryPartition.Write(
+    smallData,                          // Data to write
+    rowSize,                            // Number of rows per file
+    '/var/lib/HPCCSystems/mydropzone/dir_partitioned/',


Same comment as line 43.

testing/regress/ecl/parquet_size.ecl

testing/regress/ecl/parquet_string.ecl

testing/regress/ecl/parquet_write.ecl

testing/regress/ecl/parquet_types.ecl

testing/regress/ecl/upload.ecl

jackdelv

@ilhan2316 Before I look at the changes could you please remove the files that shouldn't be there. You can refer to my earlier comment for the list of files that should be removed.

jackdelv · 2024-10-15T19:49:23Z

@ilhan2316 it looks like there are many of my comments that are still unresolved. Make sure you are going through the "Files Changed" section and opening each file to view my comments. Also, it helps me if you respond to the comments with what you changed, if you agree or disagree, or what your thoughts are rather than just resolving the thread.

jackdelv

@ilhan2316 I know this is a lot of comments, but I think they are overall easier to fix than they have previously been. If you want me to provide some ECL for these tests let me know and I can help you. Don't hesitate to ask questions either!

testing/regress/ecl/key/parqueCorrupt.xml

testing/regress/ecl/parquetEmpty.ecl

jackdelv · 2024-10-28T19:26:28Z

testing/regress/ecl/parquetPartition.ecl

+basePath := Std.File.GetDefaultDropZone() + 'regress/parquet/';
+
+// Define partition keys as string arrays
+hivePartitionKey := ['city'];


This could easily be the cause of your error. The FUNCTIONMACROs for Write expect a string and you are passing an array. The way to pass in multiple keys is like: 'year;month;day'

jackdelv · 2024-10-28T19:26:52Z

testing/regress/ecl/parquetPartition.ecl

+rowSize := 1024;  // Increased buffer size
+
+// Define base path
+basePath := Std.File.GetDefaultDropZone() + 'regress/parquet/';


This needs to be '/regress/parquet/'.

jackdelv · 2024-10-28T19:28:18Z

testing/regress/ecl/parquetSize.ecl

+SEQUENTIAL(
+    OUTPUT(singleDataset, NAMED('singleDataset')),  // Output for the single file
+    OUTPUT(multiDataset, NAMED('multiDataset'))     // Output for the combined multi-part files
+);


Missing newline character at the end of the file.

jackdelv · 2024-10-28T19:47:51Z

testing/regress/ecl/parquetTypes.ecl

+
+setOfIntegerResult := IF(COUNT(setOfIntegerCompareResult(isEqual = FALSE)) = 0, 'Pass', 'Fail: SET OF INTEGER data mismatch');
+
+// ======================== REAL8 ========================


Rather than create a new section for each type variation you should just add an additional column to the existing REAL dataset on line 92.

jackdelv · 2024-10-28T19:49:06Z

testing/regress/ecl/parquetTypes.ecl

+
+setOfUnicodeResult := IF(COUNT(setOfUnicodeCompareResult(isEqual = FALSE)) = 0, 'Pass', 'Fail: SET OF UNICODE data mismatch');
+
+// ======================== INTEGER8 ========================


Same comment as above. Add repeating types to the first section of that type as an additional field. See the DATA section as an example on line 173.

To avoid making too many comments I haven't explicitly marked each occurrence, but there were a lot of the same cases that could be commoned up.

jackdelv · 2024-10-28T19:52:46Z

testing/regress/ecl/parquetWrite.ecl

I think this test can be deleted. The write functionality is thoroughly tested by the other tests.

jackdelv · 2024-10-28T19:53:58Z

testing/regress/ecl/parquetCompress.ecl

I think this file can be deleted and the functionality tested in parquetTypes.ecl (See my comment there).

Yeah. I agree with you on this.

jackdelv · 2024-10-28T19:56:17Z

testing/regress/ecl/parquet_partition.ecl

+ParquetIO.HivePartition.Write(
+    smallData,
+    rowSize,                            // Number of rows per file
+    '/var/lib/HPCCSystems/mydropzone/hive_partitioned/',


That doesn't look like an issue with the plugin. Perhaps your ECL is still incorrect. I noted a few issues in my comments.

ilhan2316 changed the title ~~Hpcc 31860 parquet plugin test suite~~ Hpcc-31860 parquet plugin test suite Aug 9, 2024

ilhan2316 force-pushed the HPCC-31860-Parquet-Plugin-Test-Suite branch from 905c5b2 to d148f19 Compare August 9, 2024 17:49

ilhan2316 changed the title ~~Hpcc-31860 parquet plugin test suite~~ Hpcc-31860 Test suite for the Parquet plugin Aug 9, 2024

jackdelv self-requested a review August 9, 2024 17:54

jackdelv requested changes Aug 9, 2024

View reviewed changes

ilhan2316 changed the title ~~Hpcc-31860 Test suite for the Parquet plugin~~ HPCC-31860 Test suite for the Parquet plugin Aug 12, 2024

ilhan2316 requested a review from jackdelv August 13, 2024 18:15

jackdelv requested changes Aug 13, 2024

View reviewed changes

jackdelv requested changes Aug 15, 2024

View reviewed changes

ilhan2316 requested a review from jackdelv August 22, 2024 14:03

jackdelv requested changes Aug 22, 2024

View reviewed changes

ilhan2316 added 7 commits September 17, 2024 22:13

HPCC-31860 Test Suite for the Parquet Plugin

41021f1

Updated with changes

6f2a6aa

Signed-off-by: Ilhan Gelle <[email protected]>

Fixed Issues

0462b92

Signed-off-by: Ilhan Gelle <[email protected]>

Restore files

ee77451

Signed-off-by: Ilhan Gelle <[email protected]>

Revert permissions

09d8802

Fixe's for compress & overwrite test's

3b82801

Signed-off-by: Ilhan Gelle <[email protected]>

Added changes to data type and size testing

92a925a

ilhan2316 force-pushed the HPCC-31860-Parquet-Plugin-Test-Suite branch from f50a03e to 92a925a Compare September 18, 2024 03:13

ilhan2316 added 2 commits September 18, 2024 12:04

Updates to size test

e6bca6f

update for compression testing

925bb36

ilhan2316 requested a review from jackdelv September 23, 2024 17:35

Changes to string and types testing

bb67c82

jackdelv requested changes Sep 25, 2024

View reviewed changes

ilhan2316 added 3 commits October 1, 2024 19:45

fixes to Parquet_write.ecl & Parquet_size.ecl & key files

84c24cc

string file updated and naming of suite

18baa23

Update for Types file testing

54fc536

ilhan2316 requested a review from jackdelv October 15, 2024 19:00

jackdelv requested changes Oct 15, 2024

View reviewed changes

files updated. Assert issue with partition

1221bfb

ilhan2316 requested a review from jackdelv October 28, 2024 18:15

jackdelv requested changes Oct 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HPCC-31860 Test suite for the Parquet plugin #18980

HPCC-31860 Test suite for the Parquet plugin #18980

ilhan2316 commented Aug 9, 2024 •

edited

Loading

jackdelv left a comment

jackdelv left a comment

jackdelv left a comment

jackdelv left a comment

jackdelv Aug 22, 2024

jackdelv Aug 22, 2024

jackdelv commented Sep 16, 2024 •

edited

Loading

jackdelv left a comment

jackdelv Sep 25, 2024

jackdelv Sep 25, 2024

ilhan2316 Oct 28, 2024

jackdelv Oct 28, 2024

jackdelv Sep 25, 2024

jackdelv left a comment

jackdelv commented Oct 15, 2024

jackdelv left a comment

jackdelv Oct 28, 2024

jackdelv Oct 28, 2024

jackdelv Oct 28, 2024

jackdelv Oct 28, 2024

jackdelv Oct 28, 2024

jackdelv Oct 28, 2024 •

edited

Loading

jackdelv Oct 28, 2024

jackdelv Oct 28, 2024

ilhan2316 Oct 29, 2024

jackdelv Oct 28, 2024


		outputDataset := ParquetIO.Read(recordLayout, filePath);

		compareDatasets := IF(COUNT(importedDataset) = COUNT(outputDataset),


		setOfIntegerResult := IF(COUNT(setOfIntegerCompareResult(isEqual = FALSE)) = 0, 'Pass', 'Fail: SET OF INTEGER data mismatch');

		// ======================== REAL8 ========================


		setOfUnicodeResult := IF(COUNT(setOfUnicodeCompareResult(isEqual = FALSE)) = 0, 'Pass', 'Fail: SET OF UNICODE data mismatch');

		// ======================== INTEGER8 ========================

HPCC-31860 Test suite for the Parquet plugin #18980

Are you sure you want to change the base?

HPCC-31860 Test suite for the Parquet plugin #18980

Conversation

ilhan2316 commented Aug 9, 2024 • edited Loading

Type of change:

Checklist:

Smoketest:

Testing:

jackdelv left a comment

Choose a reason for hiding this comment

jackdelv left a comment

Choose a reason for hiding this comment

jackdelv left a comment

Choose a reason for hiding this comment

jackdelv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackdelv commented Sep 16, 2024 • edited Loading

jackdelv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackdelv left a comment

Choose a reason for hiding this comment

jackdelv commented Oct 15, 2024

jackdelv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackdelv Oct 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilhan2316 commented Aug 9, 2024 •

edited

Loading

jackdelv commented Sep 16, 2024 •

edited

Loading

jackdelv Oct 28, 2024 •

edited

Loading