Integration data type test for new table format #850
Conversation
| "create or replace iceberg table %s (%s string, %s %s) " | ||
| + "catalog = 'SNOWFLAKE' " | ||
| + "external_volume = 'streaming_ingest' " | ||
| + "base_location = 'all_type_col';", |
There was a problem hiding this comment.
- lets use a different base_location for each table.
- Create a JIRA for how to cleanup all of the data that ITs upload, from the external volume.
| import org.junit.Ignore; | ||
| import org.junit.Test; | ||
|
|
||
| @Ignore("This test can be enabled after server side Iceberg EP support is released") |
There was a problem hiding this comment.
Add a backlog JIRA to track adding a different github merge gate for iceberg ITs, we'll then have one merge gate for FDN ITs and another for iceberg ITs.
| testIcebergIngestion("boolean", 1, true, new BooleanProvider()); | ||
| testIcebergIngestion("boolean", "false", false, new BooleanProvider()); |
There was a problem hiding this comment.
lets also add 0, "true", "1", "0"
|
|
||
| @Test | ||
| public void testBoolean() throws Exception { | ||
| testIcebergIngestion("boolean", true, new BooleanProvider()); |
There was a problem hiding this comment.
nit: whenever you have time in the next few weeks, lets introduce a class called ITTestcaseBuilder that has methods withDataType, withInputValue, withExpectedValue, withVerificationQuery.
THe provider business is funky as it only supports doing exact value verification with a where value = {expectedValue}. I'd much rather write the sql filter directly in the testcase instead of have code generate it, for flexibility.
| testIcebergIngestion("boolean", "false", false, new BooleanProvider()); | ||
|
|
||
| SFException ex = | ||
| Assert.assertThrows( |
There was a problem hiding this comment.
pl use fluent assertions (assertj) as they (a) have test code that reads like a story, and (b) generate very well formatted error messages. Assert.assert*() has the most useless error messages when things fail unless every single call to the method has an associated string message that details the current state of whatever was being asserted on.
For instance:
assertThat(ex).isNotNull();
assertThat(ex.getVendorCode).isEqualTo(ErrorCode.INVALID_FORMAT_ROW.getMessageCode());
assertThatCode(() -> lambda).doesNotThrow();
assertThatThrownBy(() -> lambda).hasCause();
For this you'll have to add the assertj library (I added it once and it failed in one of the mergegates, did not have the time to find and fix what happened!). Feel free to remove the org.hamcrest import from this project.
| return tableName; | ||
| } | ||
|
|
||
| protected String createIcebergTable(String dataType) throws SQLException { |
There was a problem hiding this comment.
also add an argument for boolean nullable and we can make sure the sdk does proper nullability checking for columns marked non-nullable.
| Assert.assertThrows( | ||
| SFException.class, | ||
| () -> testIcebergIngestion("boolean", new Object(), true, new BooleanProvider())); | ||
| Assert.assertEquals(ErrorCode.INVALID_FORMAT_ROW.getMessageCode(), ex.getVendorCode()); |
There was a problem hiding this comment.
(dupe) lets add nullability tests too - sending in null should work for every column, but when its a non-null column then the sdk should throw.
| SFException ex = | ||
| Assert.assertThrows( | ||
| SFException.class, | ||
| () -> testIcebergIngestion("int", Long.MAX_VALUE, new LongProvider())); |
There was a problem hiding this comment.
nit: (Long)Integer.MAX_VALUE + 1 instead of long.maxvalue?
Created data type IT for Iceberg table and verified through local VM. This test should be disabled until the sever side change is release.