API: Add Variant data type

[aihuaxu]

Add Variant data type to API module. We only add limited required type to API for now and we will promote stable methods to API after full implementation to the interface VariantLike later.

• Add Variant as a new data type

Fix #11178.

[rdblue]

What is the rationale for this size?

[aihuaxu]

We can't have the accurate size for Variant similar to Binary. So I use the same value as Binary. I'm wondering how we come up with 80 for Binary.

[rdblue]

This is okay but we may be missing information by not sanitizing based on the variant's type (i.e. date) and it would be nice to have some idea of the structure in the future.

[RussellSpitzer]

Are you suggesting something like

{
 foo:
   bar: 3 
   baz: 
      bozz: "flew"
 } 

to

{ 
  (hash-foo):
     (hash-bar) : (1 digit number)
     (hash-baz) :
        (hash-bozz) : (hash-xaxa)
}

[RussellSpitzer]

I think that would be a nice feature but probably ok for it's own issue

[aihuaxu]

Thanks for helping me understand the concept. I have filed a followup issue #11479 and will work on separately.

[RussellSpitzer]

don't we need to fix "canTransform" as well?

[RussellSpitzer]

Nvm I see that's already covered since "variant" is not considered a primitive

[RussellSpitzer]

Preconditions.checkArgument(type.typeId() != Types.VariantType);

So we can avoid adding isVariantType to the interface

[aihuaxu]

Since Variant is not a primitive type, canTransform() will be false.

[RussellSpitzer]

This is only used in a deprecated method, do we have any other reason to add this? I think it probably doesn't need to be apart of the type interface. We could always just check if the type is VARIANT

[aihuaxu]

Yeah. This is more a helper function instead of checking if the type is VARIANT. Actually it will be used in some other places later.

[RussellSpitzer]

I know this is copying a lot of tests in this class but we should start future proofing a bit more imho. We also have tests around this sort of thing in TestSchema.java. I think it is probably ok to just keep all of our schema validation tests there, but it wouldn't hurt to have some redundancy here as well.

Refactoring the whole suite can come in another pr but I think we should build a templated test that's something like

@ParameterizedTest
@ValueSource(types = {Types.TimetstampNanos, Types.Variant, ....})
testTypeSupport(Type type) {
  Schema schemaWithType =  new Schema(
          Types.NestedField.required(1, "id", Types.LongType.get()),
          Types.NestedField.optional(2, type.name, type),
          Types.NestedField.optional(3, "arr", Types.ListType.ofRequired(4, type)),
          Types.NestedField.required(5, "struct", 
            Types.StructType.of(
                  Types.NestedField.optional(6, "inner_" + type.name, type),
                  Types.NestedField.required(7, "data", Types.StringType.get()))),
          Types.NestedField.optional(8, "struct_arr",
              Types.StructType.of(
                  Types.NestedField.optional(9, "ts", type))));
     
    //Psuedo code here 
    from 0 -> MIN_FORMAT_VERSION.get(type)
       fail to make metadata
       
    from MIN_FORMAT_VERSION.get(type) -> SUPPORTED_TABLE_VERSION
       succeed
}

The most important part about this is that we wouldn't have to continually update tests every time a new valid metadata version is added. It also would be much easier to test type compatibility. (I'm thinking that Geo is going to need the exact same thing soon)

For this PR I think it is enough to write a parameterized version for just Variant, then we could raise another PR to add in nanos and remove the redundant tests

[RussellSpitzer]

Not this is part of my goal to remove all tests that have V3 or V2 in their title.

[RussellSpitzer]

Also todo add Variant to Schema.MIN_FORMAT_VERSION

[aihuaxu]

Thanks @RussellSpitzer for already creating a PR to address that. I will include that when it's merged.