Core: add variant type support

[aihuaxu]

This is to add some required changes in API and core module for Variant support, including:

• Add isVariantType() method for variant type
• Add variant support in schema visitors such as AssignIdFreshIds, ReassignIds, PruneColumns, etc. and TypeUtil.
• Add variant support in avro projection.
• Add test coverage to test out schema visitors, TypeUtil and Avro projection.

Part of: #10392

[aihuaxu]

cc @rdblue, @RussellSpitzer, @flyrain and @JonasJ-ap. This is to add the changes in core to support variant type.

[shohamyamin]

@aihuaxu very important feature that will allow a lot more options for iceberg, thank you for your contribution

[rdblue]

I think that the corresponding visit method should be updated to call a new visitor variant method. That will be cleaner.

The visitor should look for the variant logical type, so we will need to implement the logical type.

[aihuaxu]

You are right. This is to workaround before the logical type is added in Avro by using Iceberg variant type. Let me add a comment for this.

[rdblue]

If we are updating the visitor implementations, I think we should also update the visit method. We can also delay making thes changes.

[aihuaxu]

You mean handle variant in the following visit method? As I mentioned that we don't have variant logical type in Schema.getType() from avro, we can't handle that for now. Let me know if understand correctly. I think this is needed for metadata change and I can defer until then.

abstract class AvroCustomOrderSchemaVisitor<T, F> {
public static <T, F> T visit(Schema schema, AvroCustomOrderSchemaVisitor<T, F> visitor) {
switch (schema.getType()) {

[rdblue]

I don't think that this is going to exercise the path that we want to when the visitor and visit method are updated. This should create a record schema and validate that visiting a file schema gives the correct projection to request, by name.

[aihuaxu]

I will delay the change in BuildAvroProjection as mentioned above so I removed the test. Will add this back when making the code change.