GH-590: Allow VARIANT value to be omitted

[HuaHuaY]

Rationale for this change

Allow VARIANT value to be omitted.

There are some cases in parquet-testing https://github.com/apache/parquet-testing/tree/master/shredded_variant which don't have value field.

# Doesn't have top value field.
parquet-testing/shredded_variant on  master 
❯ ~/arrow/cpp/out/build/ninja-debug/debug/parquet-dump-schema case-131.parquet 
required group field_id=-1 table {
  required int32 field_id=1 id;
  required group field_id=2 var (Variant(1)) {
    required binary field_id=-1 metadata;
    optional int32 field_id=-1 typed_value;
  }
}

# Doesn't have inner value field.
parquet-testing/shredded_variant on  master 
❯ ~/arrow/cpp/out/build/ninja-debug/debug/parquet-dump-schema case-132.parquet
required group field_id=-1 table {
  required int32 field_id=1 id;
  optional group field_id=2 var (Variant(1)) {
    required binary field_id=-1 metadata;
    optional binary field_id=-1 value;
    optional group field_id=-1 typed_value {
      required group field_id=-1 a {
        optional int32 field_id=-1 typed_value;
      }
      required group field_id=-1 b {
        optional binary field_id=-1 typed_value (String);
      }
    }
  }
}

What changes are included in this PR?

Updated the descriptions in the variant-related Markdown files.

Do these changes have PoC implementations?

No.

[HuaHuaY]

@alamb @emkornfield @zeroshade Please take a look. I found that arrow-rs already allows to omit value array. https://github.com/apache/arrow-rs/blob/7616e10f12894232eacb807993e55427a067d061/parquet-variant-compute/src/variant_array.rs#L858

[wgtmac]

Could you help review this? @aihuaxu @steveloughran

[alamb]

Perhaps @rdblue or @gene-db (the authors of the original spec in #456 #460) can help take a look

The example files were generated by spark so I assume they are now out in the wild and we shoudl adjust the spec accordingly

[rdblue]

I don't think this is correct.

The test cases are the ones that we use in Parquet Java and Iceberg to ensure that we can read variant data as long as it can be interpreted correctly. Being liberal about what we accept does not mean that it is a good idea for writers to produce values that way. We don't need to adjust the spec, this is just a reasonable choice for readers so they don't needlessly fail.

[wgtmac]

Thanks @rdblue for the explanation! Then I agree that this spec doesn't need any change. Perhaps we have two follow-ups to avoid future confusion:

• Fix wording in https://github.com/apache/arrow/blob/f10c93c000775c5fb30ca230f540358611091589/docs/source/format/CanonicalExtensions.rst?plain=1#L448 to align with the Parquet spec. cc @zeroshade @alamb
• Clarify the purpose of these test files in https://github.com/apache/parquet-testing/blob/master/shredded_variant/README.md