[Epic] Complete Initial StringView in DataFusion

[alamb]

Is your feature request related to a problem or challenge?

This ticket is a follow on to #10918 where we implemented enough initial support for StringView / BinaryView that we can show some pretty sweet ClickBench results

Describe the solution you'd like

This epic tracks remaining work to complete the "initial" work which I would like to define as "enable using StringView when reading Strings from Parquet by default"

I am sure there will be additional work / support to add StringView to various other features of DataFusion that we can maybe track with another follow on ticket

Required for enabling StringView by default:

• Enable reading StringView by default from Parquet (schema_force_string_view) by default #11682
• Support string concat || for StringViewArray #11766
• COUNT(DISTINCT) on StringView panics: unreachable code: Utf8/Binary should use ArrowBytesSet #11767
• Support protobuf serialization for ScalarValue::Utf8View and ScalarValue::BinaryView #12117
• Support substrait serialization for ScalarValue::Utf8View and ScalarValue::BinaryView #12118
• Convert Utf8View/BinaryView --> Utf8 / Binary at output #12119
• Parquet statistics missing when reading Utf8 as Utf8View #12123
• Support StringView for binary operators like ~, !~, etc #12180
• Support applying parquet bloom filters to StringView columns #12499
• Support Binary --> String coercion for StringView/BinaryView in LIKE #12500
• Casting from Binary --> Utf8 to evaluate LIKE slows down some ClickBench queries  #12509

Could work around but really should be fixed upstream

• Support casting BinaryView --> Utf8 and LargeUtf8 arrow-rs#6162
• Add support for StringView and BinaryView statistics in StatisticsConverter arrow-rs#6164
• Make StringViewArray::slice() and BinaryViewArray::slice() faster / non allocating arrow-rs#6408

Additional "Nice to have" Features

• [Epic] Native StringView support for string functions #11790
• Reduce copying in CoalesceBatchesExec for StringViews #11628
• Cast Utf8 -> Utf8View (not the other way around) for binary operators #11881
• Improve performance of SUBSTR for StringViewArray #12031
• Improve performance of REPEAT functions #12015
• [Epic] Complete Initial StringView in DataFusion #11752
• Automate testing / ensuring that string functions get the same answer for String, LargeString, StringView, DictionaryString, etc  #12415

[alamb]

An update here is that @XiangpengHao has a PR with various changes in #11862

We still need to check that PR and figure out what else is in that PR is needed to be enabled "for real" (with tests, etc)

[alamb]

My ideal resolution here is that we end up in the state where the only change we need to enable string view by default is switch the config setting. I will do some more ticket triage later today to outline other items I know of

[2010YOUY01]

Do we have tickets for regexp binary operators? (like ~, !~...)
https://datafusion.apache.org/user-guide/sql/operators.html#op-re-match

I noticed stringview is not supported on them yet and they have separate implementation than regexp functions

Details

/*DML*/CREATE TABLE t0(v0 DOUBLE, v1 DOUBLE, v2 BOOLEAN, v3 BOOLEAN, v4 BOOLEAN, v5 STRING);
/*DML*/INSERT INTO t0(v1, v5, v2) VALUES (0.7183242196192607, 'Tn', true);
/*DML*/CREATE TABLE t0_stringview AS SELECT v0, v1, v2, v3, v4, arrow_cast(v5, 'Utf8View') as v5 FROM t0;

> select v5 ~ 'foo' from t0_stringview;
Internal error: Data type Utf8View not supported for binary_string_array_flag_op_scalar operation 'regexp_is_match' on string array.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

> select regexp_match(v5, 'foo') from t0_stringview;
+--------------------------------------------+
| regexp_match(t0_stringview.v5,Utf8("foo")) |
+--------------------------------------------+
|                                            |
+--------------------------------------------+
1 row(s) fetched.
Elapsed 0.034 seconds.

[alamb]

Do we have tickets for regexp binary operators? (like , !...)

Not that I know of -- it would be great to add them

[alamb]

Do we have tickets for regexp binary operators? (like , !...)

Filed #12180

[alamb]

I am going to try and polish up PR to enable string view by default PR (with the arrow upgrade and various recent improvements) and see how close we are #12092

[alamb]

StringView by default is finally merged into DataFusion: #13101 so I am claiming success and completion of this issue