diff --git a/_posts/2024-09-13-string-view-german-style-strings-part-1.md b/_posts/2024-09-13-string-view-german-style-strings-part-1.md new file mode 100644 index 0000000..46b8645 --- /dev/null +++ b/_posts/2024-09-13-string-view-german-style-strings-part-1.md @@ -0,0 +1,245 @@ +--- +layout: post +title: "Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet" +date: "2024-09-13 00:00:00" +author: Xiangpeng Hao, Andrew Lamb +categories: [performance] +--- + + + + + +_Editor's Note: This is the first of a [two part] blog series that was first published on the [InfluxData blog]. Thanks to InfluxData for sponsoring this work as [Xiangpeng Hao]'s summer intern project_ + +This blog describes our experience implementing [StringView](https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout) in the [Rust implementation](https://github.com/apache/arrow-rs) of [Apache Arrow](https://arrow.apache.org/), and integrating it into [Apache DataFusion](https://datafusion.apache.org/), significantly accelerating string-intensive queries in the [ClickBench](https://benchmark.clickhouse.com/) benchmark by 20%- 200% (Figure 1[^1]). + +Getting significant end-to-end performance improvements was non-trivial. Implementing StringView itself was only a fraction of the effort required. Among other things, we had to optimize UTF-8 validation, implement unintuitive compiler optimizations, tune block sizes, and time GC to realize the [FDAP ecosystem](https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/)’s benefit. With other members of the open source community, we were able to overcome performance bottlenecks that could have killed the project. We would like to contribute by explaining the challenges and solutions in more detail so that more of the community can learn from our experience. + +StringView is based on a simple idea: avoid some string copies and accelerate comparisons with inlined prefixes. Like most great ideas, it is “obvious” only after [someone describes it clearly](https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf). Although simple, straightforward implementation actually _slows down performance for almost every query_. We must, therefore, apply astute observations and diligent engineering to realize the actual benefits from StringView. + +Although this journey was successful, not all research ideas are as lucky. To accelerate the adoption of research into industry, it is valuable to integrate research prototypes with practical systems. Understanding the nuances of real-world systems makes it more likely that research designs[^2] will lead to practical system improvements. + +StringView support was released as part of [arrow-rs v52.2.0](https://crates.io/crates/arrow/52.2.0) and [DataFusion v41.0.0](https://crates.io/crates/datafusion/41.0.0). You can try it by setting the `schema_force_view_types` [DataFusion configuration option](https://datafusion.apache.org/user-guide/configs.html), and we are[ hard at work with the community to ](https://github.com/apache/datafusion/issues/11682)make it the default. We invite everyone to try it out, take advantage of the effort invested so far, and contribute to making it better. + +[Xiangpeng Hao]: https://haoxp.xyz/ +[InfluxData blog]: https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/ +[two part]: {{ site.baseurl }}/2024/09/13/string-view-german-style-strings-part-2/ + + + + + +Figure 1: StringView improves string-intensive ClickBench query performance by 20% - 200% + + +## What is StringView? + + + + + + +Figure 2: Use StringArray and StringViewArray to represent the same string content. + +The concept of inlined strings with prefixes (called “German Strings” [by Andy Pavlo](https://x.com/andy_pavlo/status/1813258735965643203), in homage to [TUM](https://www.tum.de/), where the [Umbra paper that describes](https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf) them originated) +has been used in many recent database systems ([Velox](https://engineering.fb.com/2024/02/20/developer-tools/velox-apache-arrow-15-composable-data-management/), [Polars](https://pola.rs/posts/polars-string-type/), [DuckDB](https://duckdb.org/2021/12/03/duck-arrow.html), [CedarDB](https://cedardb.com/blog/german_strings/), etc.) +and was introduced to Arrow as a new [StringViewArray](https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout)[^3] type. Arrow’s original [StringArray](https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout) is very memory efficient but less effective for certain operations. +StringViewArray accelerates string-intensive operations via prefix inlining and a more flexible and compact string representation. + +A StringViewArray consists of three components: + + + +1. The view array +2. The buffers +3. The buffer pointers (IDs) that map buffer offsets to their physical locations + +Each view is 16 bytes long, and its contents differ based on the string’s length: + + + +* string length < 12 bytes: the first four bytes store the string length, and the remaining 12 bytes store the inlined string. +* string length > 12 bytes: the string is stored in a separate buffer. The length is again stored in the first 4 bytes, followed by the buffer id (4 bytes), the buffer offset (4 bytes), and the prefix (first 4 bytes) of the string. + +Figure 2 shows an example of the same logical content (left) using StringArray (middle) and StringViewArray (right): + + + +* The first string – `"Apache DataFusion"` – is 17 bytes long, and both StringArray and StringViewArray store the string’s bytes at the beginning of the buffer. The StringViewArray also inlines the first 4 bytes – `"Apac"` – in the view. +* The second string, `"InfluxDB"` is only 8 bytes long, so StringViewArray completely inlines the string content in the `view` struct while StringArray stores the string in the buffer as well. +* The third string `"Arrow Rust Impl"` is 15 bytes long and cannot be fully inlined. StringViewArray stores this in the same form as the first string. +* The last string `"Apache DataFusion"` has the same content as the first string. It’s possible to use StringViewArray to avoid this duplication and reuse the bytes by pointing the view to the previous location. + +StringViewArray provides three opportunities for outperforming StringArray: + + + +1. Less copying via the offset + buffer format +2. Faster comparisons using the inlined string prefix +3. Reusing repeated string values with the flexible `view` layout + +The rest of this blog post discusses how to apply these opportunities in real query scenarios to improve performance, what challenges we encountered along the way, and how we solved them. + + +## Faster Parquet Loading + +[Apache Parquet](https://parquet.apache.org/) is the de facto format for storing large-scale analytical data commonly stored LakeHouse-style, such as [Apache Iceberg](https://iceberg.apache.org) and [Delta Lake](https://delta.io). Efficiently loading data from Parquet is thus critical to query performance in many important real-world workloads. + +Parquet encodes strings (i.e., [byte array](https://docs.rs/parquet/latest/parquet/data_type/struct.ByteArray.html)) in a slightly different format than required for the original Arrow StringArray. The string length is encoded inline with the actual string data (as shown in Figure 4 left). As mentioned previously, StringArray requires the data buffer to be continuous and compact—the strings have to follow one after another. This requirement means that reading Parquet string data into an Arrow StringArray requires copying and consolidating the string bytes to a new buffer and tracking offsets in a separate array. Copying these strings is often wasteful. Typical queries filter out most data immediately after loading, so most of the copied data is quickly discarded. + +On the other hand, reading Parquet data as a StringViewArray can re-use the same data buffer as storing the Parquet pages because StringViewArray does not require strings to be contiguous. For example, in Figure 4, the StringViewArray directly references the buffer with the decoded Parquet page. The string `"Arrow Rust Impl"` is represented by a `view` with offset 37 and length 15 into that buffer. + + + + + +Figure 4: StringViewArray avoids copying by reusing decoded Parquet pages. + +**Mini benchmark** + +Reusing Parquet buffers is great in theory, but how much does saving a copy actually matter? We can run the following benchmark in arrow-rs to find out: + +Our benchmarking machine shows that loading _BinaryViewArray_ is almost 2x faster than loading BinaryArray (see next section about why this isn’t _String_ ViewArray). + + +You can read more on this arrow-rs issue: [https://github.com/apache/arrow-rs/issues/5904](https://github.com/apache/arrow-rs/issues/5904) + + +# From Binary to Strings + +You may wonder why we reported performance for BinaryViewArray when this post is about StringViewArray. Surprisingly, initially, our implementation to read StringViewArray from Parquet was much _slower_ than StringArray. Why? TLDR: Although reading StringViewArray copied less data, the initial implementation also spent much more time validating [UTF-8](https://en.wikipedia.org/wiki/UTF-8#:~:text=UTF%2D8%20is%20a%20variable,Unicode%20Standard) (as shown in Figure 5). + +Strings are stored as byte sequences. When reading data from (potentially untrusted) Parquet files, a Parquet decoder must ensure those byte sequences are valid UTF-8 strings, and most programming languages, including Rust, include highly[ optimized routines](https://doc.rust-lang.org/std/str/fn.from_utf8.html) for doing so. + + + +Figure 5: Time to load strings from Parquet. The UTF-8 validation advantage initially eliminates the advantage of reduced copying for StringViewArray. + +A StringArray can be validated in a single call to the UTF-8 validation function as it has a continuous string buffer. As long as the underlying buffer is UTF-8[^4], all strings in the array must be UTF-8. The Rust parquet reader makes a single function call to validate the entire buffer. + +However, validating an arbitrary StringViewArray requires validating each string with a separate call to the validation function, as the underlying buffer may also contain non-string data (for example, the lengths in Parquet pages). + +UTF-8 validation in Rust is highly optimized and favors longer strings (as shown in Figure 6), likely because it leverages SIMD instructions to perform parallel validation. The benefit of a single function call to validate UTF-8 over a function call for each string more than eliminates the advantage of avoiding the copy for StringViewArray. + + + +Figure 6: UTF-8 validation throughput vs string length—StringArray’s contiguous buffer can be validated much faster than StringViewArray’s buffer. + +Does this mean we should only use StringArray? No! Thankfully, there’s a clever way out. The key observation is that in many real-world datasets,[ 99% of strings are shorter than 128 bytes](https://www.vldb.org/pvldb/vol17/p148-zeng.pdf), meaning the encoded length values are smaller than 128, **in which case the length itself is also valid UTF-8** (in fact, it is [ASCII](https://en.wikipedia.org/wiki/ASCII)). + +This observation means we can optimize validating UTF-8 strings in Parquet pages by treating the length bytes as part of a single large string as long as the length _value_ is less than 128. Put another way, prior to this optimization, the length bytes act as string boundaries, which require a UTF-8 validation on each string. After this optimization, only those strings with lengths larger than 128 bytes (less than 1% of the strings in the ClickBench dataset) are string boundaries, significantly increasing the UTF-8 validation chunk size and thus improving performance. + +The [actual implementation](https://github.com/apache/arrow-rs/pull/6009/files) is only nine lines of Rust (with 30 lines of comments). You can find more details in the related arrow-rs issue:[ https://github.com/apache/arrow-rs/issues/5995](https://github.com/apache/arrow-rs/issues/5995). As expected, with this optimization, loading StringViewArray is almost 2x faster than loading StringArray. + + +# Be Careful About Implicit Copies + +After all the work to avoid copying strings when loading from Parquet, performance was still not as good as expected. We tracked the problem to a few implicit data copies that we weren't aware of, as described in[ this issue](https://github.com/apache/arrow-rs/issues/6033). + +The copies we eventually identified come from the following innocent-looking line of Rust code, where `self.buf` is a [reference counted](https://en.wikipedia.org/wiki/Reference_counting) pointer that should transform without copying into a buffer for use in StringViewArray. + +However, Rust-type coercion rules favored a blanket implementation that _did_ copy data. This implementation is shown in the following code block where the `impl>` will accept any type that implements `AsRef<[u8]>` and copies the data to create a new buffer. To avoid copying, users need to explicitly call `from_vec`, which consumes the `Vec` and transforms it into a buffer. + +Diagnosing this implicit copy was time-consuming as it relied on subtle Rust language semantics. We needed to track every step of the data flow to ensure every copy was necessary. To help other users and prevent future mistakes, we also [removed](https://github.com/apache/arrow-rs/pull/6043) the implicit API from arrow-rs in favor of an explicit API. Using this approach, we found and fixed several [other unintentional copies](https://github.com/apache/arrow-rs/pull/6039) in the code base—hopefully, the change will help other [downstream users](https://github.com/spiraldb/vortex/pull/504) avoid unnecessary copies. + + +# Help the Compiler by Giving it More Information + +The Rust compiler’s automatic optimizations mostly work very well for a wide variety of use cases, but sometimes, it needs additional hints to generate the most efficient code. When profiling the performance of `view` construction, we found, counterintuitively, that constructing **long** strings was 10x faster than constructing **short** strings, which made short strings slower on StringViewArray than on StringArray! + +As described in the first section, StringViewArray treats long and short strings differently. Short strings (<12 bytes) directly inline to the `view` struct, while long strings only inline the first 4 bytes. The code to construct a `view` looks something like this: + +It appears that both branches of the code should be fast: they both involve copying at most 16 bytes of data and some memory shift/store operations. How could the branch for short strings be 10x slower? + +Looking at the assembly code using [Compiler Explorer](https://godbolt.org/), we (with help from [Ao Li](https://github.com/aoli-al)) found the compiler used CPU **load instructions** to copy the fixed-sized 4 bytes to the `view` for long strings, but it calls a function, [`ptr::copy_non_overlapping`](https://doc.rust-lang.org/std/ptr/fn.copy_nonoverlapping.html), to copy the inlined bytes to the view for short strings. The difference is that long strings have a prefix size (4 bytes) known at compile time, so the compiler directly uses efficient CPU instructions. But, since the size of the short string is unknown to the compiler, it has to call the general-purpose function ptr::copy_non_coverlapping. Making a function call is significant unnecessary overhead compared to a CPU copy instruction. + +However, we know something the compiler doesn’t know: the short string size is not arbitrary—it must be between 0 and 12 bytes, and we can leverage this information to avoid the function call. Our solution generates 13 copies of the function using generics, one for each of the possible prefix lengths. The code looks as follows, and [checking the assembly code](https://godbolt.org/z/685YPsd5G), we confirmed there are no calls to `ptr::copy_non_overlapping`, and only native CPU instructions are used. For more details, see [the ticket](https://github.com/apache/arrow-rs/issues/6034). + + +# End-to-End Query Performance + +In the previous sections, we went out of our way to make sure loading StringViewArray is faster than StringArray. Before going further, we wanted to verify if obsessing about reducing copies and function calls has actually improved end-to-end performance in real-life queries. To do this, we evaluated a ClickBench query (Q20) in DataFusion that counts how many URLs contain the word `"google"`: + +This is a relatively simple query; most of the time is spent on loading the “URL” column to find matching rows. The query plan looks like this: + +We ran the benchmark in the DataFusion repo like this: + +With StringViewArray we saw a 24% end-to-end performance improvement, as shown in Figure 7. With the `--string-view` argument, the end-to-end query time is `944.3 ms, 869.6 ms, 861.9 ms` (three iterations). Without `--string-view`, the end-to-end query time is `1186.1 ms, 1126.1 ms, 1138.3 ms`. + + + + +Figure 7: StringView reduces end-to-end query time by 24% on ClickBench Q20. + +We also double-checked with detailed profiling and verified that the time reduction is indeed due to faster Parquet loading. + + +## Conclusion + +In this first blog post, we have described what it took to improve the +performance of simply reading strings from Parquet files using StringView. While +this resulted in real end-to-end query performance improvements, in our [next +post], we explore additional optimizations enabled by StringView in DataFusion, +along with some of the pitfalls we encountered while implementing them. + + +[next post]: https://datafusion.apache.org/blog/2024/09/13/using-stringview-to-make-queries-faster-part-2.html + + +# Footnotes + +[^1]: Benchmarked with AMD Ryzen 7600x (12 core, 24 threads, 32 MiB L3), WD Black SN770 NVMe SSD (5150MB/4950MB seq RW bandwidth) + +[^2]: Xiangpeng is a PhD student at the University of Wisconsin-Madison + +[^3]: There is also a corresponding _BinaryViewArray_ which is similar except that the data is not constrained to be UTF-8 encoded strings. + +[^4]: We also make sure that offsets do not break a UTF-8 code point, which is [cheaply validated](https://github.com/apache/arrow-rs/blob/master/parquet/src/arrow/buffer/offset_buffer.rs#L62-L71). diff --git a/_posts/2024-09-13-string-view-german-style-strings-part-2.md b/_posts/2024-09-13-string-view-german-style-strings-part-2.md new file mode 100644 index 0000000..44b08b1 --- /dev/null +++ b/_posts/2024-09-13-string-view-german-style-strings-part-2.md @@ -0,0 +1,176 @@ +--- +layout: post +title: "Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations" +date: "2024-09-13 00:00:00" +author: Xiangpeng Hao, Andrew Lamb +categories: [performance] +--- + + + + +_Editor's Note: This blog series was first published on the [InfluxData blog]. Thanks to InfluxData for sponsoring this work as [Xiangpeng Hao]'s summer intern project_ + +In the [first post], we discussed the nuances required to accelerate Parquet loading using StringViewArray by reusing buffers and reducing copies. +In this second part of the post, we describe the rest of the journey: implementing additional efficient operations for real query processing. + +[Xiangpeng Hao]: https://haoxp.xyz/ +[InfluxData blog]: https://www.influxdata.com/blog/faster-queries-with-stringview-part-two-influxdb/ +[first post]: {{ site.baseurl }}/2024/09/13/string-view-german-style-strings-part-1/ + +## Faster String Operations + +# Faster comparison + +String comparison is ubiquitous; it is the core of +[`cmp`](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/index.html), +[`min`](https://docs.rs/arrow/latest/arrow/compute/fn.min.html)/[`max`](https://docs.rs/arrow/latest/arrow/compute/fn.max.html), +and [`like`](https://docs.rs/arrow/latest/arrow/compute/kernels/comparison/fn.like.html)/[`ilike`](https://docs.rs/arrow/latest/arrow/compute/kernels/comparison/fn.ilike.html) kernels. StringViewArray is designed to accelerate such comparisons using the inlined prefix—the key observation is that, in many cases, only the first few bytes of the string determine the string comparison results. + +For example, to compare the strings `InfluxDB` with `Apache DataFusion`, we only need to look at the first byte to determine the string ordering or equality. In this case, since `A` is earlier in the alphabet than `I,` `Apache DataFusion` sorts first, and we know the strings are not equal. Despite only needing the first byte, comparing these strings when stored as a StringArray requires two memory accesses: 1) load the string offset and 2) use the offset to locate the string bytes. For low-level operations such as `cmp` that are invoked millions of times in the very hot paths of queries, avoiding this extra memory access can make a measurable difference in query performance. + +For StringViewArray, typically, only one memory access is needed to load the view struct. Only if the result can not be determined from the prefix is the second memory access required. For the example above, there is no need for the second access. This technique is very effective in practice: the second access is never necessary for the more than [60% of real-world strings which are shorter than 12 bytes](https://www.vldb.org/pvldb/vol17/p148-zeng.pdf), as they are stored completely in the prefix. + +However, functions that operate on strings must be specialized to take advantage of the inlined prefix. In addition to low-level comparison kernels, we implemented [a wide range](https://github.com/apache/arrow-rs/issues/5374) of other StringViewArray operations that cover the functions and operations seen in ClickBench queries. Supporting StringViewArray in all string operations takes quite a bit of effort, and thankfully the Arrow and DataFusion communities are already hard at work doing so (see [https://github.com/apache/datafusion/issues/11752](https://github.com/apache/datafusion/issues/11752) if you want to help out). + + +# Faster `take `and` filter` + +After a filter operation such as `WHERE url <> ''` to avoid processing empty urls, DataFusion will often _coalesce_ results to form a new array with only the passing elements. +This coalescing ensures the batches are sufficiently sized to benefit from [vectorized processing](https://www.vldb.org/pvldb/vol11/p2209-kersten.pdf) in subsequent steps. + +The coalescing operation is implemented using the [take](https://docs.rs/arrow/latest/arrow/compute/fn.take.html) and [filter](https://arrow.apache.org/rust/arrow/compute/kernels/filter/fn.filter.html) kernels in arrow-rs. For StringArray, these kernels require copying the string contents to a new buffer without “holes” in between. This copy can be expensive especially when the new array is large. + +However, `take` and `filter` for StringViewArray can avoid the copy by reusing buffers from the old array. The kernels only need to create a new list of `view`s that point at the same strings within the old buffers. +Figure 1 illustrates the difference between the output of both string representations. StringArray creates two new strings at offsets 0-17 and 17-32, while StringViewArray simply points to the original buffer at offsets 0 and 25. + + + + + +Figure 1: Zero-copy `take`/`filter` for StringViewArray + + +# When to GC? + +Zero-copy `take/filter` is great for generating large arrays quickly, but it is suboptimal for highly selective filters, where most of the strings are filtered out. When the cardinality drops, StringViewArray buffers become sparse—only a small subset of the bytes in the buffer’s memory are referred to by any `view`. This leads to excessive memory usage, especially in a [filter-then-coalesce scenario](https://github.com/apache/datafusion/issues/11628). For example, a StringViewArray with 10M strings may only refer to 1M strings after some filter operations; however, due to zero-copy take/filter, the (reused) 10M buffers can not be released/reused. + +To release unused memory, we implemented a [garbage collection (GC)](https://docs.rs/arrow/latest/arrow/array/struct.GenericByteViewArray.html#method.gc) routine to consolidate the data into a new buffer to release the old sparse buffer(s). As the GC operation copies strings, similarly to StringArray, we must be careful about when to call it. If we call GC too early, we cause unnecessary copying, losing much of the benefit of StringViewArray. If we call GC too late, we hold large buffers for too long, increasing memory use and decreasing cache efficiency. The [Polars blog](https://pola.rs/posts/polars-string-type/) on StringView also refers to the challenge presented by garbage collection timing. + +`arrow-rs` implements the GC process, but it is up to users to decide when to call it. We leverage the semantics of the query engine and observed that the [`CoalseceBatchesExec`](https://docs.rs/datafusion/latest/datafusion/physical_plan/coalesce_batches/struct.CoalesceBatchesExec.html) operator, which merge smaller batches to a larger batch, is often used after the record cardinality is expected to shrink, which aligns perfectly with the scenario of GC in StringViewArray. +We, therefore,[ implemented the GC procedure](https://github.com/apache/datafusion/pull/11587) inside CoalseceBatchesExec[^5] with a heuristic that estimates when the buffers are too sparse. + + +## The art of function inlining: not too much, not too little + +Like string inlining, _function_ inlining is the process of embedding a short function into the caller to avoid the overhead of function calls (caller/callee save). +Usually, the Rust compiler does a good job of deciding when to inline. However, it is possible to override its default using the [`#[inline(always)]` directive](https://doc.rust-lang.org/reference/attributes/codegen.html#the-inline-attribute). +In performance-critical code, inlined code allows us to organize large functions into smaller ones without paying the runtime cost of function invocation. + +However, function inlining is **_not_** always better, as it leads to larger function bodies that are harder for LLVM to optimize (for example, suboptimal [register spilling](https://en.wikipedia.org/wiki/Register_allocation)) and risk overflowing the CPU’s instruction cache. We observed several performance regressions where function inlining caused _slower_ performance when implementing the StringViewArray comparison kernels. Careful inspection and tuning of the code was required to aid the compiler in generating efficient code. More details can be found in this PR: [https://github.com/apache/arrow-rs/pull/5900](https://github.com/apache/arrow-rs/pull/5900). + + +## Buffer size tuning + +StringViewArray permits multiple buffers, which enables a flexible buffer layout and potentially reduces the need to copy data. However, a large number of buffers slows down the performance of other operations. +For example, [`get_array_memory_size`](https://docs.rs/arrow/latest/arrow/array/trait.Array.html#tymethod.get_array_memory_size) needs to sum the memory size of each buffer, which takes a long time with thousands of small buffers. +In certain cases, we found that multiple calls to [`concat_batches`](https://docs.rs/arrow/latest/arrow/compute/fn.concat_batches.html) lead to arrays with millions of buffers, which was prohibitively expensive. + +For example, consider a StringViewArray with the previous default buffer size of 8 KB. With this configuration, holding 4GB of string data requires almost half a million buffers! Larger buffer sizes are needed for larger arrays, but we cannot arbitrarily increase the default buffer size, as small arrays would consume too much memory (most arrays require at least one buffer). Buffer sizing is especially problematic in query processing, as we often need to construct small batches of string arrays, and the sizes are unknown at planning time. + +To balance the buffer size trade-off, we again leverage the query processing (DataFusion) semantics to decide when to use larger buffers. While coalescing batches, we combine multiple small string arrays and set a smaller buffer size to keep the total memory consumption low. In string aggregation, we aggregate over an entire Datafusion partition, which can generate a large number of strings, so we set a larger buffer size (2MB). + +To assist situations where the semantics are unknown, we also [implemented](https://github.com/apache/arrow-rs/pull/6136) a classic dynamic exponential buffer size growth strategy, which starts with a small buffer size (8KB) and doubles the size of each new buffer up to 2MB. We implemented this strategy in arrow-rs and enabled it by default so that other users of StringViewArray can also benefit from this optimization. See this issue for more details: [https://github.com/apache/arrow-rs/issues/6094](https://github.com/apache/arrow-rs/issues/6094). + + +## End-to-end query performance + +We have made significant progress in optimizing StringViewArray filtering operations. Now, let’s test it in the real world to see how it works! + +Let’s consider ClickBench query 22, which selects multiple string fields (`URL`, `Title`, and `SearchPhase`) and applies several filters. + +We ran the benchmark using the following command in the DataFusion repo. Again, the `--string-view` option means we use StringViewArray instead of StringArray. + +To eliminate the impact of the faster Parquet reading using StringViewArray (see the first part of this blog), Figure 2 plots only the time spent in `FilterExec`. Without StringViewArray, the filter takes 7.17s; with StringViewArray, the filter only takes 4.86s, a 32% reduction in time. Moreover, we see a 17% improvement in end-to-end query performance. + + + + + + + +Figure 2: StringViewArray reduces the filter time by 32% on ClickBench query 22. + + +# Faster String Aggregation + +So far, we have discussed how to exploit two StringViewArray features: reduced copy and faster filtering. This section focuses on reusing string bytes to repeat string values. + +As described in part one of this blog, if two strings have identical values, StringViewArray can use two different `view`s pointing at the same buffer range, thus avoiding repeating the string bytes in the buffer. This makes StringViewArray similar to an Arrow [DictionaryArray](https://docs.rs/arrow/latest/arrow/array/struct.DictionaryArray.html) that stores Strings—both array types work well for strings with only a few distinct values. + +Deduplicating string values can significantly reduce memory consumption in StringViewArray. However, this process is expensive and involves hashing every string and maintaining a hash table, and so it cannot be done by default when creating a StringViewArray. We introduced an[ opt-in string deduplication mode](https://docs.rs/arrow/latest/arrow/array/builder/struct.GenericByteViewBuilder.html#method.with_deduplicate_strings) in arrow-rs for advanced users who know their data has a small number of distinct values, and where the benefits of reduced memory consumption outweigh the additional overhead of array construction. + +Once again, we leverage DataFusion query semantics to identify StringViewArray with duplicate values, such as aggregation queries with multiple group keys. For example, some [ClickBench queries](https://github.com/apache/datafusion/blob/main/benchmarks/queries/clickbench/queries.sql) group by two columns: + + +* `UserID` (an integer with close to 1 M distinct values) +* `MobilePhoneModel` (a string with less than a hundred distinct values) + +In this case, the output row count is` count(distinct UserID) * count(distinct MobilePhoneModel)`, which is 100M. Each string value of `MobilePhoneModel` is repeated 1M times. With StringViewArray, we can save space by pointing the repeating values to the same underlying buffer. + +Faster string aggregation with StringView is part of a larger project to [improve DataFusion aggregation performance](https://github.com/apache/datafusion/issues/7000). We have a [proof of concept implementation](https://github.com/apache/datafusion/pull/11794) with StringView that can improve the multi-column string aggregation by 20%. We would love your help to get it production ready! + + +# StringView Pitfalls + +Most existing blog posts (including this one) focus on the benefits of using StringViewArray over other string representations such as StringArray. As we have discussed, even though it requires a significant engineering investment to realize, StringViewArray is a major improvement over StringArray in many cases. + +However, there are several cases where StringViewArray is slower than StringArray. For completeness, we have listed those instances here: + + + +1. **Tiny strings (when strings are shorter than 8 bytes)**: every element of the StringViewArray consumes at least 16 bytes of memory—the size of the `view` struct. For an array of tiny strings, StringViewArray consumes more memory than StringArray and thus can cause slower performance due to additional memory pressure on the CPU cache. +2. **Many repeated short strings**: Similar to the first point, StringViewArray can be slower and require more memory than a DictionaryArray because 1) it can only reuse the bytes in the buffer when the strings are longer than 12 bytes and 2) 32-bit offsets are always used, even when a smaller size (8 bit or 16 bit) could represent all the distinct values. +3. **Filtering:** As we mentioned above, StringViewArrays often consume more memory than the corresponding StringArray, and memory bloat quickly dominates the performance without GC. However, invoking GC also reduces the benefits of less copying so must be carefully tuned. + + +# Conclusion and Takeaways + +In these two blog posts, we discussed what it takes to implement StringViewArray in arrow-rs and then integrate it into DataFusion. Our evaluations on ClickBench queries show that StringView can improve the performance of string-intensive workloads by up to 2x. + +Given that DataFusion already [performs very well on ClickBench](https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQiI6ZmFsc2UsIkF0aGVuYSAocGFydGl0aW9uZWQpIjpmYWxzZSwiQXRoZW5hIChzaW5nbGUpIjpmYWxzZSwiQXVyb3JhIGZvciBNeVNRTCI6ZmFsc2UsIkF1cm9yYSBmb3IgUG9zdGdyZVNRTCI6ZmFsc2UsIkJ5Q29uaXR5IjpmYWxzZSwiQnl0ZUhvdXNlIjpmYWxzZSwiY2hEQiAoUGFycXVldCwgcGFydGl0aW9uZWQpIjpmYWxzZSwiY2hEQiI6ZmFsc2UsIkNpdHVzIjpmYWxzZSwiQ2xpY2tIb3VzZSBDbG91ZCAoYXdzKSI6ZmFsc2UsIkNsaWNrSG91c2UgQ2xvdWQgKGF3cykgUGFyYWxsZWwgUmVwbGljYXMgT04iOmZhbHNlLCJDbGlja0hvdXNlIENsb3VkIChBenVyZSkiOmZhbHNlLCJDbGlja0hvdXNlIENsb3VkIChBenVyZSkgUGFyYWxsZWwgUmVwbGljYSBPTiI6ZmFsc2UsIkNsaWNrSG91c2UgQ2xvdWQgKEF6dXJlKSBQYXJhbGxlbCBSZXBsaWNhcyBPTiI6ZmFsc2UsIkNsaWNrSG91c2UgQ2xvdWQgKGdjcCkiOmZhbHNlLCJDbGlja0hvdXNlIENsb3VkIChnY3ApIFBhcmFsbGVsIFJlcGxpY2FzIE9OIjpmYWxzZSwiQ2xpY2tIb3VzZSAoZGF0YSBsYWtlLCBwYXJ0aXRpb25lZCkiOmZhbHNlLCJDbGlja0hvdXNlIChkYXRhIGxha2UsIHNpbmdsZSkiOmZhbHNlLCJDbGlja0hvdXNlIChQYXJxdWV0LCBwYXJ0aXRpb25lZCkiOmZhbHNlLCJDbGlja0hvdXNlIChQYXJxdWV0LCBzaW5nbGUpIjpmYWxzZSwiQ2xpY2tIb3VzZSAod2ViKSI6ZmFsc2UsIkNsaWNrSG91c2UiOmZhbHNlLCJDbGlja0hvdXNlICh0dW5lZCkiOmZhbHNlLCJDbGlja0hvdXNlICh0dW5lZCwgbWVtb3J5KSI6ZmFsc2UsIkNsb3VkYmVycnkiOmZhbHNlLCJDcmF0ZURCIjpmYWxzZSwiQ3J1bmNoeSBCcmlkZ2UgZm9yIEFuYWx5dGljcyAoUGFycXVldCkiOmZhbHNlLCJEYXRhYmVuZCI6ZmFsc2UsIkRhdGFGdXNpb24gKFBhcnF1ZXQsIHBhcnRpdGlvbmVkKSI6dHJ1ZSwiRGF0YUZ1c2lvbiAoUGFycXVldCwgc2luZ2xlKSI6ZmFsc2UsIkFwYWNoZSBEb3JpcyI6ZmFsc2UsIkRydWlkIjpmYWxzZSwiRHVja0RCIChQYXJxdWV0LCBwYXJ0aXRpb25lZCkiOnRydWUsIkR1Y2tEQiI6ZmFsc2UsIkVsYXN0aWNzZWFyY2giOmZhbHNlLCJFbGFzdGljc2VhcmNoICh0dW5lZCkiOmZhbHNlLCJHbGFyZURCIjpmYWxzZSwiR3JlZW5wbHVtIjpmYWxzZSwiSGVhdnlBSSI6ZmFsc2UsIkh5ZHJhIjpmYWxzZSwiSW5mb2JyaWdodCI6ZmFsc2UsIktpbmV0aWNhIjpmYWxzZSwiTWFyaWFEQiBDb2x1bW5TdG9yZSI6ZmFsc2UsIk1hcmlhREIiOmZhbHNlLCJNb25ldERCIjpmYWxzZSwiTW9uZ29EQiI6ZmFsc2UsIk1vdGhlcmR1Y2siOmZhbHNlLCJNeVNRTCAoTXlJU0FNKSI6ZmFsc2UsIk15U1FMIjpmYWxzZSwiT3hsYSI6ZmFsc2UsIlBhcmFkZURCIChQYXJxdWV0LCBwYXJ0aXRpb25lZCkiOmZhbHNlLCJQYXJhZGVEQiAoUGFycXVldCwgc2luZ2xlKSI6ZmFsc2UsIlBpbm90IjpmYWxzZSwiUG9zdGdyZVNRTCAodHVuZWQpIjpmYWxzZSwiUG9zdGdyZVNRTCI6ZmFsc2UsIlF1ZXN0REIgKHBhcnRpdGlvbmVkKSI6ZmFsc2UsIlF1ZXN0REIiOmZhbHNlLCJSZWRzaGlmdCI6ZmFsc2UsIlNlbGVjdERCIjpmYWxzZSwiU2luZ2xlU3RvcmUiOmZhbHNlLCJTbm93Zmxha2UiOmZhbHNlLCJTUUxpdGUiOmZhbHNlLCJTdGFyUm9ja3MiOmZhbHNlLCJUYWJsZXNwYWNlIjpmYWxzZSwiVGVtYm8gT0xBUCAoY29sdW1uYXIpIjpmYWxzZSwiVGltZXNjYWxlREIgKGNvbXByZXNzaW9uKSI6ZmFsc2UsIlRpbWVzY2FsZURCIjpmYWxzZSwiVW1icmEiOmZhbHNlfSwidHlwZSI6eyJDIjp0cnVlLCJjb2x1bW4tb3JpZW50ZWQiOnRydWUsIlBvc3RncmVTUUwgY29tcGF0aWJsZSI6dHJ1ZSwibWFuYWdlZCI6dHJ1ZSwiZ2NwIjp0cnVlLCJzdGF0ZWxlc3MiOnRydWUsIkphdmEiOnRydWUsIkMrKyI6dHJ1ZSwiTXlTUUwgY29tcGF0aWJsZSI6dHJ1ZSwicm93LW9yaWVudGVkIjp0cnVlLCJDbGlja0hvdXNlIGRlcml2YXRpdmUiOnRydWUsImVtYmVkZGVkIjp0cnVlLCJzZXJ2ZXJsZXNzIjp0cnVlLCJhd3MiOnRydWUsInBhcmFsbGVsIHJlcGxpY2FzIjp0cnVlLCJBenVyZSI6dHJ1ZSwiYW5hbHl0aWNhbCI6dHJ1ZSwiUnVzdCI6dHJ1ZSwic2VhcmNoIjp0cnVlLCJkb2N1bWVudCI6dHJ1ZSwic29tZXdoYXQgUG9zdGdyZVNRTCBjb21wYXRpYmxlIjp0cnVlLCJ0aW1lLXNlcmllcyI6dHJ1ZX0sIm1hY2hpbmUiOnsiMTYgdkNQVSAxMjhHQiI6dHJ1ZSwiOCB2Q1BVIDY0R0IiOnRydWUsInNlcnZlcmxlc3MiOnRydWUsIjE2YWN1Ijp0cnVlLCJjNmEuNHhsYXJnZSwgNTAwZ2IgZ3AyIjp0cnVlLCJMIjp0cnVlLCJNIjp0cnVlLCJTIjp0cnVlLCJYUyI6dHJ1ZSwiYzZhLm1ldGFsLCA1MDBnYiBncDIiOnRydWUsIjE5MkdCIjp0cnVlLCIyNEdCIjp0cnVlLCIzNjBHQiI6dHJ1ZSwiNDhHQiI6dHJ1ZSwiNzIwR0IiOnRydWUsIjk2R0IiOnRydWUsIjE0MzBHQiI6dHJ1ZSwiZGV2Ijp0cnVlLCI3MDhHQiI6dHJ1ZSwiYzVuLjR4bGFyZ2UsIDUwMGdiIGdwMiI6dHJ1ZSwiQW5hbHl0aWNzLTI1NkdCICg2NCB2Q29yZXMsIDI1NiBHQikiOnRydWUsImM1LjR4bGFyZ2UsIDUwMGdiIGdwMiI6dHJ1ZSwiYzZhLjR4bGFyZ2UsIDE1MDBnYiBncDIiOnRydWUsImNsb3VkIjp0cnVlLCJkYzIuOHhsYXJnZSI6dHJ1ZSwicmEzLjE2eGxhcmdlIjp0cnVlLCJyYTMuNHhsYXJnZSI6dHJ1ZSwicmEzLnhscGx1cyI6dHJ1ZSwiUzIiOnRydWUsIlMyNCI6dHJ1ZSwiMlhMIjp0cnVlLCIzWEwiOnRydWUsIjRYTCI6dHJ1ZSwiWEwiOnRydWUsIkwxIC0gMTZDUFUgMzJHQiI6dHJ1ZSwiYzZhLjR4bGFyZ2UsIDUwMGdiIGdwMyI6dHJ1ZX0sImNsdXN0ZXJfc2l6ZSI6eyIxIjp0cnVlLCIyIjp0cnVlLCI0Ijp0cnVlLCI4Ijp0cnVlLCIxNiI6dHJ1ZSwiMzIiOnRydWUsIjY0Ijp0cnVlLCIxMjgiOnRydWUsInNlcnZlcmxlc3MiOnRydWUsImRlZGljYXRlZCI6dHJ1ZX0sIm1ldHJpYyI6ImhvdCIsInF1ZXJpZXMiOlt0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLH), the level of end-to-end performance improvement using StringViewArray shows the power of this technique and, of course, is a win for DataFusion and the systems that build upon it. + +StringView is a big project that has received tremendous community support. Specifically, we would like to thank [@tustvold](https://github.com/tustvold), [@ariesdevil](https://github.com/ariesdevil), [@RinChanNOWWW](https://github.com/RinChanNOWWW), [@ClSlaid](https://github.com/ClSlaid), [@2010YOUY01](https://github.com/2010YOUY01), [@chloro-pn](https://github.com/chloro-pn), [@a10y](https://github.com/a10y), [@Kev1n8](https://github.com/Kev1n8), [@Weijun-H](https://github.com/Weijun-H), [@PsiACE](https://github.com/PsiACE), [@tshauck](https://github.com/tshauck), and [@xinlifoobar](https://github.com/xinlifoobar) for their valuable contributions! + +As the introduction states, “German Style Strings” is a relatively straightforward research idea that avoid some string copies and accelerates comparisons. However, applying this (great) idea in practice requires a significant investment in careful software engineering. Again, we encourage the research community to continue to help apply research ideas to industrial systems, such as DataFusion, as doing so provides valuable perspectives when evaluating future research questions for the greatest potential impact. + +### Footnotes + +[^5]: There are additional optimizations possible in this operation that the community is working on, such as [https://github.com/apache/datafusion/issues/7957](https://github.com/apache/datafusion/issues/7957). diff --git a/img/string-view-1/figure1-performance.png b/img/string-view-1/figure1-performance.png new file mode 100644 index 0000000..628f9aa Binary files /dev/null and b/img/string-view-1/figure1-performance.png differ diff --git a/img/string-view-1/figure2-string-view.png b/img/string-view-1/figure2-string-view.png new file mode 100644 index 0000000..9a2cd63 Binary files /dev/null and b/img/string-view-1/figure2-string-view.png differ diff --git a/img/string-view-1/figure4-copying.png b/img/string-view-1/figure4-copying.png new file mode 100644 index 0000000..cf94219 Binary files /dev/null and b/img/string-view-1/figure4-copying.png differ diff --git a/img/string-view-1/figure5-loading-strings.png b/img/string-view-1/figure5-loading-strings.png new file mode 100644 index 0000000..d287efa Binary files /dev/null and b/img/string-view-1/figure5-loading-strings.png differ diff --git a/img/string-view-1/figure6-utf8-validation.png b/img/string-view-1/figure6-utf8-validation.png new file mode 100644 index 0000000..98185ee Binary files /dev/null and b/img/string-view-1/figure6-utf8-validation.png differ diff --git a/img/string-view-1/figure7-end-to-end.png b/img/string-view-1/figure7-end-to-end.png new file mode 100644 index 0000000..bb5ff40 Binary files /dev/null and b/img/string-view-1/figure7-end-to-end.png differ diff --git a/img/string-view-2/figure1-zero-copy-take.png b/img/string-view-2/figure1-zero-copy-take.png new file mode 100644 index 0000000..44363f9 Binary files /dev/null and b/img/string-view-2/figure1-zero-copy-take.png differ diff --git a/img/string-view-2/figure2-filter-time.png b/img/string-view-2/figure2-filter-time.png new file mode 100644 index 0000000..893fa09 Binary files /dev/null and b/img/string-view-2/figure2-filter-time.png differ