-
Clickhouse has many data types and even column-level codecs to improve performance and decrease storage size. What support exists for using those in ibis? In particular, I'm interested in LowCardinality(String) and applying compression codecs. I did look through docs quickly and didn't see anything obvious. When I glanced at code, there is a test, but I didn't see how I could convert that back to usage (my limits of ibis code understanding). Any suggestions? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
Hey @seandavi! We currently don't have a way to use clickhouse's optimized types at the moment nor compression codecs or any of the other per-column attributes, other than to manually construct a For codecs, I think we could probably add a ClickHouse-specific con.create_table(..., codecs={"x": "LZ4", "y": ("Delta", "ZSTD")}) Regarding types:
We test that I'm not entirely sure how we might be able to support this use case for ClickHouse without doing something more invasive like adding a new type. Perhaps a
|
Beta Was this translation helpful? Give feedback.
To use clickhouse "well" really does require getting one's hands dirty with their SQL dialect and database-specific features. It's probably better for Ibis to spend time on more general functionality.
If someone asked about how to work with Clickhouse and Ibis together, what I would recommend at this point is to use Ibis as usual, create the table "as is" using Ibis (without clickhouse-specific features), and then use SQL (CTAS) to generate a fine-tuned table that has all the clickhouse bells-and-whistles.