Do you have any tips for handling large parquet files?

[aborruso]

Hi,
I have a 1.5 GB parquet file. Almost 20 million rows and 100 columns.

I add it as local file. After 15 minutes of processing, it is still not finished and very large temporary files are created.

And so then I'm going to stop everything, because I think he's gone wrong. Can I pre work something?
Do you have any tips for using rill with files of this size?

Thank you

[begelundmuller]

Hey @aborruso, Rill uses DuckDB under the hood to process data, and while a 1.5GB Parquet file is not usually a problem for DuckDB, it seems there's something about this particular Parquet file that causes it to generate a large amount of temporary data during ingestion.

One possible workaround is to disable insertion order preservation. Can you try starting Rill with these flags and see if it solves the issue:

rill start --reset --var connector.duckdb.boot_queries='SET preserve_insertion_order TO false'

Also, can you tell me what version of Rill you're running (rill --version) and how much RAM your machine has available?

[AndrewRTsao]

If you're also planning to deploy to Rill Cloud, you may also want to consider uploading the parquet file to an object storage instead (such as S3 or GCS). Besides what Benjamin mentioned (where we use DuckDB under the hood and these tmp files are related to the ingestion), you will notice that local files are added to your <RILL_HOME>/data directory, which are then uploaded to Github if you end up deploying the project to Rill Cloud. By default, Github does not allow file sizes greater than 100MB to be uploaded, so that will be another potential blocking point down the road even if you are able to read this file locally.

At a high level, when it comes to working with larger files, you may find that using object storage (if a possibility) will generally be more efficient from a storage standpoint and more performant / scalable. This also aligns with general best practices and what we would recommend, especially for production and/or if you wish to share dashboards with others (using Rill Cloud).

[aborruso]

Hi @begelundmuller and @AndrewRTsao first of all thank you very much for your replies. And thank you also for Rill, it seems to me great!

I have tried to use the @begelundmuller way: I don't have those huge temporary files anymore, but it doesn't display anything and it crashes after a while.
I'm using rill 0.41.1 (build commit: 9151570 date: 2024-02-26T13:50:19Z)

The file I'm using is here https://huggingface.co/datasets/aborruso/opencup/blob/refs%2Fconvert%2Fparquet/default/train/0000.parquet

Thank you

[AndrewRTsao]

Hey @aborruso I gave it a try and it's working for me (tested using v0.41.1 as well). You can see the screenshot below.

With that being said, I am using a fairly well resourced MacBook Pro 2023 and the issue you are facing could be related to resource constraints on your local machine. When you are reading in a local file as a source in Rill, it is first copied to the <RILL_HOME>/data folder, read in by DuckDB and stored as an external table under <RILL_HOME>/tmp/<name_of_source>, and then ingested as a source in Rill. If you want, you could try:

• Starting Rill in debug mode using rill start --debug, repeat the above actions, and then send over the ~/.rill/rill.log file
• Send over a screenshot of your tmp directory, including whether there is a subdirectory that matches the name of your source and whether there's a xxx.db file present in that subdirectory

However, if Rill is not displaying anything and then crashing, then I do suspect this could also be resource related as the same file works for me locally. As an alternative, you could see if hosting the parquet file in some kind of object storage makes a difference.

[aborruso]

Thank you @AndrewRTsao