How can I use visidata on even larger files? 5GB is the max I have found to be usable...

[jwnacnud]

...and that takes a while to let it load. If I just need to come to terms with visidata not handling larger files, I can. It's so good already. I would love to use it to analyze even larger files.

[saulpw]

To be honest I've found ~1GB to be the max that I feel comfortable advertising vd for; so the fact that you're willing to use it for 5GB files is already pretty ambitious!

If we think about datasets in terms of order of magnitude of size, there are roughly 4 bins:

1. <1GB: can fit in memory
2. 1GB-1TB: can fit on local disk
3. 1TB-1PB: can fit in on-prem database
4. 1PB+: needs datacenter

The challenge is that 1GB+ of data is a lot more effort to work with, but companies only start paying for data tooling above 1TB of data (and the real money is in the 1PB+ category of data). As you climb this ladder, performance and integration and reproducibility matters more, and interface and experience matters less--PB of data cost so much to store and process that it's cheaper to pay an engineer to suffer in order to save a few % of processing cycles.

So, VisiData is designed for data that can fit in memory. DuckDB is an up-and-coming contender for the second category, as it can operate efficiently on .csv and other local files without need for import. There is vdsql which can be used with any database (including DuckDB), but it has more limited functionality, and the experience isn't the same, as it's a large amount of work and no one is willing to pay for it.