Daily limit for import from csv?

[bobromeo]

CE 24.1.4.202408041450 Rel:5aug24
SQLite 3
I have csv file of 205k wirh 1.5m records.
I have a .db-file of 412k on a win-NTFS share, for trying a lot of imports, but it is now mostly empty.
After a lot of tries, with commits at 10k I succeeded a succesful import of 1,140.000 records (I chopped the csv in pieces)
For a second csv file in a separate table I tried importing 140k records, but after a lot of tries (after every time closing dBeaver) it would not work, freezing the import window.
After setting the commit to 100, i got an extra 2,200 records in a second table ?!?
So is there a daily linit for importing from a csv for around 1,142,200? :-)

[bobromeo]

It is not a daily limit (of course it is not :)
Repeating the second load in the second table, again gives only 2,200 records :-(

[E1izabeth]

We haven't heard about such issue earlier.
We would kindly ask you to reproduce the issue once again. After you reproduce the issue, please send us the DBeaver log files.
The log files are generated automatically and called .log and .dbeaver-debug.log exactly. They are located in the %APPDATA%\DBeaverData\workspace6\.metadata directory. In case your OS isn't Windows, here is our wiki page on this matter: dbeaver.com/docs/wiki/Log-files/#log-files

[bobromeo]

Hi Elizabeth, I did a fresh load, split ti 1,5m records file in 2: 1m records + 500k records.
Renamed the 2nd file to a totally different name with no trailing numbers .. (found out in the past that dB does sometimes removing trailing numbers at import.)
De 1st file is totally okay in 28 seconds, the 2nd file gets stuck at 150k records. Used Commit: 10,000 + Batch: 1000
.log
dbeaver-debug.log

[ShadelessFox]

Thanks. Unfortunately, logs don't contain any helpful information. Can you please try reproducing the issue once again but instead perform a thread dump? You can find instructions here: https://github.com/dbeaver/dbeaver/wiki/Making-a-thread-dump

[bobromeo]

dbeaver-thread-dump.txt
I Used the latest version 24.2.0
Sorry for being late :-(

[ShadelessFox]

I suspect there's something wrong with the input CSV file. Can you try importing different parts of the file (first N rows, last N rows, using binary search) to find the line causing DBeaver to freeze?