stringId
values are not checked when a DwC export is constructed, resulting in server halting queries
#5321
Labels
1 - Bug
Incorrect behavior of the product
2 - Exporting Data
Issues that are related to exporting data to DwC, GBIF, IPT, Web Portal, etc.
Milestone
Background
SAIAB created a DwCA definition and metadata pair for their National Fish Collection.
This is their mapping:
original_FC_export.xml.zip
Upon running this export (or using the RSS feed), the container would run out of memory and storage space, eventually erroring out. This didn't make a lot of sense, but I was able to recreate this despite the export ultimately being less than 14.2 MB compressed and <150 MB uncompressed. They even increased the amount of RAM accessible to the server to 32 GB.
After examining the SQL queries being run I found that the definition SAIAB provided had an invalid
stringId
defined:This should be
Without this
1,
in the XML, Specify constructs the query improperly and attempts to pull data from Accession without using the join from Collection Object.When SAIAB reported this, their export mapping was creating an export with 61,336,980 rows. Each row in the export was duplicated by approximately the number of Accession records in the database, and this caused the export to endlessly grow far beyond what should have been included. We could have kept feeding it more and more memory, but just those two characters alone broke the export.
To Reproduce
Steps to reproduce the behavior:
docker stats
:This will eat all available storage space and RAM as well as hog the CPU.
Expected behavior
It should say "Hey, that
stringId
is invalid!"Ultimately, once validation is added, #285 should be implemented.
Crash Report
The container simply exits when it runs out of space in some cases. When it crashes and stays alive, it says this:
OSError: [Errno 28] No space left on device
Reported By
Willem and Wesley at SAIAB
Additional context
queries_and_logs.zip
The text was updated successfully, but these errors were encountered: