api/rest: files limit per bucket

[diegodelemos]

We have been having problems with the number of files Files-Rest can handle in CERN Opendata.

We have put in place a workaround in the past and the investigation we went through can be accessed here.

As a summary:

• This feature in Opendata adds files from the CLI using the Files-Rest Pythonic API
• Profiling shows big amount of SQL queries for the given files to be added (factor of 10 times the number of files)
• Memory usage grows polynomially and it doesn't get cleaned

We should find out what are the limits of a bucket in Invenio, so we can add it to the documentation, and eventually solve the memory problem. To do so we can replicate the conditions on Opendata:

• Create through Pythonic API ~4000 files
• Create through REST API ~4000 files
• Try to list those files and operate them

[tiborsimko]

Example: one dataset record consisting of 27K files: http://opendata.cern.ch/record/8884