Patch "FolderModel.php - addItem" function to not use "fetchAll"

[jcfr]

Copied from http://public.kitware.com/MidasBT/view.php?id=1057

In folder with more than 16000 items, it is not possible to add new items.

Solution

• Today : Increase memory_limit set in php.ini
• In few days : Patch "FolderModel.php - addItem" function to not use "fetchAll"
• In coming weeks : Review usage of "fetchAll" in the entire code base ?

References

• http://is2.php.net/memory_get_usage
• http://stackoverflow.com/questions/12339512/replace-fetchall-by-fetch-to-avoid-memory-limit

Details

Debug statement can be re-enabled on the server http://slicer.kitwarein.com by doing:

$ sudo su - www-data
$ bash
$ cd /var/www/midas3/
$ git stash pop
$ cd /var/www/midas3/slicerpackages
$ git stash pop

... or applying the enclosed patch on your local installation. See midas-debug-stmt.patch and slicerpackages-debug-stmt.patch. See midas-debug-stmt.patch.txt and slicerpackages-debug-stmt.patch.txt

Then, when creating a new item, the execution unrolls as described below:

On my workstation:

Checkout Slicer Source, then use the script published in this gist: https://gist.github.com/jcfr/9380371

Slicer_SOURCE_DIR=/home/jcfr/Projects/Slicer

cmake -DSlicer_SOURCE_DIR:PATH=${Slicer_SOURCE_DIR} \
  -DPACKAGE_FILEPATH:FILEPATH=${Slicer_SOURCE_DIR}/Resources/3DSlicer-DesktopIcon.png \
  -P 3613-test-slicer-midas-upload.cmake

On slicer.kitwarein.com:

 # Extract from log file: /var/www/midas3/log/development.log
 #
 #
 # memory: Size of a "item" dao
 # cumul_memory: Cumulated size of all item daos instantiated in this loop
 # total_memory: Total memory used by this script

[31568] ApiComponent.php - extensionUpload - line:223
[31568] ApiComponent.php - extensionUpload - line:226
[31568] UploadComponent.php - createUploadedItem - line:245
[31568] FolderModel.php - addItem - line:913
[31568] MIDASDatabasePdo.php - link - line:219
[31568] MIDASDatabasePdo.php - getValue - line:137
[31568] MIDASDatabasePdo.php - getLinkedObject - line:165
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:0][memory:2.76 kb][cumul_memory:5.64 kb][total_size:125.05 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:1][memory:2.68 kb][cumul_memory:13.65 kb][total_size:125.06 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:2][memory:2.68 kb][cumul_memory:21.39 kb][total_size:125.06 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:3][memory:2.68 kb][cumul_memory:29.13 kb][total_size:125.07 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:4][memory:2.68 kb][cumul_memory:36.86 kb][total_size:125.08 mb]
[...]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:16677][memory:2.68 kb][cumul_memory:130.7 mb][total_size:255.74 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:16678][memory:2.68 kb][cumul_memory:130.7 mb][total_size:255.75 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:16679][memory:2.68 kb][cumul_memory:130.71 mb][total_size:255.75 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:16680][memory:2.68 kb][cumul_memory:130.72 mb][total_size:255.76 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:16681][memory:2.68 kb][cumul_memory:130.73 mb][total_size:255.77 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:16682][memory:2.68 kb][cumul_memory:130.73 mb][total_size:255.78 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:16683][memory:2.68 kb][cumul_memory:130.74 mb][total_size:255.78 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:16684][memory:2.68 kb][cumul_memory:130.75 mb][total_size:255.79 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:16685][memory:2.68 kb][cumul_memory:130.76 mb][total_size:255.8 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:16686][memory:2.68 kb][cumul_memory:130.76 mb][total_size:255.81 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:16687][memory:2.68 kb][cumul_memory:130.77 mb][total_size:255.81 mb]
[31568] MIDASDatabasePdo.php - getLinkedObject - line:204 [i:16688][memory:2.68 kb][cumul_memory:130.78 mb][total_size:255.82 mb]

Considering that the memory limit set on the server is 256MB, this explain why the upload failed.

www-data@slicer:/data/www/midas3/log$ cat /etc/php5/apache2/php.ini | grep ^memory_limit
memory_limit = 256M

Associated versions

midas version: f03ce8b2e1e21878f3c7d5552a2bb81f46b27b71

slicerpackages version: b16033698bfceb1002c918a1872b67a9a833f211

slicerappstore version: 6cc5599ae8502e555f7acdd471bfbb50d30d3e25 

[jamiesnape]

I am afraid the problems Midas has with large numbers of items in a folder are deeper than a fetchAll command. Also, using fetch with a loop would be extremely slow, so it does not solve the problem.