PyramidOMETiffWriter close() duration scales as a (number of planes)^2
#4204
Milestone
Comments
|
This issue has been mentioned on Image.sc Forum. There might be relevant details there: |
|
With a test based on https://github.com/ome/bioformats/blob/develop/components/formats-gpl/utils/MinimumPyramidWriter.java that writes one pyramid resolution: https://gist.github.com/melissalinkert/aaa9ac52c42a6b9c7596037cbadde4a3 I can see similar degradation of I also see that data writing does not exactly scale linearly either, but obviously not as badly as Incidentally, with a similar test that writes multiple resolutions: https://gist.github.com/melissalinkert/ef601854820ee1a2ce970f09559a0cb0 I could easily reproduce #3907 using: All four related issues (this, #3983, #3480, and now #3907) are now scheduled for 8.0.0, as discussed with @joshmoore and @sbesson. |
|
With the patch of @melissalinkert in this issue vanishes:
in log scale:
Raw perf data:
Code used (ijp-kheops lib): /*-
* #%L
* %%
* %%
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as
* published by the Free Software Foundation, either version 2 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public
* License along with this program. If not, see
* <http://www.gnu.org/licenses/gpl-2.0.html>.
* #L%
*/
import ch.epfl.biop.kheops.ometiff.OMETiffExporter;
import loci.common.DebugTools;
import net.imagej.ImageJ;
import net.imglib2.FinalInterval;
import net.imglib2.RandomAccessibleInterval;
import net.imglib2.position.FunctionRandomAccessible;
import net.imglib2.type.numeric.ARGBType;
import net.imglib2.view.Views;
import org.scijava.task.TaskService;
import java.time.Duration;
import java.time.Instant;
public class DemoExport {
public static void main( String[] args )
{
ImageJ ij = new ImageJ();
DebugTools.setRootLevel("Off");
ij.ui().showUI();
int sizeInPixelX = 512;
int sizeInPixelY = 512;
int tileSize = 512;
//int nT = 512;
for (int nT = 1; nT<=1024; nT*=2) {
final FunctionRandomAccessible<ARGBType> checkerboard = new FunctionRandomAccessible<>(
2,
(location, value) -> {
value.set(
ARGBType.rgba(location.getIntPosition(0), location.getIntPosition(1), location.getIntPosition(0) * 0, 255)
);
},
ARGBType::new);
RandomAccessibleInterval<ARGBType> img = Views.interval(checkerboard, new FinalInterval(new long[]{0, 0}, new long[]{sizeInPixelX - 1, sizeInPixelY - 1}));
DebugTools.setRootLevel("OFF");
try {
//String path = "C:\\Users\\nicol\\Desktop\\ometiff\\ntest-" + sizeInPixelX + "x"+sizeInPixelY+"px-" + tileSize + "tile.ome.tiff";
String path = "C:\\kheops\\test_x-" + sizeInPixelX + "_y-" + sizeInPixelY + "_tile-" + tileSize + "_nT-" + nT + ".ome.tiff";
System.out.println("Saving " + path);
Instant start = Instant.now();
OMETiffExporter.OMETiffExporterBuilder.Data.DataBuilder dataBuilder = OMETiffExporter.builder();
for (int t = 0; t < nT; t++) {
dataBuilder.putXYZRAI(0, t, img);
}
dataBuilder.defineMetaData("Image")
.imageName("Bob")
.voxelPhysicalSizeMicrometer(10, 10, 2)
.pixelsTimeIncrementInS(0.35)
.defineWriteOptions()
.tileSize(tileSize, tileSize)
.monitor(ij.get(TaskService.class))
.savePath(path)
.create()
.export();
Instant end = Instant.now();
System.out.println("Export time (ms) \t" + Duration.between(start, end).toMillis());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("done");
}
}
}(report made with the help of chatgpt) |
|
Thank you @NicoKiaru - this report is very helpful! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello,
Original post: https://forum.image.sc/t/pyramidometiffwriter-really-bad-performance-when-closing-the-writer/95294
There is a (very big) performance issue when saving an OME TIFF file with
PyramidOMETiffWriteras theclose()method duration scales as the number of planes squared.I've tested to save a 512x512 pixel file on my computer and increase the number of timepoints by powers of 2 (nT). I've measured the time required to save the file and split the required time between what happens before calling close, and the close duration:
The data writing time scales linearly with the number of timepoints, which is what we expect. However the close time is multiplied by 4 each time the number of timepoints is multiplied by 2.
As a result, 'close' takes 90% of the total export time when there are 1024 timepoints.
I've narrowed the issue to the call of
TiffParser#getIFDOffsetsfor all planes,getIFDOffsetswalking the whole file from the start. This scales very badly.PyramidOMETiffWriter#closeloops through all planes:bioformats/components/formats-bsd/src/loci/formats/out/PyramidOMETiffWriter.java
Line 125 in 7ca9980
TiffSaver#overwriteIFDValuebioformats/components/formats-bsd/src/loci/formats/out/PyramidOMETiffWriter.java
Line 145 in 7ca9980
TiffParser#getIFDOffsetsbioformats/components/formats-bsd/src/loci/formats/tiff/TiffSaver.java
Line 804 in 7ca9980
TiffParser#getIFDOffsetswalks through the whole filebioformats/components/formats-bsd/src/loci/formats/tiff/TiffParser.java
Lines 362 to 363 in 7ca9980
Is there a way to fix this ?
This issue could be related to these other issues:
#3983
#3480
The text was updated successfully, but these errors were encountered: