Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ops.image().watershed failed on large image #274

Open
bpavie opened this issue Nov 27, 2019 · 2 comments
Open

ops.image().watershed failed on large image #274

bpavie opened this issue Nov 27, 2019 · 2 comments

Comments

@bpavie
Copy link

bpavie commented Nov 27, 2019 •
edited
Loading

I ran the following macro

// @OpService ops
// @Dataset inputData

import net.imagej.ops.Ops
import net.imglib2.img.display.imagej.ImageJFunctions
import net.imglib2.type.numeric.integer.UnsignedByteType 

//Thresold the Output mask from fiji (where 0 is false and 255 is true)
maskBitType = ops.threshold().apply(inputData, new UnsignedByteType(128));
//Fill the holes
maskFilled = ops.morphology().fillHoles(maskBitType);

//Perform the watershed
useEightConnectivity=true
drawWatersheds=false
double sigma=2.0
watershedImgLabeling=ops.image().watershed(null,maskFilled,useEightConnectivity,drawWatersheds,sigma,maskFilled)

//Display the result
watershedImg=watershedImgLabeling.getIndexImg()
watershedImgLabelingImp=ImageJFunctions.wrap(watershedImg, "wrapped")
watershedImgLabelingImp.show()

which work on small dataset
image_test_small.zip
but not if the image is large (image_test.zip.

The output error for the large image is the following:

Started watershed.groovy at Wed Nov 27 14:00:32 CET 2019
java.lang.IllegalArgumentException: No matching 'net.imagej.ops.Ops$Image$DistanceTransform' op

Request:
-	net.imagej.ops.Ops$Image$DistanceTransform(
		DefaultDataset)

Candidates:
1. 	(RandomAccessibleInterval out?) =
	net.imagej.ops.image.distancetransform.DistanceTransform2D(
		RandomAccessibleInterval out?,
		RandomAccessibleInterval in)
	Inputs do not conform to op rules
		out = null
		in = test_image.tif
2. 	(RandomAccessibleInterval out?) =
	net.imagej.ops.image.distancetransform.DistanceTransform2DCalibration(
		RandomAccessibleInterval out?,
		RandomAccessibleInterval in,
		double[] calibration)
	Not enough arguments: 1 < 2
3. 	(RandomAccessibleInterval out?) =
	net.imagej.ops.image.distancetransform.DistanceTransform3D(
		RandomAccessibleInterval out?,
		RandomAccessibleInterval in)
	Inputs do not conform to op rules
		out = null
		in = test_image.tif
4. 	(RandomAccessibleInterval out?) =
	net.imagej.ops.image.distancetransform.DistanceTransform3DCalibration(
		RandomAccessibleInterval out?,
		RandomAccessibleInterval in,
		double[] calibration)
	Not enough arguments: 1 < 2
5. 	(RandomAccessibleInterval out?) =
	net.imagej.ops.image.distancetransform.DefaultDistanceTransform(
		RandomAccessibleInterval out?,
		RandomAccessibleInterval in)
	Inputs do not conform to op rules
		out = null
		in = test_image.tif
6. 	(RandomAccessibleInterval out?) =
	net.imagej.ops.image.distancetransform.DefaultDistanceTransformCalibration(
		RandomAccessibleInterval out?,
		RandomAccessibleInterval in,
		double[] calibration)
	Not enough arguments: 1 < 2

	at net.imagej.ops.DefaultOpMatchingService.singleMatch(DefaultOpMatchingService.java:432)
	at net.imagej.ops.DefaultOpMatchingService.findMatch(DefaultOpMatchingService.java:97)
	at net.imagej.ops.DefaultOpMatchingService.findMatch(DefaultOpMatchingService.java:83)
	at net.imagej.ops.OpEnvironment.module(OpEnvironment.java:269)
	at net.imagej.ops.OpEnvironment.run(OpEnvironment.java:157)
	at net.imagej.ops.image.ImageNamespace.distancetransform(ImageNamespace.java:134)
	at net.imagej.ops.image.watershed.WatershedBinarySingleSigma.compute(WatershedBinarySingleSigma.java:103)
	at net.imagej.ops.image.watershed.WatershedBinarySingleSigma.compute(WatershedBinarySingleSigma.java:80)
	at net.imagej.ops.special.hybrid.UnaryHybridCF.calculate(UnaryHybridCF.java:61)
	at net.imagej.ops.special.hybrid.UnaryHybridCF.run(UnaryHybridCF.java:71)
	at net.imagej.ops.special.hybrid.UnaryHybridCF.run(UnaryHybridCF.java:97)
	at org.scijava.command.CommandModule.run(CommandModule.java:199)
	at net.imagej.ops.OpEnvironment.run(OpEnvironment.java:950)
	at net.imagej.ops.OpEnvironment.run(OpEnvironment.java:157)
	at net.imagej.ops.image.ImageNamespace.watershed(ImageNamespace.java:602)
	at net.imagej.ops.image.ImageNamespace$watershed.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at Script9.run(Script9.groovy:21)
	at org.scijava.plugins.scripting.groovy.GroovyScriptEngine.eval(GroovyScriptEngine.java:303)
	at org.scijava.plugins.scripting.groovy.GroovyScriptEngine.eval(GroovyScriptEngine.java:122)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
	at org.scijava.script.ScriptModule.run(ScriptModule.java:160)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:228)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Started watershed.groovy at Wed Nov 27 14:18:12 CET 2019
java.lang.ClassCastException: net.imglib2.type.numeric.integer.UnsignedByteType cannot be cast to net.imglib2.type.numeric.integer.UnsignedShortType
	at net.imglib2.type.numeric.integer.UnsignedShortType.compareTo(UnsignedShortType.java:50)
	at net.imagej.ops.threshold.apply.ApplyThresholdComparable.compute(ApplyThresholdComparable.java:56)
	at net.imagej.ops.threshold.apply.ApplyThresholdComparable.compute(ApplyThresholdComparable.java:46)
	at net.imagej.ops.special.computer.BinaryComputerOp.compute(BinaryComputerOp.java:93)
	at net.imagej.ops.map.MapIterableToIterable.compute(MapIterableToIterable.java:58)
	at net.imagej.ops.map.MapIterableToIterable.compute(MapIterableToIterable.java:47)
	at net.imagej.ops.threshold.apply.ApplyConstantThreshold.compute(ApplyConstantThreshold.java:73)
	at net.imagej.ops.threshold.apply.ApplyConstantThreshold.compute(ApplyConstantThreshold.java:50)
	at net.imagej.ops.threshold.AbstractApplyThresholdIterable.compute(AbstractApplyThresholdIterable.java:52)
	at net.imagej.ops.threshold.AbstractApplyThresholdIterable.compute(AbstractApplyThresholdIterable.java:43)
	at net.imagej.ops.special.hybrid.UnaryHybridCF.calculate(UnaryHybridCF.java:61)
	at net.imagej.ops.special.hybrid.UnaryHybridCF.run(UnaryHybridCF.java:71)
	at net.imagej.ops.special.hybrid.UnaryHybridCF.run(UnaryHybridCF.java:97)
	at org.scijava.command.CommandModule.run(CommandModule.java:199)
	at net.imagej.ops.OpEnvironment.run(OpEnvironment.java:950)
	at net.imagej.ops.OpEnvironment.run(OpEnvironment.java:157)
	at net.imagej.ops.threshold.ThresholdNamespace.apply(ThresholdNamespace.java:82)
	at net.imagej.ops.threshold.ThresholdNamespace$apply.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
	at Script10.run(Script10.groovy:13)
	at org.scijava.plugins.scripting.groovy.GroovyScriptEngine.eval(GroovyScriptEngine.java:303)
	at org.scijava.plugins.scripting.groovy.GroovyScriptEngine.eval(GroovyScriptEngine.java:122)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
	at org.scijava.script.ScriptModule.run(ScriptModule.java:160)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:228)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Started watershed.groovy at Wed Nov 27 14:18:34 CET 2019
Started watershed.groovy at Wed Nov 27 14:19:17 CET 2019
java.lang.ClassCastException
Started watershed.groovy at Wed Nov 27 14:19:50 CET 2019
Started watershed.groovy at Wed Nov 27 14:20:15 CET 2019
Started watershed.groovy at Wed Nov 27 14:21:50 CET 2019
java.lang.IllegalArgumentException: No matching 'net.imagej.ops.Ops$Image$DistanceTransform' op

Request:
-	net.imagej.ops.Ops$Image$DistanceTransform(
		DefaultDataset)

Candidates:
1. 	(RandomAccessibleInterval out?) =
	net.imagej.ops.image.distancetransform.DistanceTransform2D(
		RandomAccessibleInterval out?,
		RandomAccessibleInterval in)
	Inputs do not conform to op rules
		out = null
		in = test_image.tif
2. 	(RandomAccessibleInterval out?) =
	net.imagej.ops.image.distancetransform.DistanceTransform2DCalibration(
		RandomAccessibleInterval out?,
		RandomAccessibleInterval in,
		double[] calibration)
	Not enough arguments: 1 < 2
3. 	(RandomAccessibleInterval out?) =
	net.imagej.ops.image.distancetransform.DistanceTransform3D(
		RandomAccessibleInterval out?,
		RandomAccessibleInterval in)
	Inputs do not conform to op rules
		out = null
		in = test_image.tif
4. 	(RandomAccessibleInterval out?) =
	net.imagej.ops.image.distancetransform.DistanceTransform3DCalibration(
		RandomAccessibleInterval out?,
		RandomAccessibleInterval in,
		double[] calibration)
	Not enough arguments: 1 < 2
5. 	(RandomAccessibleInterval out?) =
	net.imagej.ops.image.distancetransform.DefaultDistanceTransform(
		RandomAccessibleInterval out?,
		RandomAccessibleInterval in)
	Inputs do not conform to op rules
		out = null
		in = test_image.tif
6. 	(RandomAccessibleInterval out?) =
	net.imagej.ops.image.distancetransform.DefaultDistanceTransformCalibration(
		RandomAccessibleInterval out?,
		RandomAccessibleInterval in,
		double[] calibration)
	Not enough arguments: 1 < 2

	at net.imagej.ops.DefaultOpMatchingService.singleMatch(DefaultOpMatchingService.java:432)
	at net.imagej.ops.DefaultOpMatchingService.findMatch(DefaultOpMatchingService.java:97)
	at net.imagej.ops.DefaultOpMatchingService.findMatch(DefaultOpMatchingService.java:83)
	at net.imagej.ops.OpEnvironment.module(OpEnvironment.java:269)
	at net.imagej.ops.OpEnvironment.run(OpEnvironment.java:157)
	at net.imagej.ops.image.ImageNamespace.distancetransform(ImageNamespace.java:134)
	at net.imagej.ops.image.watershed.WatershedBinarySingleSigma.compute(WatershedBinarySingleSigma.java:103)
	at net.imagej.ops.image.watershed.WatershedBinarySingleSigma.compute(WatershedBinarySingleSigma.java:80)
	at net.imagej.ops.special.hybrid.UnaryHybridCF.calculate(UnaryHybridCF.java:61)
	at net.imagej.ops.special.hybrid.UnaryHybridCF.run(UnaryHybridCF.java:71)
	at net.imagej.ops.special.hybrid.UnaryHybridCF.run(UnaryHybridCF.java:97)
	at org.scijava.command.CommandModule.run(CommandModule.java:199)
	at net.imagej.ops.OpEnvironment.run(OpEnvironment.java:950)
	at net.imagej.ops.OpEnvironment.run(OpEnvironment.java:157)
	at net.imagej.ops.image.ImageNamespace.watershed(ImageNamespace.java:602)
	at net.imagej.ops.image.ImageNamespace$watershed.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at Script15.run(Script15.groovy:21)
	at org.scijava.plugins.scripting.groovy.GroovyScriptEngine.eval(GroovyScriptEngine.java:303)
	at org.scijava.plugins.scripting.groovy.GroovyScriptEngine.eval(GroovyScriptEngine.java:122)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
	at org.scijava.script.ScriptModule.run(ScriptModule.java:160)
	at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
	at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66)
	at org.scijava.thread.DefaultThreadService.lambda$wrap$2(DefaultThreadService.java:228)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
@bpavie bpavie changed the title ops.image().watershed fail on large image ops.image().watershed failed on large image Nov 27, 2019
@imagejan
Copy link
Member

That's an imagej-ops issue, I believe.

The distanceTransform ops (which are used by the watershed op) have a rather restrictive conforms() check:

https://github.com/imagej/imagej-ops/blob/7870ad17a3cd36e5c9915580a25d53da5893f5c3/src/main/java/net/imagej/ops/image/distancetransform/DefaultDistanceTransform.java#L75-L81

Your sample image is 35206 * 42619 pixels, and the ops will only match if:

(x*x) + (y*y) < Integer.MAX_VALUE

There is currently no ops implementation that matches for the given request.

@bpavie
Copy link
Author

bpavie commented Nov 28, 2019 •
edited
Loading

Thanks a lot for your quick answer!

So it means we cannot perform this ops on image with more than Integer.MAX_VALUE/2 pixels (so no more than 1,073,741,823, or roughly 32,000*32,000).

Is there any reason for this limit, because it should be more linked to the memory size available. It looks like somewhere in the code, an int array is used in this ops with a length equal of the number of the pixel in the image, which probably explain this limitation since java array length are limited to Integer.MAX_VALUE , could it be modified to store them into a List to avoid this limitation?
It looks to me like an old imagej1 limitation...

Benjamin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bpavie @imagejan