[Feature Request] Add support for overlapping segments

[fedorov]

What feature or change would you like to see made?

Overlapping DICOM SEG segments are currently no loaded with the error message indicating they are not supported in v3. Overlapping segments were supported in v2.

Why should we prioritize this feature?

IDC project is currently using v2, which is able to render overlapping segments. IDC contains data that contains overlapping segments, such as this one: https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.32722.99.99.62087908186665265759322018723889952421. Unless this feature is implemented, this would be a regression if IDC switches to v3.

[igoroctaviano]

@sedghi @rodrigobasilio2022 will discuss a solution to this issue.
A stack viewport is being implemented and might address this issue.

[james-hanks]

@sedghi link stack segmentation ticket (if we have one)

[fedorov]

Per @rodrigobasilio2022 (if I understand correctly), this is blocked by cornerstonejs/cornerstone3D#679.

[rodrigobasilio2022]

Yes, the steps are:

• Finish this PR @ohif/extension-vtk: Update VTK extension #679;
• Add overlapping segmentations in a new PR;
• Implement the modifications in OHIF;

[igoroctaviano]

Related to #3581

[james-hanks]

On hold because related fix will be published through industry partner project Q1 2024

[fedorov]

@james-hanks @igoroctaviano I observe that overlapping segments that are from different segmentations do work. Is this expected - that overlapping segments from different segmentations work, but from the same segmentation do not?

https://viewer.imaging.datacommons.cancer.gov/v3/viewer/?StudyInstanceUIDs=1.3.6.1.4.1.14519.5.2.1.6279.6001.191425307197546732281885591780

[fedorov]

overlapping segments from different segmentations work, but from the same segmentation do not

Yes, I confirmed the above. Interesting. The examples below can be used to demonstrate this (if you have little kids, you can ask them to play the game of "find differences" - it should be fun!):

• v2: https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.32722.99.99.71961866280433925571019872464419293819

• v3: https://viewer.imaging.datacommons.cancer.gov/v3/viewer/?StudyInstanceUIDs=1.3.6.1.4.1.32722.99.99.71961866280433925571019872464419293819

I do have to point out that I find it very disturbing that you show incorrect segmentation overlay by cropping overlap areas, and do not give any warning to the user about this. Segmentations are shown as if everything is normal. This is really bad, IMHO. I hope no one is looking at any overlapping segments that are used for patient care using v3!

[sedghi]

Yes, if there are separate segmentations they will work fine.

I find it very disturbing that you show incorrect segmentation overlay by cropping overlap areas, and do not give any warning to the user about this. Segmentations are shown as if everything is normal. This is really bad, IMHO. I hope no one is looking at any overlapping segments that are used for patient care using v3!

I thought we show a warning, but tried it now and we don't, hmmmm I will look into this, but I guess your comment is about overlapping segments in one segmentation right?

[fedorov]

your comment is about overlapping segments in one segmentation right?

Correct. I think it makes sense to warn users about lack of support of overlapping segments. Ideally, it would only show up when overlapping segments are detected. I tested, and did not see any such warnings.

[sedghi]

I'm pretty sure I had that warning, not sure how it is not showing up. I will fix it in the next PR

[fedorov]

Just discovered this today (on OHIF demo viewer) that there is still no support for overlapping segments AND there is no any kind of warning that what you see is not what you have CORRECTION: the warning does show up, but it is obscured by the research software warning popup that is in front!. In this particular case, the result is very confusing.

https://viewer.ohif.org/viewer?StudyInstanceUIDs=1.3.6.1.4.1.14519.5.2.1.3671.4754.298665348758363466150039312520

This is how it should look in reality (shown in v2): https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.14519.5.2.1.3671.4754.298665348758363466150039312520

[fedorov]

On hold because related fix will be published through industry partner project Q1 2024

For the record, it is way past Q1 2024, and this issue is still unresolved.

[fedorov]

Is there any update on the timeline for resolving this issue?

This missing feature is one of the biggest obstacles for us in IDC to switching to v3 as the primary viewer and deprecating v2.

[sedghi]

@fedorov Currently, this isn't possible. However, we have just layed the groundwork for the solution in Cornerstone3D 2.0. Once that's complete, the effort required will likely be much less.

cornerstonejs/cornerstone3D#1400

[fedorov]

What is very confusing is that it is apparently possible to display overlapping segments that come from different segmentations, but not if overlapping segments are in the same segmentation....

Example: https://viewer.imaging.datacommons.cancer.gov/v3/viewer/?StudyInstanceUIDs=1.2.840.113654.2.55.38309775581218325422753197576029670247

[igoroctaviano]

What is very confusing is that it is apparently possible to display overlapping segments that come from different segmentations, but not if overlapping segments are in the same segmentation....

Example: https://viewer.imaging.datacommons.cancer.gov/v3/viewer/?StudyInstanceUIDs=1.2.840.113654.2.55.38309775581218325422753197576029670247

I think every segmentation has its independent volume in CS3D so segments withing an SEG share the same volume which is the source the problem. @sedghi correct me if I'm wrong.

[sedghi]

@fedorov Yes, we can render two segmentations that have different segments that overlap, but overlapping segments as part of one segmentation is not supported. I mean, you can right now detect that they're overlapping segments and create two or three non-overlapping segmentations and then render them all. I guess that would suffice for you to migrate to version 3, until i fix the overlapping segments inside one segmentation and letting brush to perform on them

[fedorov]

you can right now detect that they're overlapping segments and create two or three non-overlapping segmentations and then render them all

It's even easier than this - there's an attribute that can be used to indicate whether segments overlap: https://dicom.innolitics.com/ciods/segmentation/segmentation-image/00620013.

In the general case, one would need to double-check, but in IDC we can potentially curate our segmentations to make sure this attribute is accurate for our data.

I guess the question is what is the timeline you think for your original approach? This will help me decide if we should invest into adding the workarounds within the current implementation (and by doing this save on the ongoing effort maintaining v2), or wait for the proper solution.

[sedghi]

Unfortunately, I don't think it will happen anytime soon. If it's easy, I can demonstrate how to manage two segmentations for rendering purposes. so we can upgrade IDC to v3.

The hard part is to figure out how to have the minimum number of segmentations such that within each, there are no overlapping segments.

@igoroctaviano

[fedorov]

This is very helpful. Thank you for your advice and transparency on this! We will discuss with Igor to plan the next steps.

[fedorov]

I mean, you can right now detect that they're overlapping segments and create two or three non-overlapping segmentations and then render them all.

@sedghi would you welcome such a workaround in v3, or would they need to go into IDC fork?

[sedghi]

It is welcomed indeed

[fedorov]

As of today, per @pedrokohler, we need to wait for the release of cornerstone3d 2.0, which should be released tomorrow or on Thursday to start working.

[fedorov]

Now that #4347 is merged, this can proceed.

[pedrokohler]

@fedorov FYI the first step for a solution has just been merged here cornerstonejs/cornerstone3D#1692

We still have a final step to take and allow the overlapping segs. This work should be done in the next days (maybe couple of weeks, because of the holidays)