Medical image viewer

[jakubMitura14]

I post here not as a problem but so it can be used by others, If You will consider it interesting and would like to help me develop idea father contact me

I finally managed it basically I load 3 dimensional data stored in hdf5 (I loaded it into hdf5 from raw using python)

It enables viewing transverse slices and annotate 3d pathes in a mask that will be displayed over main image

exmpleH = @spawnat persistenceWorker Main.h5manag.getExample()
minimumm = -1000
maximumm = 2000
arrr= fetch(exmpleH)
imageDim = size(arrr)
using GLMakie
maskArr = Observable(BitArray(undef, imageDim))
MyImgeViewer.singleCtScanDisplay(arrr, maskArr,minimumm, maximumm)

Now definition of the required modules

```@doc
functions responsible for displaying medical image Data
```
using DrWatson
@quickactivate "Probabilistic medical segmentation"


module MyImgeViewer

using GLMakie
using Makie
#using GeometryBasics
using GeometricalPredicates
using ColorTypes
using Distributed
using GLMakie
using Main.imageViewerHelper
using Main.workerNumbers
## getting id of workers 

```@doc
simple display of single image - only in transverse plane we are adding also a mask  that 
arrr - main 3 dimensional data representing medical image for example in case of CT each voxel represents value of X ray attenuation
minimumm, maximumm - approximately minimum and maximum values we can have in our image
```
function singleCtScanDisplay(arrr ::Array{Number, 3}, maskArr , minimumm, maximumm) 
#we modify 2 pixels just in order to make the color range constant so slices will be displayed in the same windows
arrr[1,1,:].= minimumm 
arrr[2,1,:].= maximumm 
  

imageDim = size(arrr) # dimenstion of the primary image for example CT scan
slicesNumb =imageDim[3] # number of slices 

#defining layout variables
scene, layout = GLMakie.layoutscene(resolution = (600, 400))
ax1 = layout[1, 1] = GLMakie.Axis(scene, backgroundcolor = :transparent)
ax2 = layout[1, 1] = GLMakie.Axis(scene, backgroundcolor = :transparent)

#control widgets
sl_x =layout[2, 1]= GLMakie.Slider(scene, range = 1:1: slicesNumb , startvalue = slicesNumb/2 )
sliderXVal = sl_x.value


#color maps
cmwhite = cgrad(range(RGBA(10,10,10,0.01), stop=RGBA(0,0,255,0.4), length=10000));
greyss = createMedicalImageColorSchemeB(200,-200,maximumm, minimumm )
####heatmaps

#main heatmap that holds for example Ct scan
currentSliceMain = GLMakie.@lift(arrr[:,:, convert(Int32,$sliderXVal)])
hm = GLMakie.heatmap!(ax1, currentSliceMain ,colormap = greyss) 

#helper heatmap designed to respond to both changes in slider and changes in the bit matrix
currentSliceMask = GLMakie.@lift($maskArr[:,:, convert(Int32,$sliderXVal)])
hmB = GLMakie.heatmap!(ax1, currentSliceMask ,colormap = cmwhite) 

#adding ability to be able to add information to mask  where we clicked so in casse of mit matrix we will set the point where we clicked to 1 
indicatorC(ax1,imageDim,scene,maskArr,sliderXVal)

#displaying
colorB = layout[1,2]= Colorbar(scene, hm)
GLMakie.translate!(hmB, Vec3f0(0,0,5))  

scene

end

```@doc
inspired by   https://github.com/JuliaPlots/Makie.jl/issues/810
Generaly thanks to this function  the viewer is able to respond to clicking on the slices and records it in the supplied 3 dimensional AbstractArray
  ax - Axis which store our heatmap slices which we want to observe wheather user clicked on them and where
  dims - dimensions of  main image for example CT
  sc - Scene where our axis is
  maskArr - the 3 dimensional bit array  that has exactly the same dimensions as main Array storing image 
  sliceNumb - represents on what slide we are on currently on - ussually it just give information from slider 
```
function indicatorC(ax::Axis,dims::Tuple{Int64, Int64, Int64},sc::Scene,maskArr,sliceNumb::Observable{Any})
  register_interaction!(ax, :indicator) do event::GLMakie.MouseEvent, axis
  if event.type === MouseEventTypes.leftclick
    println("clicked")
    #@async begin
      #appropriately modyfing wanted pixels in mask array
   @async calculateMouseAndSetmaskWrap(maskArr, event,sc,dims,sliceNumb)            
    #  
    #  
    #  println("fetched" + fetch(maskA))

    #  finalize(maskA)
    #end
    return true
    #print("xMouse: $(xMouse)  yMouse: $(yMouse)   compBoxWidth: $(compBoxWidth)  compBoxHeight: $(compBoxHeight)   calculatedXpixel: $(calculatedXpixel)  calculatedYpixel: $(calculatedYpixel)      pixelsNumbInX  $(pixelsNumbInX)         ") 
  end
  
end
end
```@doc
wrapper for calculateMouseAndSetmask  - from imageViewerHelper module
  given mouse event modifies mask accordingly
  maskArr - the 3 dimensional bit array  that has exactly the same dimensions as main Array storing image 
  event - mouse event passed from Makie
  sc - scene we are using in Makie
  ```
function calculateMouseAndSetmaskWrap(maskArr, event,sc,dims,sliceNumb) 
  maskArr[] = calculateMouseAndSetmask(maskArr, event,sc,dims,sliceNumb)
end


end #module

and helper methods

```@doc
functions responsible for helping in image viewer - those functions are  meant to be invoked on separate process
- in parallel
```
using DrWatson
@quickactivate "Probabilistic medical segmentation"

module imageViewerHelper
using Documenter
using ColorTypes
using Colors, ColorSchemeTools
using Makie

export calculateMouseAndSetmask
export createMedicalImageColorSchemeB
# using AbstractPlotting
```@doc
  given mouse event modifies mask accordingly
  maskArr - the 3 dimensional bit array  that has exactly the same dimensions as main Array storing image 
  event - mouse event passed from Makie
  sc - scene we are using in Makie
  ```
function calculateMouseAndSetmask(maskArr, event,sc,dims,sliceNumb) 
  #position from top left corner 
  xMouse= Makie.to_world(sc,event.data)[1]
  yMouse= Makie.to_world(sc,event.data)[2]
  #data about height and width in layout
  compBoxWidth = 510 
  compBoxHeight = 510 
  #image dimensions - number of pixels  from medical image for example ct scan
  pixelsNumbInX =dims[1]
  pixelsNumbInY =dims[2]
  #calculating over which image pixel we are
  calculatedXpixel =convert(Int32, round( (xMouse/compBoxWidth)*pixelsNumbInX) )
  calculatedYpixel =  convert(Int32,round( (yMouse/compBoxHeight)*pixelsNumbInY ))
  sliceNumbConv =convert(Int32,round( sliceNumb[] ))
  #appropriately modyfing wanted pixels in mask array
  return markMaskArrayPatch( maskArr ,CartesianIndex(calculatedXpixel, calculatedYpixel, sliceNumbConv  ),2)
end



```@doc
  maskArr - the 3 dimensional bit array  that has exactly the same dimensions as main Array storing image 
  point - cartesian coordinates of point around which we want to modify the 3 dimensional array from 0 to 1

```
function markMaskArrayPatch(maskArr, pointCart::CartesianIndex{3}, patchSize ::Int64)

  ones = CartesianIndex(patchSize,patchSize,patchSize) # cartesian 3 dimensional index used for calculations to get range of the cartesian indicis to analyze
  maskArrB = maskArr[]
  for J in (pointCart-ones):(pointCart+ones)
    diff = J - pointCart # diffrence between dimensions relative to point of origin
      if cartesianTolinear(diff) <= patchSize
        maskArrB[J]=1
      end
      end
return maskArrB
end
```@doc
works only for 3d cartesian coordinates
  cart - cartesian coordinates of point where we will add the dimensions ...
```
function cartesianTolinear(pointCart::CartesianIndex{3}) :: Int16
   abs(pointCart[1])+ abs(pointCart[2])+abs(pointCart[3])
end

```@doc
creating grey scheme colors for proper display of medical image mainly CT scan
min_shown_white - max_shown_black range over which the gradint of greys will be shown
truemax - truemin the range of values in the image for which we are creating the scale
```
#taken from https://stackoverflow.com/questions/67727977/how-to-create-julia-color-scheme-for-displaying-ct-scan-makie-jl/67756158#67756158

function createMedicalImageColorSchemeB(min_shown_white,max_shown_black,truemax,truemin ) ::Vector{Any} 
# println("max_shown_black - truemin + 1")
# println(max_shown_black - truemin + 1)
# println(" min_shown_white - max_shown_black - 1")
# println( min_shown_white - max_shown_black - 1)
# println("truemax - min_shown_white + 1")
# println(truemax - min_shown_white + 1)

return  [fill(colorant"black", max_shown_black - truemin + 1);
               collect(make_colorscheme(identity, identity, identity,
                   length = min_shown_white - max_shown_black - 1));
               fill(colorant"white", truemax - min_shown_white + 1)]

end





end #module

[SimonDanisch]

Cool, do you plan to put this in a package?

[jakubMitura14]

Yes ultimately yes I am working on integration of hdf5 filesystem in a distributed manner and apply some algorithms for segmentation - still work in progress

[SimonDanisch]

btw, there is an under documented contour(volume) function, which may be nice for this kind of data!
It's shown in the WGLMakie documentation :D :
http://juliaplots.org/WGLMakie.jl/stable/#Record-a-statemap

[jakubMitura14]

Thanks! I will investigate it

[jakubMitura14]

I see It got reactivated as a discussion, Back in 2021 I had huge problems with speed of makie to enable scrolling through 3d data fast enough to be smooth ( it may no longer be the issue) hence the package MedEye3d was developed, and now it is during the process of introducing multiple improvements thanks to Google summer of Code program. If anybody is interested about it you can reach out to me :)

https://github.com/JuliaHealth/MedEye3d.jl

[asinghvi17]

It would be pretty cool to get MedEye3D to interface with Makie, maybe as a Block layoutable? I'm not sure how that would fit into the GLMakie renderloop though, unless one uses an RPRMakie-like approach where you write to the framebuffer directly.

[jakubMitura14]

From rendering perspective image is a texture (or series of textures) displayed on 2 triangles that create rectange, during scroll one just update texture data and the shader is responsible for clipping (for radiological window) overlay (display simultaneously images of different modalities and annotation) . Additionally medeye is capturing keyboard via glfw as it has some shortcuts, and mouse position for annotation.

Would be nice to get makie integration- do you think in such case it is possible to do?

[SimonDanisch]

I'd be really surprised if performance was ever an issue, if you didn't run into a performance bug or used makie incorrectly;) did you ever open an issue about what's slow?

[jakubMitura14]

Frankly it was 2021 and I do not remember exactly :); but back then i just tried what is in the begining of the issue not getting custom shader etc.

[SimonDanisch]

Well if you ever give it a shot to port it back to Makie let me know ;) Should have lots of advantages, with the UI and everything.
I'm happy to look at any performance problems you encounter, which I'm pretty sure should be easily solvable.

[jakubMitura14]

ok thanks !