[Feature Request] Can I Integrating Cellpose Model into RectLabel Pro for Enhanced Cell Segmentation

[zengzpi]

I am using RectLabel Pro to label cells as shown below:

However, I find that Cellpose is better suited for segmenting these objects due to its advanced accuracy and efficiency. I would like to integrate the Cellpose model into RectLabel Pro to automate and enhance my cell segmentation workflow.

I have converted the Cellpose model into an .mlpackage, but it seems I cannot load the Core ML model due to an input error.

Questions:

can I integrate the Cellpose model into RectLabel Pro to generate segmentation masks directly instead of SAM?

[ryouchinsa]

Thanks for writing the issue.

We will start reading the Cellpose repository and find out how to convert to the Core ML model. If we could convert, we will implement on RectLabel to decode the output layer to the pixels mask and polygon.
https://github.com/MouseLand/cellpose

[zengzpi]

Great! I’m very excited about the possibility of successfully integrating this into the app. Hopefully, the steps I’ve outlined maybe help a little:

Step 1: Load the Cyto3 Model

Load the Cyto3 model using the Cellpose library as shown below:
Run Cyto3 Model

from cellpose import models
model = models.Cellpose(gpu=True, model_type="cyto3")

---

Step 2: Save the Model to a .pt File Using TorchScript

import torch

device = torch.device("cpu")  # Use "cpu" in order to successfully convert into mlpackage in a macos

# 1) Move the model to the target device (CPU or GPU)
model.cp.net.to(device)
model.cp.net.eval()

# Example input for tracing (adjust the shape as needed based on your model's input)
example_input = torch.rand(1, 2, 256, 256)

# Trace the model using TorchScript
traced_model = torch.jit.trace(model.cp.net, example_input)

# Save the traced model
traced_model.save("cellpose_model.pt")

---

Step 3: Convert to .mlpackage Using CoreMLTools

import coremltools as ct

# Load the TorchScript model
model = ct.convert(
    model="cellpose_model.pt", 
    source="pytorch",
    inputs=[ct.TensorType(name="input", shape=(1, 2, 256, 256))],
    minimum_deployment_target=ct.target.macOS15  # Specify the deployment target
)

# Optionally, specify input types for better conversion
# input_shape = (1, 3, 256, 256)  # Adjust based on your model's input if necessary
# model = ct.convert("cellpose_model.pt", inputs=[ct.TensorType(name="input", shape=input_shape)])

# Save the CoreML model
model.save("cellpose.mlpackage")

---

Notes:

1. Step 1 and Step 2 maybe run in a linux system.
2. The CoreMLTools library requires macOS for creating and deploying .mlpackage files.