Enable GUI tests in CI

[constantinpape]

It would help with all the refactoring tasks to enable tests for the GUI. Currently this is not possible, becaue napari cannot be imported in the Github CLI due to missing graphical libraries (QT, but might lack even more things like OpenGL).

The best strategy here would probably look into what napari does for the CI. (I tried this myself at some point, but gave up because the napari test suite is quite extensive and it was unclear what the best starting point is).

[GenevieveBuckley]

Testing napari plugins

• For reference, link to the documentation page on testing napari plugins: https://napari.org/stable/plugins/test_deploy.html
• We will need to switch to pytest (instead of the current unittest). We need both pytest and pytest-qt to handle tests involving the GUI. Fortunately, pytest can already run the current unit tests with no modification, so there is no extra work needed there (the existing tests are already compatible with pytest).

Github actions CI

The complications around GUI testing in the GitHub actions CI can be solved by adding two actions to the workflow yaml file:

• tlambert03/setup-qt-libs, and
• aganders3/headless-gui

Setup Qt libraries

First, to set up the Qt graphical libraries on the remote CI machine, we can use Talley Lambert's tlambert03/setup-qt-libs action.

Click here to see the exact line where it is used in the napari workflow file

- uses: tlambert03/setup-qt-libs@v1

Run headless GUI during testing

Then, in the step where the tests are run, we can use Ashley Anderson's aganders3/headless-gui action. This gives us a virtual display for the tests.

Click here to see the exact line where it is used in the napari workflow file

- uses: aganders3/headless-gui@v1

Workflow YAML file

We'll use this section of the napari yaml workflow to base ours on. It can likely be simplified a lot for our use.

[GenevieveBuckley]

Ok here's a demo you can try.

1. First, install pytest and pytest-qt into your python development environment
2. pytest test/test_gui.py

test_gui.py:

import skimage.data
from micro_sam.sam_annotator import annotator_2d
from micro_sam.sam_annotator.annotator_2d import _initialize_viewer


def test_something_with_a_viewer(make_napari_viewer_proxy):
    """The demo example from the napari plugin testing docs."""
    viewer = make_napari_viewer_proxy()
    # carry on with your test
    image = skimage.data.chelsea()
    viewer.add_image(image, name="raw")
    viewer.close()


def test_annotator_2d(make_napari_viewer_proxy, tmp_path):
    """Integration test for annotator_2d widget."""
    model_type = "vit_b"
    image = skimage.data.camera()
    embedding_path = tmp_path / "test-embedding.zarr"

    viewer = make_napari_viewer_proxy()
    viewer = _initialize_viewer(image, None, None, None)  # TODO: fix hacky workaround
    # test generating image embedding, then adding micro-sam dock widgets to the GUI
    viewer = annotator_2d(
        image,
        embedding_path,
        show_embeddings=False,
        model_type=model_type,
        v=viewer,
        return_viewer=True
    )
    assert len(viewer.layers) == 6
    expected_layer_names = ['raw', 'auto_segmentation', 'committed_objects', 'current_object', 'point_prompts', 'prompts']
    for layername in expected_layer_names:
        assert layername in viewer.layers
    # ... continue if you want to actually segment, commit, or clear annotatons
    viewer.close()  # must close the viewer at the end of tests

[constantinpape]

• We will need to switch to pytest (instead of the current unittest). We need both pytest and pytest-qt to handle tests involving the GUI. Fortunately, pytest can already run the current unit tests with no modification, so there is no extra work needed there (the existing tests are already compatible with pytest).

👍

Ok here's a demo you can try.

Thanks! I will give this a try later and then give some more feedback.

[constantinpape]

Hi @GenevieveBuckley,
I had a look at the test now, and it works for me locally when:

• Installing pytest and pytest-gui in the environment.
• Copy pasting your test file (I called it gui_test.py)
• And running it via pytest -v gui_test.py.

Regarding the test itself:

• we could use vit_t instead of vit_b, so that the model download is smaller and the embedding prediction faster.
• and then we could add some point and/or box labels, pass them to the respective layers, run segmentation, and compare to the expected result (which can be efficiently saved in the test via RLE).

But we can iterate on the exact details of the test later (and I can also take over some of the actual functionality testing once we have a scaffold for the test and it runs in the CI).
I assume we should finalize #214 next before you turn to implementing this in the CI.

[GenevieveBuckley]

I assume we should finalize #214 next before you turn to implementing this in the CI.

#214 is ready for review and merge now. It's not a blocker to work on this issue (but yes it is nice to have the test coverage information)

we could use vit_t instead of vit_b, so that the model download is smaller and the embedding prediction faster.

Good idea, will do.

and then we could add some point and/or box labels, pass them to the respective layers, run segmentation, and compare to the expected result (which can be efficiently saved in the test via RLE).

I've just spent some time on this problem today.

• I put something together that runs, but I was very surprised at how little extra test coverage this added (maybe those functions are already covered by unit tests?). It may not be worth the extra complexity to include it.
• I also tried running the automatic mask generation too, but this is pretty slow (even when I make the input image tiny).

Possible problems I've found:

• annotator_3d (and probably lots of others) doesn't allow you to pass an existing viewer in, it will automatically create a new napari viewer for you. This doesn't work well with the make_napari_viewer_proxy test fixture, which creates a viewer object to use in the tests.
• Running two tests involving the GUI is producing the error message RuntimeError: wrapped C/C++ object of type QWidget has been deleted. There might be some problem with how we're handling the dock widgets? I'm not sure.

[GenevieveBuckley]

Today I learned about pytest-gui, I'd always been using pytest-qt. That's cool to know! (Looks like it might also work for Tkinter, whereas pytest-qt is only for pyqt and pyside - I'll have to look at the docs more sometime)

[GenevieveBuckley]

Ok, here's the work I did extending tests to actually run some of the annotation functions.
I don't think it is worth including this in the integration tests - it doesn't help our code coverage much, and it adds complexity to the test. Also, in the case of the automatic mask generation, it is quite slow to run.

Here's an example GUI test involving interaction with the annotation functions (click to expand)

import numpy as np
import skimage.data
from micro_sam.sam_annotator import annotator_2d
from micro_sam.sam_annotator.annotator_2d import _initialize_viewer, _segment_widget
from micro_sam.sam_annotator.util import _clear_widget, _commit_segmentation_widget


def test_annotator_2d(make_napari_viewer_proxy, tmp_path):
    """Integration test for annotator_2d widget.

    * Creates 2D image embedding
    * Opens annotator_2d widget in napari
    * Test point prompts (add points, segment object, clear, and commit)
    * Test box prompt (add rectangle prompt, segment object, clear, and commit)
    ...
    """
    model_type = "vit_t"
    image = skimage.data.camera()
    embedding_path = tmp_path / "test-embedding.zarr"

    viewer = make_napari_viewer_proxy()
    viewer = _initialize_viewer(image, None, None, None)  # TODO: fix hacky workaround
    # test generating image embedding, then adding micro-sam dock widgets to the GUI
    viewer = annotator_2d(
        image,
        embedding_path,
        show_embeddings=False,
        model_type=model_type,
        v=viewer,
        return_viewer=True
    )
    # check the initial layer setup is correct
    assert len(viewer.layers) == 6
    expected_layer_names = ['raw', 'auto_segmentation', 'committed_objects', 'current_object', 'point_prompts', 'prompts']
    for layername in expected_layer_names:
        assert layername in viewer.layers
    # Check layers are empty before beginning tests
    np.testing.assert_equal(viewer.layers["auto_segmentation"].data, 0)
    np.testing.assert_equal(viewer.layers["current_object"].data, 0)
    np.testing.assert_equal(viewer.layers["committed_objects"].data, 0)
    np.testing.assert_equal(viewer.layers["point_prompts"].data, 0)
    assert viewer.layers["prompts"].data == []  # shape data is list, not numpy array
    # ========================================================================
    # TEST POINT PROMPTS
    # Add three points in the sky region of the camera image
    sky_point_prompts = np.array([[70, 80],[50, 320],[80, 470 ]])
    viewer.layers["point_prompts"].data = sky_point_prompts

    # Segment sky region of image
    _segment_widget(v=viewer)  # segment slice
    # We expect roughly 25% of the image to be sky
    sky_segmentation = np.copy(viewer.layers["current_object"].data)
    segmented_pixel_percentage = (np.sum(sky_segmentation == 1) / image.size) * 100
    assert segmented_pixel_percentage > 25
    assert segmented_pixel_percentage < 30

    # Clear segmentation current object and prompts
    _clear_widget(v=viewer)
    np.testing.assert_equal(viewer.layers["current_object"].data, 0)
    np.testing.assert_equal(viewer.layers["point_prompts"].data, 0)
    assert viewer.layers["prompts"].data == []  # shape data is list, not numpy array

    # Repeat segmentation and commit segmentation result
    viewer.layers["point_prompts"].data = sky_point_prompts
    _segment_widget(v=viewer)  # segment slice
    np.testing.assert_equal(sky_segmentation, viewer.layers["current_object"].data)
    # Commit segmentation
    _commit_segmentation_widget(v=viewer)
    np.testing.assert_equal(sky_segmentation, viewer.layers["committed_objects"].data)

    # ========================================================================
    # TEST BOX PROMPTS
    # Add rechangle bounding box prompt
    camera_bounding_box_prompt = np.array([[139, 254],[139, 324],[183, 324],[183, 254]])
    viewer.layers["prompts"].data = [camera_bounding_box_prompt]
    # Segment slice
    _segment_widget(v=viewer)  # segment slice
    # Check segmentation results
    camera_segmentation = np.copy(viewer.layers["current_object"].data)
    segmented_pixels = np.sum(camera_segmentation == 1)
    assert segmented_pixels > 2500  # we expect roughly 2770 pixels
    assert segmented_pixels < 3000  # we expect roughly 2770 pixels
    assert (camera_segmentation[150:175,275:310] == 1).all()  # small patch which should definitely be inside segmentation

    # Clear segmentation current object and prompts
    _clear_widget(v=viewer)
    np.testing.assert_equal(viewer.layers["current_object"].data, 0)
    np.testing.assert_equal(viewer.layers["point_prompts"].data, 0)
    assert viewer.layers["prompts"].data == []  # shape data is list, not numpy array

    # Repeat segmentation and commit segmentation result
    viewer.layers["prompts"].data = [camera_bounding_box_prompt]
    _segment_widget(v=viewer)  # segment slice
    np.testing.assert_equal(camera_segmentation, viewer.layers["current_object"].data)
    # Commit segmentation
    _commit_segmentation_widget(v=viewer)
    committed_objects = viewer.layers["committed_objects"].data
    # We expect two committed objects
    # label id 1: sky segmentation
    # label id 2: camera segmentation
    np.testing.assert_equal(np.unique(committed_objects), np.array([0, 1, 2]))
    np.testing.assert_equal(committed_objects == 2, camera_segmentation == 1)

    # ========================================================================
    viewer.close()  # must close the viewer at the end of tests

Here's a GUI test involving automatic mask generation (click to expand)

import numpy as np
from micro_sam.sam_annotator import annotator_2d
from micro_sam.sam_annotator.annotator_2d import _initialize_viewer, _autosegment_widget


def test_annotator_2d_amg(make_napari_viewer_proxy, tmp_path):
    """Integration test for annotator_2d widget with automatic mask generation.

    * Creates 2D image embedding
    * Opens annotator_2d widget in napari
    * Test automatic mask generation
    """
    model_type = "vit_t"
    embedding_path = tmp_path / "test-embedding.zarr"
    # example data - a basic checkerboard pattern
    image = np.zeros((16,16))
    image[:8,:8] = 1
    image[8:,8:] = 1

    viewer = make_napari_viewer_proxy()
    viewer = _initialize_viewer(image, None, None, None)  # TODO: fix hacky workaround
    # test generating image embedding, then adding micro-sam dock widgets to the GUI
    viewer = annotator_2d(
        image,
        embedding_path,
        show_embeddings=False,
        model_type=model_type,
        v=viewer,
        return_viewer=True
    )
    # check the initial layer setup is correct
    assert len(viewer.layers) == 6
    expected_layer_names = ['raw', 'auto_segmentation', 'committed_objects', 'current_object', 'point_prompts', 'prompts']
    for layername in expected_layer_names:
        assert layername in viewer.layers
    # Check layers are empty before beginning tests
    np.testing.assert_equal(viewer.layers["auto_segmentation"].data, 0)
    np.testing.assert_equal(viewer.layers["current_object"].data, 0)
    np.testing.assert_equal(viewer.layers["committed_objects"].data, 0)
    np.testing.assert_equal(viewer.layers["point_prompts"].data, 0)
    assert viewer.layers["prompts"].data == []  # shape data is list, not numpy array
    # ========================================================================
    # Automatic mask generation
    _autosegment_widget(v=viewer, min_object_size=30)
    # We expect four segmentation regions to be identified
    expected_segmentation_label_ids = np.array([0,1,2,3])
    np.testing.assert_equal(np.unique(viewer.layers["auto_segmentation"].data),
                            expected_segmentation_label_ids)
    viewer.close()  # must close the viewer at the end of tests

I have also run into problems with errors when I run two GUI tests. I'm not entirely sure how to fix that, something funny might be going on with the widgets and garbage collection(?)
RuntimeError: wrapped C/C++ object of type QWidget has been deleted

[constantinpape]

• I put something together that runs, but I was very surprised at how little extra test coverage this added (maybe those functions are already covered by unit tests?). It may not be worth the extra complexity to include it.

Yes, the functions are covered by unit tests, but I think it makes sense to add an integration test.
I am at a conference this week and have a bit fewer time to look into this, but I will follow up briefly in the afternoon, and also see if I can add the codecov token.

[constantinpape]

Hi @GenevieveBuckley ,
my suggestion from my side:

• you go ahead and clean up CI testing - automatic code coverage reports #214 so that we can merge it and have test coverage + installation via mamba
• then you finish up Simple GUI test for CI #223 , but just add a simple integration test that starts the GUI. I will look into more complex integration tests that use the actual functionality (but thanks for starting with this, I will definitely take a look!)
  • Also, it would be great if you can figure out why running more than one test currently fails.

Let me know if you think we should go ahead differently, or if you run into any issues.

[constantinpape]

This is implemented now. Thanks @GenevieveBuckley!