Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge cp4-windows to master branch #41

Open
wants to merge 84 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
f04354a
MAINT: add .gitignore
vkhodygo Apr 21, 2021
50e977d
WIP, MAINT: code refactoring/consolidation
vkhodygo Apr 21, 2021
87aac4c
UPD: correct handle_csv function
vkhodygo Apr 23, 2021
fe2c334
WIP, MAINT: code refactoring, pt2
vkhodygo Apr 23, 2021
f88576b
MAINT: add updated requirements, explicit only
vkhodygo Apr 23, 2021
3f22744
UPD: fix #24
vkhodygo Apr 26, 2021
3781a53
UPD: correct rev number, remove bad attribute init/redundant function
vkhodygo Aug 5, 2021
21149ce
MAINT: new requirements, new .gitignore
vkhodygo Aug 5, 2021
10f1d79
MAINT: move tests to a separate folder, reformat the code
vkhodygo Aug 5, 2021
41ddda9
MAINT: update imports, reformat code
vkhodygo Aug 5, 2021
7bde9d6
FIX: add missing style to wx text fields, remove bad attribute init
vkhodygo Aug 5, 2021
6142c96
Update README.md
Tiany1Pan Feb 6, 2022
f80fa6b
Update README.md
Tiany1Pan Feb 6, 2022
7139e10
Update requirements.txt
Tiany1Pan Feb 6, 2022
c4ccdbc
Update README.md
Tiany1Pan Feb 6, 2022
f11ac3f
Update README.md
Tiany1Pan Feb 6, 2022
6f8833a
Update README.md
Tiany1Pan Feb 6, 2022
62f6d51
Update README.md
Tiany1Pan Feb 6, 2022
666aef8
Update README.md
Tiany1Pan Feb 6, 2022
9c1da22
Update README.md
Tiany1Pan Feb 6, 2022
4a1aff8
Update README.md
Tiany1Pan Feb 6, 2022
7b36d2b
Update README.md
Tiany1Pan Feb 6, 2022
bae3465
Update requirements.txt
Tiany1Pan Feb 8, 2022
d737377
Update runoncluster.py
Tiany1Pan Feb 8, 2022
6fae1ef
Update clusterview.py
Tiany1Pan Feb 8, 2022
34edd55
Update README.md
Tiany1Pan Feb 8, 2022
7c93b52
Update README.md
Tiany1Pan Feb 8, 2022
3683d25
Update README.md
Tiany1Pan Feb 8, 2022
e72404c
Update README.md
Tiany1Pan Feb 9, 2022
9adf672
Update requirements.txt
Tiany1Pan Feb 9, 2022
3cd87a2
Update README.md
Tiany1Pan Feb 9, 2022
d4dfb30
Update README.md
Tiany1Pan Feb 9, 2022
7c53ae7
Update clusterview.py
Tiany1Pan Feb 9, 2022
91f1d18
Update runoncluster.py
Tiany1Pan Feb 9, 2022
3380ff6
Update README.md
Tiany1Pan Feb 9, 2022
566ad8e
Update README.md
Tiany1Pan Feb 9, 2022
df0210c
Update CPRynner.py
Tiany1Pan Feb 9, 2022
415ef87
Update README.md
Tiany1Pan Mar 15, 2022
7818217
Update requirements.txt
Tiany1Pan Mar 15, 2022
0e8ea60
Update README.md
Tiany1Pan Apr 6, 2022
650caff
Update README.md
Tiany1Pan Apr 6, 2022
9898216
Update README.md
Tiany1Pan Apr 8, 2022
4955f93
Update CPRynner.py
Tiany1Pan May 5, 2022
6e602b6
Update requirements.txt
Tiany1Pan May 5, 2022
a655f49
Update clusterview.py
Tiany1Pan May 5, 2022
1da83ce
Update runoncluster.py
Tiany1Pan May 5, 2022
3e22980
Update README.md
Tiany1Pan May 5, 2022
0a65c8f
Update clusterview.py
Tiany1Pan May 5, 2022
bbac556
Update runoncluster.py
Tiany1Pan May 5, 2022
c5c893a
Update .gitignore
Tiany1Pan May 5, 2022
d37dc88
Add .gitignore
MYPei May 3, 2022
a450234
Update imports and requirements.txt
MYPei May 3, 2022
e73ff71
Update dask requirement and clusterview docstrings
MYPei May 5, 2022
ee9b6e3
Updated .gitignore after directory shift
MYPei May 5, 2022
75fc829
Ready ClusterView for CPRynner functionality
MYPei May 26, 2022
e9bf269
wx fixes and pipeline trimming
MYPei Jun 15, 2022
3ed0b5c
Some pathing checks
MYPei Aug 9, 2022
32cc2d2
Separate remote pipeline and local pipeline saves
MYPei Aug 9, 2022
33f6eef
Local script saving and cluster URL fixes
MYPei Aug 10, 2022
00349cf
Cellprofiler command update
MYPei Aug 25, 2022
2195a9a
Run command field and pipeline adjustment removal
MYPei Sep 1, 2022
34e8608
ClusterView run visibility
MYPei Nov 11, 2022
e3e4f95
ClusterView download in panel
MYPei Nov 18, 2022
ee5e39a
RoC Function refactoring and testing
MYPei Dec 1, 2022
bee08a3
Modifications from review
MYPei Dec 5, 2022
534b05c
Updated requirements with branches
MYPei Dec 5, 2022
3159e6c
Script sanitisation and test fixes
MYPei Dec 13, 2022
e8f5c11
update README
Tiany1Pan Dec 13, 2022
d67651e
ClusterView test and temp directory cleanup
MYPei Dec 19, 2022
81a131a
Merge branch 'cp4-windows' of https://github.com/sa2c/CellProfiler-Ru…
MYPei Dec 19, 2022
d3d070d
Update README.md
Tiany1Pan Jan 10, 2023
9458476
Delete todo.txt
Tiany1Pan Jan 10, 2023
524acca
Delete __init__.py
Tiany1Pan Jan 10, 2023
c812cb2
final code review
Tiany1Pan Jan 10, 2023
3e6a4de
update gitignore and insert newlines
Tiany1Pan Jan 24, 2023
ca349b6
remove import future and pdb
Tiany1Pan Jan 24, 2023
914da35
Warning: code formatter
Tiany1Pan Jan 24, 2023
dab705d
Fixed formatting problems and removed commented-out lines of code
Tiany1Pan Jan 24, 2023
22bc620
updated README and added f-strings into clusterview
Tiany1Pan Jan 24, 2023
7764c3c
update requirements and import
Tiany1Pan Jan 24, 2023
0d94313
Formatted with Black
Tiany1Pan Jan 25, 2023
0b00223
re-formatted with black and isort
Tiany1Pan Jan 25, 2023
66838cb
fix rynner installation issue and rename it as lower-case
Tiany1Pan Jan 27, 2023
3123699
update requirements.txt
Tiany1Pan Jan 27, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
venv/
.idea/
__pycache__/
RunOnCluster.egg-info/
CellProfiler-plugins/
*/__pycache__/
__init__.py
setup.py
loadimagesfromomero.py
*.pyc
359 changes: 230 additions & 129 deletions CPRynner/CPRynner.py

Large diffs are not rendered by default.

41 changes: 21 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,43 +1,44 @@
# CellProfiler-RunOnCluster
# CellProfiler-RunOnCluster

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3275888.svg)](https://doi.org/10.5281/zenodo.3275888)

A CellProfiler module for submitting batch jobs to a cluster running slurm.

## Installation
### Windows

A precompiled executable with default settings for the Sunbird cluster in Swansea is provided in [releases](https://github.com/sa2c/CellProfiler-RunOnCluster/releases/download/v1.0/CellProfiler.exe). These executables require that a Java Runtime Environment is installed. You can get one for example form the [Java downloads page](https://www.java.com/en/download/).

### Source Installation
Note as our plugin requires additional libraries which aren’t packaged with CellProfiler 4, you’ll need to build CellProfiler from source rather than using a pre-packaged version. ([Reference](https://cellprofiler-manual.s3.amazonaws.com/CellProfiler-4.0.6/help/other_plugins.html?highlight=plugins)). The most straightforward method is to copy the plugin sources to your Cellprofiler plugins directory. Using the default `CellProfiler\plugins` folder as your plugin directory is recommended, but you can still have the plugins in other locations. Follow the instructions for installing CellProfiler on the [Wiki](https://github.com/CellProfiler/CellProfiler/wiki). Choose your operating system on the right-side panel. Once you have installed CellProfiler, set the plugin's directory in Cellprofiler preferences, then save and restart CellProfiler.

When running on Linux or developing your own plugins, the most straight forward method is to copy the plugin sources to your Cellprofiler plugins directory.
Please download the [plugins](https://codeload.github.com/sa2c/CellProfiler-RunOnCluster/zip/refs/heads/master) to your plugins directory. In the plugins directory, install the requirements for the plugins:

Follow the instructions for installing CellProfiler on the [wiki](https://github.com/CellProfiler/CellProfiler/wiki). Choose you operating system on the right side panel. Once you have installed CellProfiler, set the plugins directory in Cellprofiler preferences. Download [the plugins](https://github.com/sa2c/CellProfiler-RunOnCluster/archive/master.zip) and copy move the files to your plugins directory.

In the plugins directory, install the additional requirements for the plugins:
```
python2 -m pip install -r requirements.txt
python -m pip install -r requirements.txt
```

## Usage

### Submitting to Cluster
### Submitting jobs to the cluster

Once you have tested your pipeline on your local machine, add all images to be processed into the Images plugin in the usual way. Add the `RunOnCluster` module in the `Other` category to the end of the pipeline.

The module has four settings:
* Run Name: An identifier that allows you to recognize the pipeline and image batch.
* Number of images per measurement: If several image files are required for a single measurement, adjust this to the number of images required.
* Image type first: Select `Yes` if the image type appears before the measurement number in the image file name. Select `No` if the measurement number appears before the image type.
* Maximum Runtime (hours): The amount of time to reserve a node for on the cluster. The actual runtime can be lower, but not larger than this. If the run takes longer than the time given, it will be terminated before completion. Must be less than 72.

Submit the pipeline by pressing `Analyze Images`. The plugin will copy the image files and the pipeline to the cluster and add the process to the queue.

### Checking run status
- Run Name: An identifier that allows you to recognise the pipeline and image batch.
- Number of images per measurement: If several image files are required for a single measurement, adjust this to the number of images required.
- Image type first: Select `Yes` if the image type appears before the measurement number in the image file name. Select `No` if the measurement number appears before the image type.
- Maximum Runtime (hours): The amount of time to reserve a node on the cluster. The actual runtime can be lower, but not larger than this. If the run takes longer than the time given, it will be terminated before completion. Must be less than 72.
- Project Code: Specify your Supercomputing Wales project code, for example, scw0123.
- Partition: Select the partition you wish to run your job on. (Defaults to `compute` partition)
- Local script directory: Choose where local copies of remote scripts and batch data are saved.

Open the ClusterView module in the Data Tools menu. You will see a list of all runs submitted to the cluster. Under the run name the module will display `PENDING` for runs in queue or currently running and `COMPLETED` for runs that have stopped running. Click `Update` in the upper left corner to refresh the status of the runs. Use the `Download Results` button to download and inspect the results.
If you have already downloaded the results, the button label will change to `Download Again`.
Submit the pipeline by pressing `Analyze Images`. The plugin will copy the image files and the pipeline to the cluster and add the process to the queue.

### Checking job status

Please add the `ClusterView` plugin from the `Data Tools` category in the Module list into the pipeline, then you will be able to select a list of all runs submitted to the cluster. Under the run name, the module will display `PENDING` for runs in the queue or currently running and `COMPLETED` for runs that have stopped running. Click the `Update` button to refresh the status of the runs. Use the `Download Results` button to download and inspect the results. If you have already downloaded the results, the button label will change to `Download Again`.

## TODO

- [ ] Light refactoring of ClusterView functions to separate directory management from Rynner functions for easier testing.
- [ ] Include testing for directory and file management functions in ClusterView.
- [ ] (Optional) Consider folding the ClusterView window into RunOnCluster as a single module.
- [ ] MacOS version
Loading