[API] Add dicom upload #9154

maximemulder · 2024-03-20T20:25:24Z

Brief summary of changes

This is a port of #6161, which contains a description of the changes. Since the old PR has diverged quite a bit from the main branch, and that its author no longer works here, I found it easier to copy the changes on the current main branch and open a new PR rather than to rebase the PR. While porting, I only rewrote the old PR code whenever static analysis or practical stests showed that it was no longer in sync with the main branch (or when my IDE removed trailing spaces).

Testing instructions (if applicable)

Test the four new API routes, described in section 5 of the documentation of the API v0.0.4.

Notes for existing projects

This PR requires to run the following SQL patch SQL/New_patches/2024-03_19-MRIUploadServerProcessRel.sql.

maximemulder · 2024-03-20T20:33:26Z

I must say I am not sure what I am supposed to do regarding the failed tests on files that I did not modify in the PR. Any direction @driusan (or someone else) 🤔.

kongtiaowang · 2024-04-24T18:08:35Z

maximemulder · 2024-04-24T19:56:54Z

Hum, yeah, I already saw that.

I was just suprised to see that the errors are marked as originating from FilesDownloadHandler.php and SiteIDGenerator.php, which are not modified by this PR. I guess I must call these files from elsewhere and that the location reporting is just imprecise.

I'll tackle this tomorrow probably.

maximemulder · 2024-05-22T17:31:12Z

I've looked at the static errors in more details, it appears to be mainly due to an upgrade of phpstan, which changes the typing of string conversions with strval and (string) from mixed -> string to (bool|float|int|resource|string|null) -> string. I still don't know why this PR triggers the error and not other ones (it did trigger on my main branch though), but at least the errors are clear.

However, I find the code that triggers the errors to be quite cryptic, with some lines that seem obviously wrong (at least according to typing). If someone who knows this code better than me could look into it, it would be best. In the meantime, I'm adding this to the next meeting.

laemtl · 2024-05-28T15:33:12Z

I still don't know why this PR triggers the error and not other ones (it did trigger on my main branch though), but at least the errors are clear.

Probably because composer update was run, which triggered an update of the static checker.

maximemulder · 2024-05-28T18:12:02Z

Ok it seems the linting errors are ~~fixed~~, silenced. @laemtl had noticed that the errors were due to a composer update on my end. The errors should still appear in any PR that updates composer and should be fixed eventually. However, this can have its own issue and be another PR.

maximemulder · 2024-08-29T16:27:06Z

@ridz1208 Ok I finally finished updating and re-testing this PR. Generally it seems to work. However, if the pipeline fails during the latter stages, the DICOM archive previously inserted does not have its session initialized. This should be fixed with the new DICOM archive PR, but I don't expect it to be merged before at least a few weeks.

cmadjar

@maximemulder I did a first pass of review. I did not have a chance to actually test the changes as my MCIN sandbox is not letting me logging in. Have to figure out why and it could take some time so figured I would do an initial review.

modules/api/docs/LorisRESTAPI_v0.0.4-dev.md

cmadjar · 2024-09-03T20:10:03Z

modules/api/docs/LorisRESTAPI_v0.0.4-dev.md

+GET /candidates/$CandID/$VisitLabel/dicoms/$Tarname/processes
+```
+
+The response contains all `mri_upload` attempts with the specified `$tarname`. And for


technically, there should only be one UploadID per TarchiveID. So when you overwrite, you should grep the UploadID associated to what is being overwritten instead of inserting a new entry in mri_upload. Especially since the incoming directory content will be overwritten anyway.

I agree that there is only one MRI upload per DICOM archive, but I've seen a few cases in testing (even without this PR) where there are a few failed MRI uploads and one successful for a single file. This is how the batch uploader does it (don't know about other pipelines).

yeah. I think in principle, there should only be one upload per tarchive but unfortunately the batch upload script is not compliant with this set up. However, for the API, I think we should make sure this is the behaviour at minimum, to avoid duplicated entries.

Is this thread resolved?

modules/api/php/endpoints/candidate/visit/dicoms.class.inc

modules/imaging_browser/test/imaging_browserTest.php

php/libraries/DicomTar.class.inc

src/Data/Models/DicomSeriesDTO.php

driusan · 2024-09-04T16:51:58Z

Please remove all the unrelated changes (whitespace, variable name changes, etc). If necessary you can send those in another PR. It is not feasible to review this PR with so many unrelated changed mixed together.

maximemulder · 2024-09-17T20:18:42Z

I replied to all the comments and applied the suggested changes.

I reverted the breaking changes to the API including floats / strings and naming. Back when I wrote them, I thought it was part of the original PR and not something that was already in LORIS.

I made the terminology more consistent by using $VisitLabel everywhere in the DICOM part (section 5) of the API documentation.

I also disabled my trailing spaces auto-removal and removed the associated changes.

Ready for review @cmadjar @driusan

maximemulder · 2024-09-18T14:10:28Z

modules/api/docs/LorisRESTAPI.md

                    "SeriesDescription" : "MPRAGE_ipat2",
-                    "SeriesNumber" : "2",
+                    "SeriesNumber" : 2,


Regarding this change, the current documentation is inaccurate, I can extract this to a separate PR if wanted.

Example:
https://demo.loris.ca/api/v0.0.4-dev/candidates/587630/V1/dicoms

I need to look into this to see if it's a bug in the documentation or the code. It seems weird that it's only the SeriesNumber that does it and it might be related to PHP (not LORIS) changes in the PDO over time.

It might be faster to split it out.

cmadjar

@maximemulder I re-reviewed quickly your PR. A few additional comments:

In the description of the PR, could you add a "Notes for existing project" with the information of the SQL patches to run?

modules/api/docs/LorisRESTAPI_v0.0.4-dev.md

modules/api/php/endpoints/candidate/visit/dicoms.class.inc

driusan

various comments

driusan · 2024-11-07T15:06:55Z

modules/api/docs/LorisRESTAPI.md

+The JSON can optionally have a boolean `Overwrite` attribute that can be used
+to overwrite an existing file (`false` if not present).
+A successful request will be answered by a `303 See Other` response with its
+`Location` header pointing to the processes list of the new upload.


Shouldn't it be pointing to the new upload?

If an API user wanted the process list instead of the file itself after uploading a file, they could just append "/processes".

driusan · 2024-11-07T15:09:11Z

modules/api/docs/LorisRESTAPI.md


-Only `GET` is currently supported.
+To get a list of the processes and their status for a given DICOM study previously uploaded use the following:


This is a different endpoint, seems to be missing a subheader.

Added subheaders. Please tell me if there is anything wrong.

84b6398

EDIT: Well, I'll remove those I added to existing 0.0.3 endpoints then.
EDIT 2: Whoopsie, there are a few mistakes in the commit. I'll correct it.

Subheaders v2:

0c52124

driusan · 2024-11-07T15:10:42Z

modules/api/docs/LorisRESTAPI.md

                    "SeriesDescription" : "MPRAGE_ipat2",
-                    "SeriesNumber" : "2",
+                    "SeriesNumber" : 2,


I need to look into this to see if it's a bug in the documentation or the code. It seems weird that it's only the SeriesNumber that does it and it might be related to PHP (not LORIS) changes in the PDO over time.

It might be faster to split it out.

modules/api/docs/LorisRESTAPI.md

driusan · 2024-11-07T15:13:43Z

modules/api/docs/LorisRESTAPI_v0.0.4-dev.md

@@ -823,90 +823,192 @@ The JSON object is of the form:

 ## 5.0 DICOM Data

-Like the imaging data, the DICOM data mostly lives in the `/candidates/$CandID/$Visit` 
-portion of the REST API namespace, but is defined in a separate section of this 
+Like the imaging data, the DICOM data mostly lives in the `/candidates/$CandID/$VisitLabel`


I just realized my above comments were made on the 0.0.3 documentation which shouldn't be changed because it's already released. Pretend I made them on the v0.0.4-dev..

Isn't it ok to change the 0.0.3 documentation if it only clarifies the current behavior without changing it ? I would think it is ok personally.

Yes, clarifying the documentation is fine, changing the behaviour (incl adding new things) should only go into the dev version. An API client that uses version $x of the API should be able to expect the same behaviour if they point to a different LORIS instance.

So I am not sure of what you are asking me to do. Do you want the API DICOM upload endpoints to only be added to 0.0.4-dev ?

Yes. (And pretend that the comments I made on them in 0.0.3 before realizing I was looking at the wrong one were made on 0.0.4-dev)

Do you want me to duplicate the comments?

No need to duplicate the comments, I am going through each of them one by one anyway.

8230e89

This commit removes the documentation from the API v0.0.3. In practice, the new endpoints were already v0.0.4-only, but there are a few v0.0.4 POST routes that are added to v0.0.3 endpoints, not sure what do do here (ignore or implement method specific supported versions ?).

modules/api/php/views/visit/dicom/processes.class.inc

driusan · 2024-11-07T15:42:00Z

modules/server_processes_manager/php/mriuploadserverprocess.class.inc

@@ -52,25 +52,25 @@ class MriUploadServerProcess extends AbstractServerProcess
    private $_sourceLocation;

    /**
-     * Root of the MRI files directory.
+     * The value of MRICodePath config setting.


The changes to this file don't seem to be related to the API and should be moved to a different PR so someone more familiar with the server_processes_manager can review them.

src/Data/Models/MRIUploadDTO.php

src/Http/Response/JSON/Accepted.php

src/Http/Response/JSON/SeeOther.php

kongtiaowang closed this May 28, 2024

kongtiaowang reopened this May 28, 2024

maximemulder mentioned this pull request Aug 13, 2024

Draft architecture for LORIS API calls within LORIS-MRI Python aces/Loris-MRI#1148

Closed

maximemulder force-pushed the api-dicom-upload branch from fd349f1 to 222b5d4 Compare August 29, 2024 16:22

maximemulder force-pushed the api-dicom-upload branch 8 times, most recently from 4b8a6dc to 9765668 Compare August 29, 2024 18:04

maximemulder requested review from cmadjar and ridz1208 September 3, 2024 15:23

cmadjar requested changes Sep 3, 2024

View reviewed changes

maximemulder force-pushed the api-dicom-upload branch 2 times, most recently from 65869da to 4c63612 Compare September 17, 2024 20:15

maximemulder commented Sep 18, 2024

View reviewed changes

maximemulder force-pushed the api-dicom-upload branch 2 times, most recently from 6f882bc to 8112fc8 Compare September 23, 2024 15:28

cmadjar reviewed Sep 30, 2024

View reviewed changes

modules/api/docs/LorisRESTAPI_v0.0.4-dev.md Show resolved Hide resolved

modules/api/php/endpoints/candidate/visit/dicoms.class.inc Outdated Show resolved Hide resolved

maximemulder force-pushed the api-dicom-upload branch from 07a0ec2 to acd5bc8 Compare October 30, 2024 15:08

driusan mentioned this pull request Nov 7, 2024

[API] Adding dicom upload #6161

Closed

3 tasks

driusan requested changes Nov 7, 2024

View reviewed changes

maximemulder mentioned this pull request Nov 7, 2024

Add JSON HTTP responses for "accepted" and "see other" #9456

Merged

maximemulder added php Pull requests that update Php code Passed Manual Tests PR has undergone proper testing by at least one peer API PR or issue introducing/requiring modifications to the API code SQL PR contains SQL modifications such as schema changes and new SQL scripts labels Nov 7, 2024

maximemulder mentioned this pull request Nov 7, 2024

[server process monitor] minor doc comment changes #9457

Open

maximemulder added 6 commits November 7, 2024 13:54

api dicom upload

8c15276

apply suggestions

6d43a36

use correct api version in redirect

9999ffc

improve no file error message

b1a01d7

fix rebase

ca69f79

simplify overwrite

6c7af79

maximemulder force-pushed the api-dicom-upload branch from acd5bc8 to 6c7af79 Compare November 7, 2024 18:54

add property constructor

4d6eeb1

maximemulder mentioned this pull request Nov 7, 2024

[API] Fix API DICOM get documentation #9458

Open

$TarName

34d600e

maximemulder force-pushed the api-dicom-upload branch from 5660cf8 to 34d600e Compare November 7, 2024 19:38

maximemulder added 2 commits November 7, 2024 14:40

that is

5f327c7

remove 0.0.3 docs

8230e89

maximemulder force-pushed the api-dicom-upload branch from 84b6398 to 8230e89 Compare November 7, 2024 20:08

maximemulder added 2 commits November 7, 2024 15:13

subheaders v2

0c52124

error log

0488bb0

maximemulder force-pushed the api-dicom-upload branch from 6f3651d to 0488bb0 Compare November 7, 2024 20:36

maximemulder added 3 commits November 7, 2024 17:13

use snake_case for variables

3b6141a

fix process link

bfb2873

file size message

59437de

driusan self-requested a review November 11, 2024 18:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[API] Add dicom upload #9154

[API] Add dicom upload #9154

maximemulder commented Mar 20, 2024 •

edited

Loading

maximemulder commented Mar 20, 2024 •

edited

Loading

kongtiaowang commented Apr 24, 2024 •

edited

Loading

maximemulder commented Apr 24, 2024

maximemulder commented May 22, 2024

laemtl commented May 28, 2024 •

edited

Loading

maximemulder commented May 28, 2024

maximemulder commented Aug 29, 2024 •

edited

Loading

cmadjar left a comment

cmadjar Sep 3, 2024

maximemulder Sep 17, 2024

cmadjar Sep 30, 2024

driusan Nov 7, 2024

driusan commented Sep 4, 2024

maximemulder commented Sep 17, 2024 •

edited

Loading

maximemulder Sep 18, 2024 •

edited

Loading

driusan Nov 7, 2024

cmadjar left a comment

driusan left a comment

driusan Nov 7, 2024

driusan Nov 7, 2024

maximemulder Nov 7, 2024 •

edited

Loading

maximemulder Nov 7, 2024

driusan Nov 7, 2024

driusan Nov 7, 2024

maximemulder Nov 7, 2024

driusan Nov 7, 2024

maximemulder Nov 7, 2024

driusan Nov 7, 2024

driusan Nov 7, 2024

maximemulder Nov 7, 2024

maximemulder Nov 7, 2024

driusan Nov 7, 2024


		Only `GET` is currently supported.
		To get a list of the processes and their status for a given DICOM study previously uploaded use the following:

[API] Add dicom upload #9154

Are you sure you want to change the base?

[API] Add dicom upload #9154

Conversation

maximemulder commented Mar 20, 2024 • edited Loading

Brief summary of changes

Testing instructions (if applicable)

Notes for existing projects

maximemulder commented Mar 20, 2024 • edited Loading

kongtiaowang commented Apr 24, 2024 • edited Loading

maximemulder commented Apr 24, 2024

maximemulder commented May 22, 2024

laemtl commented May 28, 2024 • edited Loading

maximemulder commented May 28, 2024

maximemulder commented Aug 29, 2024 • edited Loading

cmadjar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

driusan commented Sep 4, 2024

maximemulder commented Sep 17, 2024 • edited Loading

maximemulder Sep 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmadjar left a comment

Choose a reason for hiding this comment

driusan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maximemulder Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maximemulder commented Mar 20, 2024 •

edited

Loading

maximemulder commented Mar 20, 2024 •

edited

Loading

kongtiaowang commented Apr 24, 2024 •

edited

Loading

laemtl commented May 28, 2024 •

edited

Loading

maximemulder commented Aug 29, 2024 •

edited

Loading

maximemulder commented Sep 17, 2024 •

edited

Loading

maximemulder Sep 18, 2024 •

edited

Loading

maximemulder Nov 7, 2024 •

edited

Loading