Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/zenodo uploader #214

Merged
merged 41 commits into from
Dec 22, 2023
Merged

Feature/zenodo uploader #214

merged 41 commits into from
Dec 22, 2023

Conversation

jsmatias
Copy link
Member

In summary the endpoint upload/datasets/{identifier}/zenodo was created to upload content to zenodo.
The main changes include:

  1. The folder uploader was renamed to uploaders and zenodo_uploader.py was created inside.
  2. Folder routers/uploader_routers was created and file upload_router_zenodo.py was created inside.
  3. For consistency, the uploader_router_hugginface.py file was moved inside and the required modifications were done:
    a. File uploader_router.py was created with The abstract class UploaderRouter. This was passed as a parent to UploadRouterZenodo, as well as to UploadRouterHuggingface. The latter was modified to inherit the type of the abstract class and avoid type error with the router_list in the src/routers/uploader_routers/__init__.py file.
    b. Minor changes were made to the main.py to include the new router_list
  4. Automated tests were created.

Because the zenodo workflow is a bit complex, I'll summarise it as follows:

  1. An empty resource is created or the metadata of an existing one is requested.
  2. Metadata is uploaded if necessary
  3. A file is uploaded to Zenodo
  4. The processes above can be repeated to upload more files to the same resource.
  5. Finally, the resource is published. Once this is done, no more files can be uploaded to the same resource.

 Changes to be committed:
	modified:   src/routers/uploader_routers/upload_router_zenodo.py
	new file:   src/tests/uploader/zenodo/mock_zenodo.py
	modified:   src/tests/uploader/zenodo/test_dataset_zenodo_uploader.py
	modified:   src/uploader/zenodo_uploader.py
 Changes to be committed:
upload_router_huggingface.py moved inside uploader_routers folder, and needed changes on the __init__ files and main.py were made.
	modified:   src/main.py
	modified:   src/routers/__init__.py
	new file:   src/routers/uploader_router.py
	modified:   src/routers/uploader_routers/__init__.py
	renamed:    src/routers/upload_router_huggingface.py -> src/routers/uploader_routers/upload_router_huggingface.py
	modified:   src/routers/uploader_routers/upload_router_zenodo.py
	modified:   src/tests/uploader/zenodo/test_dataset_zenodo_uploader.py
uploader folder renamed to uploaders after having zenodo_uploader file created
	renamed:    src/uploader/__init__.py -> src/uploaders/__init__.py
	renamed:    src/uploader/hugging_face_uploader.py -> src/uploaders/hugging_face_uploader.py
	renamed:    src/uploader/utils.py -> src/uploaders/utils.py
	renamed:    src/uploader/zenodo_uploader.py -> src/uploaders/zenodo_uploader.py
 Changes to be committed:
    Minor changes added to improve swagger documentation.
        modified:   src/routers/uploader_routers/upload_router_zenodo.py
    More tests added and code was cleaned up.
	modified:   src/tests/uploader/zenodo/mock_zenodo.py
	modified:   src/tests/uploader/zenodo/test_dataset_zenodo_uploader.py
    Publish option added as a paramenter.
	modified:   src/uploaders/zenodo_uploader.py
@jsmatias jsmatias added the enhancement New feature or request label Nov 29, 2023
Copy link

@josvandervelde josvandervelde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Jean!
Generally it looks good, no real problems. Just a lot of small comments.

src/routers/uploader_router.py Outdated Show resolved Hide resolved
src/routers/uploader_routers/upload_router_zenodo.py Outdated Show resolved Hide resolved
src/tests/uploader/zenodo/test_dataset_zenodo_uploader.py Outdated Show resolved Hide resolved
src/tests/uploader/zenodo/test_dataset_zenodo_uploader.py Outdated Show resolved Hide resolved
src/tests/uploader/zenodo/test_dataset_zenodo_uploader.py Outdated Show resolved Hide resolved
src/uploaders/zenodo_uploader.py Outdated Show resolved Hide resolved
src/uploaders/zenodo_uploader.py Outdated Show resolved Hide resolved
src/uploaders/zenodo_uploader.py Outdated Show resolved Hide resolved
src/uploaders/zenodo_uploader.py Outdated Show resolved Hide resolved
src/tests/uploader/zenodo/test_dataset_zenodo_uploader.py Outdated Show resolved Hide resolved
Copy link

@josvandervelde josvandervelde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Looks good - some new comments added

src/database/validators/zenodo_validators.py Show resolved Hide resolved
src/database/validators/zenodo_validators.py Outdated Show resolved Hide resolved
src/error_handlers/error_handlers.py Outdated Show resolved Hide resolved
src/uploaders/uploader.py Outdated Show resolved Hide resolved
src/uploaders/uploader.py Outdated Show resolved Hide resolved
src/tests/uploader/zenodo/test_dataset_uploader.py Outdated Show resolved Hide resolved
src/routers/uploader_routers/upload_router_zenodo.py Outdated Show resolved Hide resolved
src/uploaders/zenodo_uploader.py Outdated Show resolved Hide resolved
src/uploaders/zenodo_uploader.py Outdated Show resolved Hide resolved
 Conflicts:
	src/routers/uploader_routers/upload_router_huggingface.py

 On branch feature/zenodo_uploader
 Changes to be committed:
	modified:   README.md
	modified:   pyproject.toml
	modified:   src/authentication.py
	modified:   src/database/model/agent/contact.py
	modified:   src/database/model/agent/organisation.py
	modified:   src/database/model/concept/concept.py
	modified:   src/database/model/serializers.py
	modified:   src/main.py
	modified:   src/routers/resource_ai_asset_router.py
	modified:   src/routers/resource_router.py
	modified:   src/routers/search_router.py
	modified:   src/tests/routers/generic/test_authentication.py
	modified:   src/tests/routers/resource_routers/test_router_contact.py
	modified:   src/tests/testutils/default_sqlalchemy.py
Copy link

@josvandervelde josvandervelde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!
Some comments about the regex, and it's good to go

src/database/validators/zenodo_validators.py Outdated Show resolved Hide resolved
src/database/validators/zenodo_validators.py Outdated Show resolved Hide resolved
src/database/validators/zenodo_validators.py Outdated Show resolved Hide resolved
src/database/validators/zenodo_validators.py Outdated Show resolved Hide resolved
src/database/validators/zenodo_validators.py Show resolved Hide resolved
src/uploaders/uploader.py Outdated Show resolved Hide resolved
src/uploaders/uploader.py Outdated Show resolved Hide resolved
Conflicts (resolved):
	src/routers/resource_ai_asset_router.py

 Changes to be committed:
	modified:   pyproject.toml
	modified:   src/routers/resource_ai_asset_router.py
	modified:   src/tests/routers/ai_asset_routers/test_router_aiassets_retrieve_content.py
Copy link

@josvandervelde josvandervelde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, looks good!

@jsmatias jsmatias merged commit 392a30e into develop Dec 22, 2023
1 check passed
@andrejridzik andrejridzik deleted the feature/zenodo_uploader branch October 29, 2024 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants