Releases: CanDIG/candigv2-ingest
Releases · CanDIG/candigv2-ingest
v4.2.2: hotfix
What's Changed
Full Changelog: v4.2.1...v4.2.2
Contributors
daisieh
Assets 2
v4.2.1: respawn daemon
Contributors
daisieh
Assets 2
v4.2.0: Performance fixes, site curator role
Async ingest, MoH v3, site curator role
What's Changed
- Update Stable to 2.1.0 by @OrdiNeu in #41
- v3.0.0: Refactoring, new authx by @daisieh in #55
- v4.0.0: Hg38, DateInterval, Genomic Ingest by @OrdiNeu in #78
- v4.1.0: Program and user authorizations by @daisieh in #95
- DIG-1653: S3 credential storage by @daisieh in #97
- Remove OPA_SECRET by @daisieh in #100
- Replace 'single quote' in PR titles by @mshadbolt in #102
- Bump requests from 2.31.0 to 2.32.2 by @dependabot in #99
- DIG-1635 : Allow ingest to be completed in batches per object by @SonQBChau in #103
- Update katsu_ingest.py by @SonQBChau in #104
- DIG-1658: Update commandline ingest instructions by @mshadbolt in #105
- DIG-1719: switch to candigv2-logging module by @daisieh in #106
- Upgrade to model 3 by @SonQBChau in #107
- Remove branch checkout from generate_test_data by @mshadbolt in #108
- Update requirements.txt by @daisieh in #109
- DIG-1720: clinical ingest is asynchronous by @daisieh in #110
- DIG-1762: switch to gunicorn by @daisieh in #112
- Remove commandline instructions from README by @mshadbolt in #111
- DIG-1721, DIG-1776, DIG-1777: async htsget ingest by @daisieh in #113
- DIG-1407: ingest validates against running katsu by @daisieh in #114
- hotfix: add back default warnings for site admin to endpoints by @daisieh in #116
- update clinical_etl by @mshadbolt in #117
- DIG-1712: Ingest should prevent ingest if program auth doesn't exist by @daisieh in #119
- Add an endpoint to echo back the user's refresh token. by @OrdiNeu in #118
- Add option to automatically delete existing tmp directory by @mshadbolt in #120
- Fixing auto delete by @mshadbolt in #121
- check to make sure the email isn't already in the role by @daisieh in #123
- Update ingest README to include site curator and program registration information by @mshadbolt in #122
- v4.2.0: Performance fixes, site curator role by @daisieh in #124
New Contributors
- @dependabot made their first contribution in #99
Full Changelog: v4.1.0...v4.2.0
Contributors
daisieh, OrdiNeu, and 3 other contributors
Assets 2
v4.1.0: Program and user authorizations
Highlights
- ProgramAuthorization endpoints
- UserReadAuthorization endpoints
- Site role endpoints
- test data updates
What's Changed
- DIG-1517: Create ProgramAuthorizationSchema and its endpoints by @daisieh in #80
- DIG-1549: Use private urls if running within a container by @daisieh in #82
- DIG-1543: better separation of results and errors by @daisieh in #83
- Extend timeout for ingest to be 60s by @OrdiNeu in #84
- Hotfix: Loosen requirement for awscli by @daisieh in #85
- DIG-1275: site admin role can be updated to a real user by @daisieh in #86
- Shift dates in test data by @mshadbolt in #87
- Add script to automatically generate synthetic data by @mshadbolt in #81
- DIG-1502, DIG-1496: user authorizations by @daisieh in #91
- Ingest updates for consistency new authx methods by @mshadbolt in #92
- Fix generate_test_data docs and add API payload example for Program authorization by @mshadbolt in #93
- Update htsget_ingest.py to use new site_admin method by @mshadbolt in #94
Full Changelog: v4.0.0...v4.1.0
Contributors
daisieh, OrdiNeu, and mshadbolt
Assets 2
v4.0.0: Hg38, DateInterval, S3 parsing
Summary of Changes
- Switch from hg37 to hg38 in the test data
- Change from dates to intervals
- Correct s3 URL parsing
What's Changed
- Switch hg37 to hg38 in the test data by @OrdiNeu in #54
- DIG-1471: DateInterval compatibility by @mshadbolt in #56
- DIG-1471: fix schema url by @mshadbolt in #57
- error message raises an error by @daisieh in #59
- add vim to container by @mshadbolt in #58
- Correct branch name for dispatch action by @mshadbolt in #60
- DIG-1482: Fix multi-program statistics bug by @mshadbolt in #61
- remove new lines by @mshadbolt in #62
- DIG-1376: add specific PR_NAME to dispatch action by @mshadbolt in #64
- DIG-1376: Add PR_NAME to title by @mshadbolt in #65
- DIG-1376: Informative PR name with github actions context by @mshadbolt in #66
- Improve automated PR titles and description by @mshadbolt in #67
- Update GitHub actions versions by @mshadbolt in #68
- Update github actions versions by @mshadbolt in #69
- Update to github-action-pr-expanded@v2 by @mshadbolt in #70
- Update GitHub action version by @mshadbolt in #71
- DIG-1459: Index files on ingest by @daisieh in #63
- Test update to latest version by @mshadbolt in #72
- DIG-1376 :Update GH Action to improve PR title and description by @mshadbolt in #73
- DIG-1463: Verify genomic files on ingest by @daisieh in #74
- Add reads test file to genomic_ingest by @daisieh in #75
- DIG-1490: Correct and improve s3 url parsing in ingest by @daisieh in #76
- Add a mapping for the multisample_2 sample ingested during integration tests by @OrdiNeu in #77
Full Changelog: v3.0.0...v3.1.0
Contributors
daisieh, OrdiNeu, and mshadbolt
Assets 2
v3.0.0: refactoring, new authx
Highlights
- HTSGet ingest significantly improved
- genomic ingest is validated
- dataset access adding/updating
- new authx
What's Changed
- DIG-1360: ingest can ingest clinical data from multiple programs by @daisieh in #40
- [Snyk] Fix for 9 vulnerabilities by @daisieh in #42
- Add testing for clinical ingest by @justin-ys in #16
- DIG-1357, DIG-1356, DIG-1272: testing and refactor of htsget ingest by @daisieh in #43
- Picking up Marion's suggestions from #43 by @daisieh in #44
- Improved README and identifier refactoring by @mshadbolt in #45
- Hotfix: don't try to ingest program if not valid by @daisieh in #46
- Add validation for genomic input format by @daisieh in #47
- adding endpoints for adding/removing user/program access by @daisieh in #48
- DIG-1388: update candig-ingest with new katsu endpoints by @daisieh in #49
- DIG-1219: import clinical_etl_code as a package instead of a submodule by @daisieh in #50
- DIG-1144: Update to Python 3.12 by @daisieh in #51
- DIG-1402: Update to new vault-based auth methods by @daisieh in #52
- hotfix: change genomic_file_id to match file names by @daisieh in #53
New Contributors
- @mshadbolt made their first contribution in #45
Full Changelog: v2.0.0...v3.0.0
Contributors
daisieh, mshadbolt, and justin-ys
Assets 2
v2.0.0: Ingest microservice
Summary
- Large change involving the move from multiple .json files to a singular .json file import, as well as the introduction of this repo as a microservice within CanDIG
What's Changed
- Ingest into katsu without auth enabled by @daisieh in #2
- add awscli to requirements by @kcranston in #3
- DIG-895: switch to using candigv2-authx module by @daisieh in #4
- DIG-931: update to match CanDIG/candigv2-authx#3 by @daisieh in #5
- moving settings.py to CanDIGv2 repo by @daisieh in #6
- Small tweaks for UX by @daisieh in #7
- Sonchau/moh ingest by @SonQBChau in #9
- Update README.md by @SonQBChau in #10
- Update moh_ingest.py by @SonQBChau in #11
- Sonchau/hotfix katsu url by @SonQBChau in #12
- Sonchau/hotfix katsu url by @SonQBChau in #13
- Sonchau/update 110 by @SonQBChau in #14
- Sonchau/rename moh katsu by @SonQBChau in #15
- DIG-1143: Sample DRS objects on htsget ingest by @daisieh in #17
- fix token timeout by @SonQBChau in #19
- Sonchau/delete datasets by @SonQBChau in #20
- change license to lgpl-3 by @kcranston in #21
- Validated clinical sample data by @daisieh in #22
- Update katsu_ingest.py by @SonQBChau in #23
- DIG-1188: Create microservice & API for ingest by @justin-ys in #18
- HTSGet stubs by @justin-ys in #26
- New API + schema for htsget/katsu ingest by @justin-ys in #27
- Add ETL validation by @justin-ys in #28
- DIG-1280: Detailed error codes for clinical ingest by @justin-ys in #29
- Remove Katsu Trailing Slash option by @justin-ys in #30
- Update README by @justin-ys in #31
- Add CanDIGv2 PR dispatch by @justin-ys in #32
- DIG-1223: clinical data ingest accepts metadata about cohorts by @daisieh in #33
- Sonchau/katsu ingest in batch by @SonQBChau in #34
- DIG-1305: Switch to debian by @daisieh in #35
- DIG-1342: Remove extraneous code from ingest by @daisieh in #36
- DIG-1346: only fail on validation errors, not warnings by @daisieh in #38
- DIG-1344: IDs only need to be unique within a schema by @daisieh in #37
New Contributors
- @SonQBChau made their first contribution in #9
- @justin-ys made their first contribution in #18
Full Changelog: v1.0.1...v2.0.0
Contributors
kcranston, daisieh, and 2 other contributors
Assets 2
Adding validation tools
No ingest changes, but added katsu_validate_dataset and env.sh.template.
Initial release
This version of ingest scripts were used for TFRI demo on July 7, 2022.