song-1.3.0

Summary

Bug Fixes

• validate saveStudy endpoint
• validate that the filename field in the file section of the payload is a basename (no directories)

New Features

• /schema endpoint to retireve schema info for analyses
• absolute pathname now used in manifest outputs for the java client, go client and python sdk
• /unpublish endpoint to unpublish analyses that are published
• added analysisStates url parameter to the GetAnalysisForStudy endpoint to filter by a set of analysisStates
• study is now a requirement in the payload. Reduces accidental uploads to the wrong study
• validate the studyId url parameter of SaveStudy endpoint with the studyId of the payload
• analysisId revoke mechanism. AnalysisIds are committed to the id server only on success

Performance

• improved GetAnalysisForStudy performance. LIRI-JP down from 50min to 2min.

Misc

• fixed vulnerbilities flagged by github
• upgraded dcc-id dependency to 5.2.0

DB Migration

1. Save the following sql script to a file called song-1.3.0-migration.sql

CREATE VIEW FullView AS
SELECT
       A.id as analysis_id,
       A.study_id as study_id,
       A.type as analysis_type,
       A.state as analysis_state,
       I_A.info as analysis_info,
       F.id as file_object_id,
       F.access as file_access,
       F.md5 as file_md5,
       F.name as file_name,
       F.size as file_size,
       F.type as file_type,
       I_F.info as file_info,
       SA.id as sample_id,
       SA.type as sample_type,
       SA.submitter_id as sample_submitter_id,
       I_SA.info as sample_info,
       SP.id as specimen_id,
       SP.type as specimen_type,
       SP.class as specimen_class,
       SP.submitter_id as specimen_submitter_id,
       I_SP.info as specimen_info,
       D.id as donor_id,
       D.gender as donor_gender,
       D.submitter_id as donor_submitter_id,
       I_D.info as donor_info,
       I_SR.info as sequencingread_info,
       I_VC.info as variantcall_info
FROM Analysis A
       left join Info I_A on I_A.id = A.id and I_A.id_type = 'Analysis'
       left join File F on F.analysis_id = A.id
       left join Info I_F on I_F.id = F.id and I_F.id_type = 'File'
       left join SampleSet SS on SS.analysis_id = A.id
       left join Sample SA on SA.id = SS.sample_id
       left join Info I_SA on I_SA.id = SA.id and I_SA.id_type = 'Sample'
       left join Specimen SP on SP.id = SA.specimen_id
       left join Info I_SP on I_SP.id = SP.id and I_SP.id_type = 'Specimen'
       left join Donor D on D.id = SP.donor_id
       left join Info I_D on I_D.id = D.id and I_D.id_type  = 'Donor'
       left join Info I_VC on I_VC.id = A.id and I_VC.id_type = 'VariantCall'
       left join Info I_SR on I_SR.id = A.id and I_SR.id_type = 'SequencingRead';

2. Run the sql script on the existing song database

PGPASSWORD=<dbpasswd>  psql  -h <dbhostname>  -p <dbport>  -U <dbuser>  song < song-1.3.0-migration.sql

3. Test that the migration worked. There should be no error

PGPASSWORD=<dbpasswd>  psql  -h <dbhostname>  -p <dbport>  -U <dbuser>  song -c 'select * from fullview limit 5;'

Downloads

Song-Client-1.3.0 Distribution

Song-Server-1.3.0 Distribution

Docker Compose

song-docker-1.3.0

git clone --branch song-docker-1.3.0 https://github.com/overture-stack/SONG.git  song-docker-1.3.0

Python SDK

overture-song=1.3.0

pip install overture-song==1.3.0

Details

Full Changelog

Implemented enhancements:

• add get-json-schema feature endpoint #373
• Update manifest command in song-go-client to match song-client #349
• Support unpublishing analysis object #346
• Optimize GetAnalysisForStudy to be faster #338
• Assign collab domain and letsencrypt cert to half-song #330
• Update dcc-repository with override flag for indexing UNPUBLISHED data #317
• Add 'study' field in SONG payload and make it required #308
• Request param in saveStudy endpoint not validated with payload #300
• Feature/song 308 add study id payload #352 (rtisma)
• Feature/song 338 get analyses optimize #342 (rtisma)

Fixed bugs:

• COLLAB info id doesnt exist #358
• AWS null pointer for info field error #356
• Fix readthedocs compile error #334
• GetAnalysesForStudy endpoint should only return PUBLISHED analyses #325
• Add analysisId revoke for when save fails but id was already created #321
• Error about Info record for Donor not found #318
• Value for fileName field must contain file basename only #311
• Request param in saveStudy endpoint not validated with payload #300

Closed issues:

• Export endpoint includes analysisState #366
• Add go-client building with maven #363
• Fix vulnerabilities flagged by GitHub #360
• help #359
• Create a HELM package for SONG #354
• Assist in update to XMLs in production #351
• Create SONG sandbox environments #340
• Write documentation about how to deploy an app (song-pdc) on Kubernetes #336
• Create Postgres docker image that has schema built in #335
• Create Half-Song for PDC #316
• Change group name from dcc to overture #271
• New feature requests or adjustments to existing behaviour after first experience using SONG in workflow batch processing #241
• In SONG analysis submission payload, allow user to specify access control at file level #141

Merged pull requests:

• Feature/song 373 json schema endpoint #375 (rtisma)
• chore: Updated requests to 2.20.0 due to vulnerbilities #369 (rtisma)
• fix: Removed analysisState from export ouptut #367 (rtisma)
• Song 321 revoke analysis #364 (rtisma)
• fix: Fixed null pointer issue with info fields #357 (rtisma)
• Update go manifest command #355 (d8660091)
• Support unpulishing analysis object #350 (d8660091)
• song-338-added batching #348 (rtisma)
• Implement new song-server endpoints #4 #345 (d8660091)
• Validate study payload to prevent mismatch study ids #344 (d8660091)
• Move go client into monorepo #343 (d8660091)
• Feature/song 311 file basename only #333 (rtisma)
• added longer default retry for the id client #332 (rtisma)
• Renaming org.icgc.dcc to bio.overture #331 (rtisma)
• Added url param for getAnalyseseByStudy so can filter by analysisState #328 (rtisma)