-
Notifications
You must be signed in to change notification settings - Fork 6
Data Delivery
As part of an initiative to promote good data stewardship practices for data providers and their customers, DukeDSClient integrates with a web service to facilitate data delivery through Duke Data Service.
We've designed a data delivery workflow to streamline the following activities:
- After uploading a dataset to DukeDS as a project, providers can share it in read-only mode with a recipient (
ddsclient share). For a core facility, the recipient will be a customer. - Recipients receive an email notification of the share, and can reply with feedback
- Upon finalization of the data set, providers can initiate delivery of the uploaded project to a recipient (
ddsclient deliver). - Recipients receive an email notification that the project is ready for delivery, and are directed to a website to accept or decline
- Upon acceptance, ownership of the project is transferred to the recipient. Upon decline, the user must provide a reason, and no ownership transfer takes place. In either case, email notifications are sent to the provider.
- Notification messages and user actions are recorded in a database.
These operations are driven by the data provider using the DukeDSClient command-line tool, but trigger email notifications through the Data Delivery Website. You can also monitor your deliveries by from the Data Delivery Website.
Data delivery is currently in pilot phase. Please contact [email protected] for more information.
DukeDSClient can share and transfer projects to other DukeDS users by using the share and deliver modes.
-
ddsclient share -p <Project Name> --user <NetID>: Gives a user read-only access to a project, and emails them a notification -
ddsclient deliver -p <Project Name> --user <NetID>: User receives an email with a link to accept or decline the transfer. When the user accepts, ownership of the project is transferred to the user. If you wish to deliver a copy of the project and retain the original add the--copyflag.
If delivering a project as a copy, it is up to the provider to archive (delete) their retained copy of the project if and when appropriate.
share and deliver both trigger notification emails from our system. We use a simple template system so that providers of data can customize the message and signature in these emails.
Prior to sharing or delivering data with ddsclient, you must have a set of email templates registered. Contact [email protected] for more information
Email templates use the jinja templating system, which allows simple variable substitution for data like the project's name and URL, but also flexible conditionals. The following templates are available for use in email templates:
| Variable Name | Description |
|---|---|
project_name |
Name of the project in Duke Data Service |
recipient_name |
Full name of the user receiving the share or delivery |
recipient_email |
Email address of the user receiving the share or delivery |
sender_email |
Email address of the user sending the share or delivery |
sender_name |
Full name of the user sending the share or delivery |
project_url |
URL of the project in Duke Data Service Web Portal |
accept_url |
URL where recipient can accept or decline project transfer (delivery only) |
process_type |
State of the delivery (e.g. "accept" or "decline", delivery status only) |
message |
Reason given by recipient if delivery was declined (delivery status only) |
user_message |
Additional information supplied by the sender when creating the share or delivery (see below) |
Email templates are designed to provide a consistent way to communicate with users, so most of the fields and text will be based on the common details above. In many cases though, it's common to add a note about a particular data set or delivery. To do this, we've added the user_message variable. It can be included in the email template, and provided by DukeDSClient when sharing or delivering.
Example email template:
Hello {{ recipient_name }},
You have been granted access to '{{ project_name }}'. This project can be viewed at {{ project_url }}.
{% if user_message %}
{{ sender_name }} also wanted to inform you that:
{{ user_message }}
{% endif %}
Regards,
{{ sender_name }}
{{ sender_email }}
Example command. The user message can be read from a text file, or interactively as shown below:
$ ddsclient share -p mouse --user msmith --msg-file -
Enter message and press CTRL-d when done:
We were unable to analyze sample 412, so it is not included in these results.
Sharing project.
...
Resulting share email:
Hello Mary Smith,
You have been granted access to 'Project ABC'. This project can be viewed at https://dataservice.duke.edu/#/project/faba-c3-de
Fred Jones also wanted to inform you that:
We were unable to analyze sample 412, so it is not included in these results.
Regards,
Fred Jones
[email protected]
Share and deliver require a valid config file, as described in Agent User Keys (setup)
Share and delivery use the email address associated with the user for notifications. So, if you deliver data from your NetID, the notification email will come from your email address and the acceptance notices will be sent there as well.
If you wish to use a service account, you must login to Duke Data Service with that account and download its keys.
Sharing is intended as the first step in a data delivery workflow. As the owner, you maintain ownership of the data, but grant (read-only) access to it. The project to be shared is identified by its name; project names can be listed using ddsclient list.
To share a project:
ddsclient share -p <Project Name> --user <username>
or
ddsclient share -p <Project Name> --email <[email protected]>
Specifying recipient by username (NetID for Duke users) is recommended, as sharing by email address will fail if the recipient has never logged into Duke Data Service.
When sharing a project, the recipient will be assigned the file_downloader role. To specify a different role, use the --auth_role flag. Roles can be listed with ddsclient list_auth_roles.
After sharing and reviewing the project, the owner can deliver it to the customer. Delivery removes access given by share, and notifies the customer that the delivery is waiting for their action.
To deliver a project:
ddsclient deliver -p <Project Name> --user <username>
or
ddsclient deliver -p <Project Name> --email <[email protected]>
Specifying recipient by username (NetID for Duke users) is recommended, as sharing by email address will fail if the recipient has never logged into Duke Data Service.
ddsclient deliver by default transfers the project to the recipient and you will be given download permissions. To deliver a copy of the project you can run a command like so:
ddsclient deliver -p project1 --user abc12 --copy
To filter files and directories when making the copy (e.g. to suppress a raw directory), you can use the --exclude or --include options:
ddsclient deliver -p project1 --user abc12 --copy --exclude raw
ddsclient deliver -p project1 --user abc12 --copy --include results