From 977a6aa579691caa76825c2d9caf412222690331 Mon Sep 17 00:00:00 2001 From: Adrienne Mendrik <79082794+AdrienneMendrik@users.noreply.github.com> Date: Mon, 15 Jan 2024 17:52:23 +0100 Subject: [PATCH 1/9] Update README.md Edited the introduction: Digital Trace Data Donation (Port) --- README.md | 24 ++++++------------------ 1 file changed, 6 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index d72a78fe..a595c5e2 100644 --- a/README.md +++ b/README.md @@ -1,27 +1,15 @@ # Feldspar -Integration mechanism for developers to build an extension that can be hosted on the Next platform. An example of this is Port, a software service program that enables individuals to donate their digital trace data. See documentation below. +Integration mechanism for developers to build an extension (flow application) that can be hosted on the Next platform. This is for example used in the Port program for data donation, as described below. -# Port: A frontend for data donation +## Digital Trace Data Donation (Port) +More information about the Port program can be found [here](https://eyra.notion.site/Port-Program-4bbf0bbc466547af95f05c609405c4b2?pvs=4). -Port is a research tool that enables individuals to donate their digital trace data for academic research in a secure, transparent, and privacy-preserving way. +Data donation allows researchers to invite participants to share their data download packages (DDPs). However, DDPs potentially contain very sensitive data and often not all data is needed to answer the specific research question under investigation. Feldspar enables researchers to extract only the data of interest through local processing (on the participants device). Participants can then inspect the extracted data after which they can consent (or decline) to donate. Feldspar creates a frontend that guides participants through the data donation steps. -Data donation allows researchers to invite participants to share their data download packages (DDPs). -A major challenge is however that DDPs potentially contain very sensitive data, and often not all data is needed to answer the specific research question under investigation. -To circumvent these challenges, an alternative framework was developed: +Feldspar is open-source under the AGPL license and allows researchers to configure their own data donation study. -1. The research participant requests their personal DDP at the platform of interest. -2. They download it onto their own personal device. -3. By means of local processing, only the features of interest to the researcher are extracted from that DDP. -4. The participant inspects the extracted features after which they can consent (or decline) to donate. - -To allow for the local processing (step 3) to take place, we developed the software Port. -Port creates a frontend that guides participants through the data donation steps. - -Port is open-source and allows for researchers to fully configure their own data donation study. - -_Note_: Port is only a frontend. In order for it to be used in a live study, it needs to be hosted with a server and integrate with a solution to store and retrieve the donated data. -This repository will discuss a some readily available options. +_Note_: Feldspar is only a frontend. In order for it to be used in a live study, it needs to be hosted on a server and connected to a storage to retrieve the donated data. ## Installation From 23a3937e41e0655fc76a57a9192ef7943c241968 Mon Sep 17 00:00:00 2001 From: Adrienne Mendrik <79082794+AdrienneMendrik@users.noreply.github.com> Date: Mon, 15 Jan 2024 18:04:36 +0100 Subject: [PATCH 2/9] Update README.md Small update to intro --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a595c5e2..d045d0c9 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ Integration mechanism for developers to build an extension (flow application) th ## Digital Trace Data Donation (Port) More information about the Port program can be found [here](https://eyra.notion.site/Port-Program-4bbf0bbc466547af95f05c609405c4b2?pvs=4). -Data donation allows researchers to invite participants to share their data download packages (DDPs). However, DDPs potentially contain very sensitive data and often not all data is needed to answer the specific research question under investigation. Feldspar enables researchers to extract only the data of interest through local processing (on the participants device). Participants can then inspect the extracted data after which they can consent (or decline) to donate. Feldspar creates a frontend that guides participants through the data donation steps. +Data donation allows researchers to invite participants to share their data download packages (DDPs). However, DDPs potentially contain very sensitive data and often not all data is needed to answer a specific research question. Feldspar enables researchers to extract only the data of interest through local processing (on the participants device). Participants can then inspect the extracted data after which they can consent (or decline) to donate. Feldspar creates a frontend that guides participants through the data donation steps. Feldspar is open-source under the AGPL license and allows researchers to configure their own data donation study. From f661fecded15c804b0b64c35b8d10b4312d07e33 Mon Sep 17 00:00:00 2001 From: Adrienne Mendrik <79082794+AdrienneMendrik@users.noreply.github.com> Date: Mon, 15 Jan 2024 18:15:06 +0100 Subject: [PATCH 3/9] Update README.md Small changes to the intro --- README.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index d045d0c9..b6897c05 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,16 @@ Integration mechanism for developers to build an extension (flow application) th ## Digital Trace Data Donation (Port) More information about the Port program can be found [here](https://eyra.notion.site/Port-Program-4bbf0bbc466547af95f05c609405c4b2?pvs=4). -Data donation allows researchers to invite participants to share their data download packages (DDPs). However, DDPs potentially contain very sensitive data and often not all data is needed to answer a specific research question. Feldspar enables researchers to extract only the data of interest through local processing (on the participants device). Participants can then inspect the extracted data after which they can consent (or decline) to donate. Feldspar creates a frontend that guides participants through the data donation steps. +Data donation allows researchers to invite participants to share their data download packages (DDPs). However, DDPs potentially contain very sensitive data and often not all data is needed to answer a specific research question. -Feldspar is open-source under the AGPL license and allows researchers to configure their own data donation study. +Feldspar enables researchers to: +- extract only the data of interest through local processing (on the participants device) using Python (Pyodide) +- prompt participants for questions about the data +- enable participants to inspect the extracted data before donation +- enable participants to delete table rows before donation +- consent or decline to donate the extracted data + +Feldspar is open-source under the AGPL license and allows researchers to configure the frontend that guides participants through the data donation steps. _Note_: Feldspar is only a frontend. In order for it to be used in a live study, it needs to be hosted on a server and connected to a storage to retrieve the donated data. From 26d71e98bffd9fef34f6f3223d0a9a0d3316aab6 Mon Sep 17 00:00:00 2001 From: Adrienne Mendrik <79082794+AdrienneMendrik@users.noreply.github.com> Date: Tue, 6 Feb 2024 15:18:29 +0100 Subject: [PATCH 4/9] Update README.md Added release notes --- README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/README.md b/README.md index b6897c05..6848149c 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,19 @@ In order to start a local instance of Port go through the following steps: If the installation went correctly you should be greeted with a mock data donation study. +## Release + +1. Create release file: + + ```sh + npm run release + ``` + +2. Use release file: + +The generated release.zip file can be installed on the Next platform. [screenshot Next of verwijzen naar andere file op mono] + + ## How to use Port A researcher can implement their own data donation flow by altering the Python script. From fdcef9468010d66c019e54aec3cc8d9ca37a8748 Mon Sep 17 00:00:00 2001 From: Adrienne Mendrik <79082794+AdrienneMendrik@users.noreply.github.com> Date: Mon, 12 Feb 2024 14:32:51 +0100 Subject: [PATCH 5/9] Update README.md Edited note in intro. --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 6848149c..429e47ea 100644 --- a/README.md +++ b/README.md @@ -16,11 +16,11 @@ Feldspar enables researchers to: Feldspar is open-source under the AGPL license and allows researchers to configure the frontend that guides participants through the data donation steps. -_Note_: Feldspar is only a frontend. In order for it to be used in a live study, it needs to be hosted on a server and connected to a storage to retrieve the donated data. +_Note_: Feldspar is only a frontend. In order for it to be used in a live study, it needs to be hosted on a server and connected to a storage to retrieve the donated data. To run a local instance see [installation](https://github.com/eyra/feldspar/tree/master?tab=readme-ov-file#installation). To create a release for the Next platform or the self hosted version, see [release](https://github.com/eyra/feldspar/tree/master?tab=readme-ov-file#release) ## Installation -In order to start a local instance of Port go through the following steps: +In order to start a local instance of Feldspar go through the following steps: 0. Pre-requisites @@ -33,7 +33,7 @@ In order to start a local instance of Port go through the following steps: 1. Install dependencies & tools: ```sh - cd ./port + cd ./feldspar npm install npm run prepare ``` From 64da2dbebc8abdbdf04c3f0d105b2d7fff3a0aab Mon Sep 17 00:00:00 2001 From: Adrienne Mendrik <79082794+AdrienneMendrik@users.noreply.github.com> Date: Mon, 12 Feb 2024 16:26:47 +0100 Subject: [PATCH 6/9] Update README.md Updated: How to use Feldspar? --- README.md | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 429e47ea..12de5aaa 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ _Note_: Feldspar is only a frontend. In order for it to be used in a live study, In order to start a local instance of Feldspar go through the following steps: -0. Pre-requisites +0. Prerequisites - Fork or clone this repo - Install [Node.js](https://nodejs.org/en) @@ -58,26 +58,25 @@ If the installation went correctly you should be greeted with a mock data donati 2. Use release file: -The generated release.zip file can be installed on the Next platform. [screenshot Next of verwijzen naar andere file op mono] +The generated release.zip file can be installed on the Next platform or the self-hosted version, by adding a "Donate task" and at "Flow application" select the generated zip-file. -## How to use Port +## How to use Feldspar? -A researcher can implement their own data donation flow by altering the Python script. -The Python scripts has 2 main functions: +You can implement your own data donation flow by altering the Python script, which can be used to: -1. It determines the data donation flow. i.e. what screens (for example a file prompt) does the participant gets to see and when. You can use the Port API ([`props.py`](src/framework/processing/py/port/api/props.py)) for this. -2. It determines what data gets extract from the participants submission. Here is were Python really shines, you can use most data extraction methods you are familiar with! (As long as it's available in [Pyodide](https://pyodide.org/en/stable/)) +1. customize the participant data donation flow in terms of screen content, type of screen (e.g. a file prompt) and screen order. You can use the Port API ([`props.py`](src/framework/processing/py/port/api/props.py)) for this. +2. extract specific data from the participant DDP that is required for the research question. You can use the data extraction methods that are available in [Pyodide](https://pyodide.org/en/stable/) A typical script includes the following steps: -1. Prompt the participant to submit a file -2. Handling the submission from step 1. This is the step where you can extract the data you are interested in. -3. The extracted data is presented on screen accompanied with a consent button. After consent is given, the data is sent to a storage location of the researcher. +1. Prompt the participant to select the DDP file +2. Extract the data of interest from the selected DDP file. Try to aggregate and anonymize as much as possible. +3. Present the extracted data on screen in clear tables to allow the participant to investigate the data that they are about to donate and buttons to choose to either donate or not. If a data storage is connected, the extracted data is stored only when participants agree to donate. -A example such a script is included in this repo: [`script.py`](src/framework/processing/py/port/script.py). -We recommend you use that script as starting point for your own data donation study. -You can find another example of such a script in this [repository](https://github.com/d3i-infra/port-d3i-pilot). +Example script: [`script.py`](src/framework/processing/py/port/script.py). + +We recommend to use the example script as starting point for your own data donation study. ### Port API examples From 4f0b436728d043889b321a3e0f9c365174971e53 Mon Sep 17 00:00:00 2001 From: Adrienne Mendrik <79082794+AdrienneMendrik@users.noreply.github.com> Date: Mon, 12 Feb 2024 17:07:42 +0100 Subject: [PATCH 7/9] Update README.md Adjusted "Port API examples" --- README.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 12de5aaa..f689bfac 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ _Note_: Feldspar is only a frontend. In order for it to be used in a live study, ## Installation -In order to start a local instance of Feldspar go through the following steps: +In order to start a local instance of Feldspar follow these steps: 0. Prerequisites @@ -72,7 +72,7 @@ A typical script includes the following steps: 1. Prompt the participant to select the DDP file 2. Extract the data of interest from the selected DDP file. Try to aggregate and anonymize as much as possible. -3. Present the extracted data on screen in clear tables to allow the participant to investigate the data that they are about to donate and buttons to choose to either donate or not. If a data storage is connected, the extracted data is stored only when participants agree to donate. +3. Present the extracted data on screen in clear tables to allow the participant to investigate the data that they are about to donate and buttons to choose to either donate or not (consent screen). If a data storage is connected, the extracted data is stored only when participants agree to donate. Example script: [`script.py`](src/framework/processing/py/port/script.py). @@ -80,7 +80,7 @@ We recommend to use the example script as starting point for your own data donat ### Port API examples -Below you can find examples on how to use the Port API in your `script.py` +Below some examples on how to use the Port API in your `script.py`
Main function @@ -100,7 +100,7 @@ def process(sessionId): yield CommandUIRender(page3) ``` -[`ScriptWrapper`](src/framework/processing/py/port/main.py) and [py_worker](src/framework/processing/py_worker.js) using `send` to iterate over the commands one by one. For more information on yield and Generators: https://realpython.com/introduction-to-python-generators +[`ScriptWrapper`](src/framework/processing/py/port/main.py) and [py_worker](src/framework/processing/py_worker.js) using `send` to iterate over the commands one by one. For more information on yield and Generators visit https://realpython.com/introduction-to-python-generators.
@@ -122,8 +122,8 @@ platform = "Twitter" progress = 25 file_input_description = props.Translatable({ - "en": f"Please follow the download instructions and choose the file that you stored on your device. Click “Skip” at the right bottom, if you do not have a {platform} file. ", - "nl": f"Volg de download instructies en kies het bestand dat u opgeslagen heeft op uw apparaat. Als u geen {platform} bestand heeft klik dan op “Overslaan” rechts onder." + "en": f"Please follow the download instructions and choose the file that you stored on your device.", + "nl": f"Volg de download instructies en kies het bestand dat u opgeslagen heeft op uw apparaat." }) allowed_extensions = "application/zip, text/plain" file_input = props.PropsUIPromptFileInput(file_input_description, allowed_extensions) @@ -132,7 +132,7 @@ file_input = props.PropsUIPromptFileInput(file_input_description, allowed_extens
-Create a consent page +Create consent tabels ```Python import pandas as pd @@ -161,7 +161,7 @@ consent_form = props.PropsUIPromptConsentForm(tables, meta_tables)
-Create donation page +Create donation screens ```Python header = props.PropsUIHeader(title) @@ -173,7 +173,7 @@ page = props.PropsUIPageDonation(platform, header, body, footer)
-Create page with radio buttons +Create user input screen with radio buttons ```Python header = props.PropsUIHeader(title) @@ -185,7 +185,7 @@ page = props.PropsUIPageDonation(platform, header, body, footer)
-Handling the result from a file input +Extract data from input file ```Python page = props.PropsUIPageDonation(platform, header, file_input, footer) @@ -197,8 +197,8 @@ if result.__type__ == "PayloadString": filename = result.value zipfile = zipfile.ZipFile(filename) - # Extract the data you are interested contained in zipfile - # Typically you will use your own written functions here + # Extract the data of interest from the selected file + # Write your own functions for data extraction ... else: # No file selected @@ -207,7 +207,7 @@ else:
-Handling consent result +Handle user consent input ```Python platform = "Twitter" @@ -220,7 +220,7 @@ if result.__type__ == "PayloadJSON": # User gave consent yield CommandSystemDonate(donation_key, result.value) else: - # User declined or skipped + # User declined ```
From 9f165b6599ccbe228e35a91b052761e6e8bb0a33 Mon Sep 17 00:00:00 2001 From: Adrienne Mendrik <79082794+AdrienneMendrik@users.noreply.github.com> Date: Mon, 12 Feb 2024 17:23:56 +0100 Subject: [PATCH 8/9] Update README.md Data donation as a service --- README.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index f689bfac..5fbcfacf 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ Feldspar enables researchers to: Feldspar is open-source under the AGPL license and allows researchers to configure the frontend that guides participants through the data donation steps. -_Note_: Feldspar is only a frontend. In order for it to be used in a live study, it needs to be hosted on a server and connected to a storage to retrieve the donated data. To run a local instance see [installation](https://github.com/eyra/feldspar/tree/master?tab=readme-ov-file#installation). To create a release for the Next platform or the self hosted version, see [release](https://github.com/eyra/feldspar/tree/master?tab=readme-ov-file#release) +_Note_: Feldspar is only a frontend. In order for it to be used in a live study, it needs to be hosted on a server and connected to a storage to retrieve the donated data. To run a local instance see [installation](https://github.com/eyra/feldspar/tree/master?tab=readme-ov-file#installation). To create a release for the Next platform or the self hosted version, see [release](https://github.com/eyra/feldspar/tree/master?tab=readme-ov-file#release). ## Installation @@ -238,16 +238,18 @@ yield CommandSystemDonate(tracking_key, data) -## Use Port in a data donation study +## Use Feldspar in a data donation study -Port serves as the frontend, providing the application with which participants -engage. It defines the flow and logic for data donation. To utilize Port in a +Feldspar serves as the frontend, providing the application with which participants +engage. It facilitates the flow and logic for data donation. To utilize Feldspar in a data donation study, it must be hosted on a server capable of storing the -donated data. +donated data. -You can host Port by embedding it in an +You can host Feldspar on the Next platform or the self-hosted version as explained [here](https://github.com/eyra/feldspar/tree/master?tab=readme-ov-file#release). + +Alternatively, you can host Feldspar by embedding it in an [iframe](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe). -After the iframe loads, send a message that includes a channel. The Port +After the iframe loads, send a message that includes a channel. The Feldspar application will use this channel to relay messages with data ready for storage. Here's a JavaScript example: @@ -255,7 +257,7 @@ Here's a JavaScript example: // ... wait until the iframe is loaded const channel = new MessageChannel(); channel.port1.onmessage = (e) => { - console.log("Message receive from Port app", e); + console.log("Message receive from Feldspar app", e); }; // get the iframe via querySelector or another method iframe.contentWindow.postMessage("init", "*", [this.channel.port2]); @@ -263,8 +265,7 @@ iframe.contentWindow.postMessage("init", "*", [this.channel.port2]); ### Data donation as a service -Data donation sounds intriguing but seems like a lot to handle? Don't hesitate -to reach out to [Eyra](https://eyra.co/). +Would you like to get support with setting up your data donation study or host your data donation study on the Next platform? Reach out to Eyra for custom pricing: [connect@eyra.co](mailto:connect@eyra.co?subject='Data donation pricing request'). ### Previous studies done with Port From c2121c91f677025ba1503f9477dfed187e9645a7 Mon Sep 17 00:00:00 2001 From: Adrienne Mendrik <79082794+AdrienneMendrik@users.noreply.github.com> Date: Mon, 12 Feb 2024 18:34:18 +0100 Subject: [PATCH 9/9] Update README.md Edited last sections --- README.md | 35 ++++++++++------------------------- 1 file changed, 10 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 5fbcfacf..8f34face 100644 --- a/README.md +++ b/README.md @@ -267,20 +267,18 @@ iframe.contentWindow.postMessage("init", "*", [this.channel.port2]); Would you like to get support with setting up your data donation study or host your data donation study on the Next platform? Reach out to Eyra for custom pricing: [connect@eyra.co](mailto:connect@eyra.co?subject='Data donation pricing request'). -### Previous studies done with Port +### Example studies -Here you can find a [list](https://github.com/eyra/port/wiki/Previous-data-donation-studies) of previous studies that were completed using Port. These studies can serve as inspiration or example for your own data donation study! +The feldspar repository was previously named port. This repository is now depricated. However, you can check out the [list](https://github.com/eyra/port/wiki/Previous-data-donation-studies) of example studies that were performed using various versions of the port repository as inspiration for your data donation study. -# Technical specifications of Port +# Technical specifications of Feldspar -Port is ready for use out of the box (just update the Python script). However, -if your study requires specific adjustments (new interactive elements etc.), you -have the flexibility to modify Port's functionalities. If customization is what -you're after, leverage the following technical insights to suit your needs. +If your study requires specific adjustments (new interactive elements etc.), you +have the flexibility to modify the Feldspar functionalities. Leverage the following technical insights to suit your needs. ## Data model -Port uses the following data model (also see: [src/framework/types](src/framework/types)) +Feldspar uses the following data model (also see: [src/framework/types](src/framework/types)) - [Modules](src/framework/types/modules.ts) @@ -356,7 +354,7 @@ See: [src/framework/processing/py/port](src/framework/processing/py/port) ## Code instructions -These instructions give you some pointers on things you might like to change or add to Port. +These instructions give you some pointers on things you might like to change or add to Feldspar.
Change copy (texts shown on the web pages) @@ -670,22 +668,9 @@ The project uses [ts-standard](https://github.com/standard/ts-standard) for mana Before committing to github [Husky](https://github.com/typicode/husky) runs all the necessary scripts to make sure the code conforms to `ts-standard`, all the tests run green, and the `dist` folder is up-to-date. -## Digital Data Donation Infrastructure (D3I) +## Funding -Port is funded by the PDI-SSH and is a collaboration between six Dutch universities and Eyra. - -The consortium is composed of researchers from: - -- University of Amsterdam -- Radboud University Nijmegen -- VU Amsterdam -- Utrecht University -- Tilburg University -- Erasmus University Rotterdam - -### D3I Pilot - -The first phase of the project ended in December 2022 and resulted in an MVP solution to run one Port app on top of a Next bundle. This Next + Port combi can be released as a Docker image and deployed on [Azure Web App Service](https://azure.microsoft.com/en-us/products/app-service/web). +Feldspar is part of the Port program for data donation and has been funded by the UU, PDI-SSH ([D3i project](https://datadonation.eu/)), and Eyra. # Contributing @@ -696,5 +681,5 @@ We want to make contributing to this project as easy and transparent as possible - Submitting a fix - Proposing new features -If you have any questions, find any bugs, or have any ideas, read how to contribute [here](https://github.com/eyra/port/blob/master/CONTRIBUTING.md). +If you have any questions, find any bugs, or have any ideas, read how to contribute [here](https://github.com/eyra/feldspar/blob/master/CONTRIBUTING.md).