dccvalidator

dccvalidator is a package and Shiny app to perform data validation and QA/QC. It’s used in the AMP-AD and PsychENCODE consortia to validate data prior to data releases.

Installation

You can install dccvalidator from CRAN:

install.packages("dccvalidator")

To install the development version from GitHub, run:

devtools::install_github("Sage-Bionetworks/dccvalidator")

Many functions in dccvalidator use reticulate and the Synapse Python client. See the reticulate documentation for information on how to set R to use a specific version of Python if you don’t want to use the default Python installation on your machine. Whichever Python installation you choose should have synapseclient installed.

Because dccvalidator uses reticulate, it is not compatible with the synapser package.

Dockerize the App

Authentication

The dccmonitor can be authorized to log in to Synapse using Synapse Authentication (OAuth) client. Please view instructions here to learn how to request a client. Our OAuth clients were created using Synapse service accounts in order to enable multiple Sage employees to maintain the applications. In the Shared-SysBio LastPass folder, credentials for each client are recorded. In the notes section of the credentials (click on the entry > Edit to see notes), the service account used to create the client is noted.

Build a docker image using Dockerfile

docker build -t dccvalidator_1kd -f Dockerfile .  

Pull docker image from GitHub Container Registry

docker pull ghcr.io/sage-bionetworks/dccvalidator_1kd:v1.0.0

Create a container from the docker image

docker run --rm -it -p 8100:3838 -e APP_REDIRECT_URL=<APP_REDIRECT_URL> -e R_CONFIG_ACTIVE=1kD -e client_id=<Oauth client id> -e client_name=<Oauth client name> -e client_secret=<Oauth client secret> --name <container name> <dccvalidator_1kd or ghcr.io/sage-bionetworks/dccvalidator_1kd:v1.0.0>

Once the container is created, you can head to the APP_REDIRECT_URL you specified to enter the app.

Check data

dccvalidator provides functions for checking the following common data quality issues:

• Annotation keys and values conform to a controlled vocabulary
• Column names match those of an associated metadata template
• Certain columns are not empty
• Certain columns are complete
• Identifiers match between two metadata files (e.g. all individuals in one file are also present in another)
• Check that identifiers are unique within a file

Data submission validation

This package contains a Shiny app to validate manifests and metadata for AMP-AD studies. It uses the dccvalidator package to check for common data quality issues and gives realtime feedback to the data contributor on errors that need to be fixed. The reporting UI is heavily inspired by the MetaDIG project’s metadata quality reports.

The application also allows users to submit documentation of their study, a description of the methods used, etc.

See the customizing dccvalidator vignette for information on how to spin up a customized version of the application

Requesting New Features or Bug Fixes

For new feature requests or bug fixes, please create an issue to let the maintainers know. If you’ve found a bug, create an associated issue and illustrate the bug with a minimal reprex. If there is agreement that the problem exists or that a new feature is desired, then the issue will be triaged for future development based on priority.

Please note that the dccvalidator project is released with a Contributor Code of Conduct. By contributing to this project via issue creation, comment, or pull request, you agree to abide by its terms.

Contributing

See the Contributing Guide to see how you can get involved in the development of this application.