depmap_omics

This repository contains code that processes data for the biannual DepMap data release. State of the pipeline for each release can be found under the "Releases" tab in this repo.

Table of Contents

• Getting Started
  • Installation
• Repository File Structure
• Running the Pipeline
  • Uploading and Preprocessing
  • Running Terra Pipelines
  • Downloading and Postprocessing
  • QC, Grouping and Uploading

Getting Started

The processing pipeline relies on the following tools:

• python
• R
• jupyter
• WDL
• gcp
• docker
• Terra
• The Terra Convention: The dos and donts for maintaining a cleaner terra.
• dalmatian

Installatiion

git clone http://github.com/BroadInstitute/depmap_omics.git && cd depmap_omics

pip install -e .

⚠️ This repository needs other repos

Some important data and code from the genepy Library.

Use the instructions in the genepy page to install the package.

⚠️ You need the following R and python packages

1. You will need to install jupyter notetbooks and google cloud sdk

• install Google Cloud SDK.
• authenticate my SDK account by running gcloud auth application-default login in the terminal, and follow the instrucion to log in.

2. and GSVA for ssGSEA in R R run R -e 'if(!requireNamespace("BiocManager", quietly = TRUE)){install.packages("BiocManager")};BiocManager::install(c("GSEABase", "erccdashboard", "GSVA", "DESeq2"));'

3. For Python use the requirements.txt file pip install -r requirements.txt

⚠️ Follow instructions here to set up Terra and obtain access to services required for running the pipeline.

Repository File Structure

ccle_tasks/ Contains a notebook for each of the different additional processing that the CCLE team has to perform as well as one-off tasks run by the omics team

data/ Contains important information used for processing, including terra workspace configurations from past quarters

depmapomics/ Contains the core python code used in the pipeline and called by the processing jupyter notebooks

*_pipeline/ Contains some of the workflows' wdl files and script files used by these workflows

temp/ Contains the temp file that can get removed after processing (should be empty)

documentation/ Contains some additional files and diagrams for documenting the pipelines

tests/ Contains automated pytest functions used internally for development

jupyter notebooks: RNA_CCLE.ipynb contains the DepMap processing pipelines for Expression and Fusion (from RNAseq data), and WGS_CCLE.ipynb contains the DepMap processing pipelines for Copy number and Mutations (from WGS/WES data)

Pipeline Walkthrough

The processing pipelines are encapsulated in two jupyter notebooks (RNA_CCLE.ipynb and WGS_CCLE.ipynb). Each is divided into four steps: uploading, running Terra pipelines, local postprocessing, and uploading. Here is a detailed walkthrough (Note that the steps that are "internal only" are run as part of DepMap's data processing, but not meant for external users to reproduce due to various dependencies that are unique to our team at the Broad. The "internal only" functions below can be found in the depmap_omics_upload repo):

1. Uploading and Preprocessing (internal only)

Currently, sequenced data for DepMap is generated by the Genomics Platform (GP) at the Broad who deposits them into several different Terra workspaces. Therefore, the first step of this pipeline is to look at these workspaces and

• identify new samples by looking at the bam files and compare them with bams we have already onboarded
• remove duplicates and ones with broken file paths
• onboard new samples and new versions of old cell lines if we find any

2. Running Terra Pipelines

We are using Dalmatian to send requests to Terra, so before running this part, external users need to make sure that the dalmatian WorkspaceManager object is initialized with the right workspace and that the functions are taking the correct workflow names as inputs. You can then run the RNAseq and/or WGS pipelines on your samples.

For a more in-depth documentation on what our pipelines contain, including the packages, input references, and parameters, please refer to this summary of DepMap processing pipeline.

3. Downloading and Postprocessing (sections under on local in the notebooks)

This step will do a set of tasks:

• clean the workspaces by deleting large useless files, including unmapped bams.
• retrieve from the workspace interesting QC results.
• copy realigned bam files to our own data storage bucket (internal only).
• download the outputs from Terra pipelines.

The main postprocessing steps for each pipeline are as followed:

Copy Number

copynumbers.py contains the main postprocessing function postProcess() responsible for postprocessing segments and creating gene-level (relative and absolute) CN files and genomic feature table. Gene mapping information is retrieved from BioMart version nov2020. The function also applies the following filters to segment and CN data:

• Remove chrY segments from cell lines where their chrY segment count is bigger than 150
• Mark samples that have more than 1500 segments as QC fails and remove them
• Remove genes whose Entrez ID is NaN in BioMart in the gene-level matrices

Internal only: dm_omics.cnPostProcessing() calls the above function on both WES and WGS data, merges them, renames the indices into ProfileIDs, and upload them to taiga.

Note: to get the exact same results as in DepMap, be sure to run genecn = genecn.apply(lambda x: np.log2(1+x)) to the genecn dataframe in the CNV pipeline

Mutation

mutations.py contains postProcess(), the function responsible for postprocessing aggregated MAF files, genotyped mutation matrices (hot spot and damaging), binary guide mutation matrices, and structural variants (SVs).

Internal only: dm_omics.mutationPostProcessing() calls the above function on both WES and WGS data, merges them, renames the indices into ProfileIDs, removes genes whose hugo symbol is not in biomart, generates individual mutation datasets for variant types, and uploads them to taiga. It also generates and uploads a binary matrix for germline mutations.

Expression

expressions.py contains the main postprocessing function responsible for postprocessing aggregated expression data from RSEM, which removes duplicates and QC failures, renames genes, filters and log transforms values, and generates transcrip-level, gene-level, and protein-coding gene-level expression data files. Gene mapping information is retrieved from BioMart version nov2020. Optionally, in addition, it also generates Single-sample GSEA (ssGSEA) data.

Internal only: dm_omics.expressionPostProcessing() is a wrapper for the above function. It renames the indices into ProfileIDs and uploads the files to taiga.

Fusion

Functions that postprocess aggregated fusion data can be found in fusions.py. We want to apply filters to the fusion table to reduce the number of artifacts in the dataset. Specifically, we filter the following:

• Remove fusions involving mitochondrial chromosomes, or HLA genes, or immunoglobulin genes
• Remove red herring fusions (from STAR-Fusion annotations column)
• Remove fusions recurrent in CCLE (>= 25 samples)
• Remove fusions that have (SpliceType=" INCL_NON_REF_SPLICE" AND LargeAnchorSupport="No" AND FFPM < 0.1)
• Remove fusions with FFPM < 0.05 (STAR-Fusion suggests using 0.1, but looking at the translocation data, this looks like it might be too aggressive)

Internal only: dm_omics.fusionPostProcessing() is a wrapper for the above function. It renames the indices into ProfileIDs and uploads the data to taiga.

4. QC, Grouping and Uploading to the Portal (internal use only)

We then perform the following QC tasks for each dataset:

CN

Once the CN files are saved, we load them back in python and do some validations, in brief:

• mean, max, var...
• to previous release: same mean, max, var...
• checkAmountOfSegments: flag any samples with a very high number of segments

Mutation

Compare to previous release (broad only)

We compare the results to the previous releases MAF. Namely:

• Count the total number of mutations per cell line, split by type (SNP, INS, DEL)
• Count the total number of mutations observed by position (group by chromosome, start position, end position and count the number of mutations)

REMARK:

Overall the filters applied after the CGA pipeline are the following:

We remove everything that:

• has AF<.1
• OR coverage <4
• OR alt cov=1
• OR is not in coding regions
• OR is in Exac with a frequency of >0.005%
  • except if it is either
    • in TCGA > 3 times
    • OR in Cosmic > 10 times
  • AND in a set of known cancer regions.
• OR exist in >5% of the CCLE samples
  • except if they are in TCGA >5 times

RNA

Once the expression files are saved, we do the following validations:

• mean, max, var...
• comparison to previous release: same mean, max, var...
• we QC on the amount of genes with 0 counts for each samples

After QC, data is uploaded to taiga for all portal audiences according to release dates in Gumbo.

@jkobject @gkugener @gmiller @5im1z @__BroadInsitute

If you have any feedback or run into any issues, feel free to post an issue on the github repo.