Notebooks/Methods-1-Retrieve published HiC datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Retrieve-HiC-dataset-from-NCBI\" data-toc-modified-id=\"Retrieve-HiC-dataset-from-NCBI-1\">Retrieve HiC dataset from NCBI</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Compression\" data-toc-modified-id=\"Compression-1.0.1\">Compression</a></span></li><li><span><a href=\"#Cleanup\" data-toc-modified-id=\"Cleanup-1.0.2\">Cleanup</a></span></li></ul></li></ul></li><li><span><a href=\"#References\" data-toc-modified-id=\"References-2\">References</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retrieve HiC dataset from NCBI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use data from <a name=\"ref-1\"/>[(Stadhouders R, Vidal E, Serra F, Di Stefano B et al. 2018)](#cite-ralph), which comes from mouse cells where Hi-C experiment where conducted in different states during highly-efficient somatic cell reprogramming.\n",
    "\n",
    "The data can be downloaded from:\n",
    "\n",
    "https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE53463\n",
    "\n",
    "Once downloaded the files can be converted to the FASTQ format in order for TADbit to read them.\n",
    "\n",
    "The easiest way to download the data might be through the `fastq-dump` program from the SRA Toolkit (http://www.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?cmd=show&f=software&m=software&s=software).\n",
    "\n",
    "We download 100M reads for each of 4 replicates (2 replicates from B cells and 2 from Pluripotent Stem Cells),and organize each in two files, one per read-end (this step is long and can take __up to 6 hours__):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read 100000000 spots for SRR5344921\n",
      "Written 100000000 spots for SRR5344921\n",
      "Read 100000000 spots for SRR5344925\n",
      "Written 100000000 spots for SRR5344925\n",
      "Read 100000000 spots for SRR5344969\n",
      "Written 100000000 spots for SRR5344969\n",
      "Read 100000000 spots for SRR5344973\n",
      "Written 100000000 spots for SRR5344973\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "mkdir -p FASTQs\n",
    "\n",
    "fastq-dump -A SRR5344921 -DQ '+' --defline-seq '@$ac.$si' -X 100000000 --split-files --outdir FASTQs/\n",
    "mv FASTQs/SRR5344921_1.fastq FASTQs/mouse_B_rep1_1.fastq\n",
    "mv FASTQs/SRR5344921_2.fastq FASTQs/mouse_B_rep1_2.fastq\n",
    "\n",
    "fastq-dump -A SRR5344925 -DQ '+' --defline-seq '@$ac.$si' -X 100000000 --split-files --outdir FASTQs/\n",
    "mv FASTQs/SRR5344925_1.fastq FASTQs/mouse_B_rep2_1.fastq\n",
    "mv FASTQs/SRR5344925_2.fastq FASTQs/mouse_B_rep2_2.fastq\n",
    "\n",
    "fastq-dump -A SRR5344969 -DQ '+' --defline-seq '@$ac.$si' -X 100000000 --split-files --outdir FASTQs\n",
    "mv FASTQs/SRR5344969_1.fastq FASTQs/mouse_PSC_rep1_1.fastq\n",
    "mv FASTQs/SRR5344969_2.fastq FASTQs/mouse_PSC_rep1_2.fastq\n",
    "\n",
    "fastq-dump -A SRR5344973 -DQ '+' --defline-seq '@$ac.$si' -X 100000000 --split-files --outdir FASTQs/\n",
    "mv FASTQs/SRR5344973_1.fastq FASTQs/mouse_PSC_rep2_1.fastq\n",
    "mv FASTQs/SRR5344973_2.fastq FASTQs/mouse_PSC_rep2_2.fastq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Files are renamed for convenience.\n",
    "\n",
    "_Note: the parameter used here for fastq-dump are for generating simple FASTQ files, -DQ ‘+’ tells to put a single plus sign as header of the quality input, `--defline-seq ‘@$ac.$si’` reduces the information in the headers to the accession number and the read id, `--split-files` is to separate both read-ends in different files, finally `-X 100000000` is to download only the first 100 Million reads of each replicate_\n",
    "\n",
    "_Note: alternatively you can also directly download the FASTQ from http://www.ebi.ac.uk/_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of these 8 files, contains 100M reads of 75 nucleotides each, and occupies ~17 Gb (total 130 Gb).\n",
    "\n",
    "Internally we use DSRC <a name=\"ref-4\"/>[(Roguski and Deorowicz, 2014)](#cite-roguski2014dsrc) that allows better compression ration and, more importantly, faster decompression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "dsrc c -t8 FASTQs/mouse_B_rep1_1.fastq FASTQs/mouse_B_rep1_1.fastq.dsrc\n",
    "dsrc c -t8 FASTQs/mouse_B_rep1_2.fastq FASTQs/mouse_B_rep1_2.fastq.dsrc\n",
    "dsrc c -t8 FASTQs/mouse_B_rep2_1.fastq FASTQs/mouse_B_rep2_1.fastq.dsrc\n",
    "dsrc c -t8 FASTQs/mouse_B_rep2_2.fastq FASTQs/mouse_B_rep2_2.fastq.dsrc\n",
    "dsrc c -t8 FASTQs/mouse_PSC_rep1_1.fastq FASTQs/mouse_PSC_rep1_1.fastq.dsrc\n",
    "dsrc c -t8 FASTQs/mouse_PSC_rep1_2.fastq FASTQs/mouse_PSC_rep1_2.fastq.dsrc\n",
    "dsrc c -t8 FASTQs/mouse_PSC_rep2_1.fastq FASTQs/mouse_PSC_rep2_1.fastq.dsrc\n",
    "dsrc c -t8 FASTQs/mouse_PSC_rep2_2.fastq FASTQs/mouse_PSC_rep2_2.fastq.dsrc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After compression we reduce the total size to 27 Gb (__20% of the original size, and dsrc ensures fast reading of the compressed data__)\n",
    "\n",
    "_Note:_\n",
    "   - _using gzip  instead reduces size to ~38 Gb (occupies ~40% more than dsrc compressed files)_\n",
    "   - _using bzip2 instead reduces size to ~31 Gb (occupies ~15% more than dsrc compressed files)_\n",
    "\n",
    "  _Both are much slower to generate and read_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "rm -f FASTQs/mouse_B_rep1_1.fastq\n",
    "rm -f FASTQs/mouse_B_rep1_2.fastq\n",
    "rm -f FASTQs/mouse_B_rep2_1.fastq\n",
    "rm -f FASTQs/mouse_B_rep2_2.fastq\n",
    "rm -f FASTQs/mouse_PSC_rep1_1.fastq\n",
    "rm -f FASTQs/mouse_PSC_rep1_2.fastq\n",
    "rm -f FASTQs/mouse_PSC_rep2_1.fastq\n",
    "rm -f FASTQs/mouse_PSC_rep2_2.fastq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "<a name=\"cite-ralph\"/><sup>[^](#ref-1) </sup>Stadhouders R, Vidal E, Serra F, Di Stefano B et al. 2018. _Transcription factors orchestrate dynamic interplay between genome topology and gene regulation during cell reprogramming_.\n",
    "\n",
    "<a name=\"cite-roguski2014dsrc\"/><sup>[^](#ref-4) </sup>Roguski, \\Lukasz and Deorowicz, Sebastian. 2014. _DSRC 2—Industry-oriented compression of FASTQ files_.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--bibtex\n",
    "@article{le2014distinct,\n",
    "  title={Distinct structural transitions of chromatin topological domains correlate with coordinated hormone-induced gene regulation},\n",
    "  author={Le Dily, Fran{\\c{c}}ois and Ba{\\`u}, Davide and Pohl, Andy and Vicent, Guillermo P and Serra, Fran{\\c{c}}ois and Soronellas, Daniel and Castellano, Giancarlo and Wright, Roni HG and Ballare, Cecilia and Filion, Guillaume and others},\n",
    "  journal={Genes \\& development},\n",
    "  volume={28},\n",
    "  number={19},\n",
    "  pages={2151--2162},\n",
    "  year={2014},\n",
    "  publisher={Cold Spring Harbor Lab}\n",
    "}\n",
    "@article{roguski2014dsrc,\n",
    "  title={DSRC 2—Industry-oriented compression of FASTQ files},\n",
    "  author={Roguski, {\\L}ukasz and Deorowicz, Sebastian},\n",
    "  journal={Bioinformatics},\n",
    "  volume={30},\n",
    "  number={15},\n",
    "  pages={2213--2215},\n",
    "  year={2014},\n",
    "  publisher={Oxford Univ Press}\n",
    "}\n",
    "-->"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}