ukb_rap_siloed_analyses/04_ukb_plink_merge_bed_files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Merge BED files\n",
    "\n",
    "In this notebook, we merge the filtered BED files using [PLINK](https://www.cog-genomics.org/plink/1.9/).\n",
    "\n",
    "Note that this work is part of a larger project to [Demonstrate the Potential for Pooled Analysis of All of Us and UK Biobank Genomic Data](https://github.com/all-of-us/ukb-cross-analysis-demo-project). Specifically this is for the portion of the project that is the **siloed** analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup \n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <b>Cloud Environment</b>: This notebook was written for use on the UK Biobank Research Analysis Platform.\n",
    "    <ul>\n",
    "        <li>Use compute type 'Single Node' with sufficient CPU and RAM (e.g. start with 8 CPUs and 30 GB RAM, increase if needed).</li>\n",
    "        <li>This notebook can take a while to run (e.g., 90 minutes). Recommend that it is run in the background via <kbd>dx run dxjupyterlab</kbd> which will also capture provenance.</li>\n",
    "    </ul>\n",
    "</div>\n",
    "\n",
    "```\n",
    "dx run dxjupyterlab \\\n",
    "    --instance-type=mem2_ssd1_v2_x8 \\\n",
    "    -icmd=\"papermill 04_ukb_plink_merge_bed_files.ipynb 04_ukb_plink_merge_bed_files_$(date +%Y%m%d).ipynb\" \\\n",
    "    -iin=04_ukb_plink_merge_bed_files.ipynb \\\n",
    "    --folder=outputs/plink-merge-bed/$(date +%Y%m%d)/\n",
    "```\n",
    "See also https://platform.dnanexus.com/app/dxjupyterlab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup plink\n",
    "\n",
    "https://www.cog-genomics.org/plink/1.9/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "##### plink 1 install\n",
    "PLINK_VERSION=20210606\n",
    "PLINK_ZIP_PATH=/tmp/plink-$PLINK_VERSION.zip\n",
    "curl -L -o $PLINK_ZIP_PATH https://s3.amazonaws.com/plink1-assets/plink_linux_x86_64_$PLINK_VERSION.zip\n",
    "mkdir -p /tmp/plink/\n",
    "unzip -o $PLINK_ZIP_PATH -d /tmp/plink/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!/tmp/plink/plink --version # --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define constants\n",
    "\n",
    "This takes as input the WES data from UK Biobank further filtered by notebook `1_ukb_plink_bed_filter.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Papermill parameters. See https://papermill.readthedocs.io/en/latest/usage-parameterize.html\n",
    "\n",
    "#---[ Inputs ]---\n",
    "# These files were created via ukb_rap_siloed_analyses/03_ukb_plink_bed_filter.ipynb\n",
    "BED_PATTERN = '/mnt/project/outputs/plink-make-bed/20220420/ukb_200kwes_chr*_plink_makebed'\n",
    "\n",
    "#---[ Outputs ]---\n",
    "BED_MERGE_LIST = 'bed_merge_list.txt'\n",
    "PLINK_OUTPUT_FILENAME_PREFIX = f'ukb_200kwes_filtered'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Write out the merge list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "chrom = 1\n",
    "INITIAL_BED = BED_PATTERN.replace('*', str(chrom))\n",
    "\n",
    "INITIAL_BED\n",
    "\n",
    "with open(BED_MERGE_LIST, mode='w') as the_file:\n",
    "    for chrom in range(2, 23):\n",
    "        path = BED_PATTERN.replace('*', str(chrom))\n",
    "        the_file.write(f'{path}.bed {path}.bim {path}.fam \\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat {BED_MERGE_LIST}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Write out the merged BED file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!/tmp/plink/plink \\\n",
    "  --bed {INITIAL_BED}.bed \\\n",
    "  --bim {INITIAL_BED}.bim \\\n",
    "  --fam {INITIAL_BED}.fam \\\n",
    "  --merge-list {BED_MERGE_LIST} \\\n",
    "  --make-bed \\\n",
    "  --out {PLINK_OUTPUT_FILENAME_PREFIX}_plink_mergebed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "ls -lat | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Provenance "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "pip3 freeze"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "214px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}