pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas learn tutorial\n",
    "===\n",
    "陈同 chentong_biology@163.com\n",
    "========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is pandas\n",
    "\n",
    "Pandas是python中用于处理矩阵样数据的功能强大的包，提供了R中的`dataframe`和`vector`的操作，使得我们在使用python时，也可以方便、简单、快捷、高效地进行矩阵数据处理。\n",
    "\n",
    "具体介绍详见<http://pandas.pydata.org/>。\n",
    "\n",
    "\n",
    "* A fast and efficient **DataFrame** object for data manipulation with integrated indexing;\n",
    "* Tools for reading and writing data between in-memory data structures and different formats: CSV and text files, Microsoft Excel, SQL databases, and the fast **HDF5** format;\n",
    "* Intelligent **data alignment** and integrated handling of missing data: gain automatic label-based alignment in computations and easily manipulate **messy data into an orderly form**;\n",
    "* Flexible **reshaping** and **pivoting** of data sets;\n",
    "* Intelligent label-based slicing, fancy indexing, and subsetting of large data sets;\n",
    "* Columns can be inserted and deleted from data structures for size mutability;\n",
    "* Aggregating or transforming data with a powerful group by engine allowing split-apply-combine operations on data sets;\n",
    "* High performance **merging** and **joining** of data sets;\n",
    "* Hierarchical axis indexing provides an intuitive way of working with high-dimensional data in a lower-dimensional data structure;\n",
    "* Time series-functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging. Even create domain-specific time offsets and join time series without losing data;\n",
    "* Highly optimized for performance, with critical code paths written in Cython or C.\n",
    "* Python with pandas is in use in a wide variety of academic and commercial domains, including Finance, Neuroscience, Economics, Statistics, Advertising, Web Analytics, and more.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "#import plotly\n",
    "#plotly.offline.init_notebook_mode()\n",
    "\n",
    "import matplotlib\n",
    "matplotlib.style.use('ggplot')\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import os\n",
    "from glob import glob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Pandas读取文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 获取目标文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['pandas_data/ENCFF289HGQ.tsv',\n",
       " 'pandas_data/gencode.v24.ENS2SYN',\n",
       " 'pandas_data/ENCFF262OBL.tsv',\n",
       " 'pandas_data/Gene_metadata_primary_wt_whole_cell.tsv',\n",
       " 'pandas_data/ENCFF673KYR.tsv',\n",
       " 'pandas_data/ENCFF060LPA.tsv']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir_1 = \"pandas_data/\"\n",
    "glob(dir_1+'*')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 查看目标文件内容和格式\n",
    "Ipython中可以通过在Linux命令前加`!`调用系统命令，更多使用见\n",
    "http://ipython.org/ipython-doc/3/interactive/reference.html#system-shell-access."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gene_id\tgene_symbol\r\n",
      "ENSG00000223972.5\tDDX11L1\r\n",
      "ENSG00000227232.5\tWASH7P\r\n",
      "ENSG00000278267.1\tMIR6859-1\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 4 pandas_data/gencode.v24.ENS2SYN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "transcript_id\tgene_id\tlength\teffective_length\texpected_count\tTPM\tFPKM\r\n",
      "ENST00000373020.4\tENSG00000000003.10\t2206\t1925.57\t997.23\t1.71\t7.21\r\n",
      "ENST00000494424.1\tENSG00000000003.10\t820\t539.58\t24.77\t0.15\t0.64\r\n",
      "ENST00000496771.1\tENSG00000000003.10\t1025\t744.57\t0.00\t0.00\t0.00\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 4 pandas_data/ENCFF556YSD.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 读取两列文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ens2syn_file = \"pandas_data/gencode.v24.ENS2SYN\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# pandas中的计数都是从0开始的\n",
    "# header=0: 指定第一行包含列的名字\n",
    "# index_col=0: 指定第一列为行的名字\n",
    "ens2syn = pd.read_table(ens2syn_file, header=0, index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 数据表的索引\n",
    "    * 数值索引和布尔值索引是按行选取\n",
    "    * 字符串索引是按列选取\n",
    "    * 行和列是等效的，应用于行的选取函数也可应用于列，反之亦然"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 按行选取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gene_symbol</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000223972.5</th>\n",
       "      <td>DDX11L1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000227232.5</th>\n",
       "      <td>WASH7P</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000278267.1</th>\n",
       "      <td>MIR6859-1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  gene_symbol\n",
       "gene_id                      \n",
       "ENSG00000223972.5     DDX11L1\n",
       "ENSG00000227232.5      WASH7P\n",
       "ENSG00000278267.1   MIR6859-1"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ens2syn[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 取出索引中包含特定值的行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gene_symbol</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000227232.5</th>\n",
       "      <td>WASH7P</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  gene_symbol\n",
       "gene_id                      \n",
       "ENSG00000227232.5      WASH7P"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ens2syn[ens2syn.index==\"ENSG00000227232.5\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 取出某列包含特定值列表的行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gene_symbol</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000223972.5</th>\n",
       "      <td>DDX11L1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000278267.1</th>\n",
       "      <td>MIR6859-1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  gene_symbol\n",
       "gene_id                      \n",
       "ENSG00000223972.5     DDX11L1\n",
       "ENSG00000278267.1   MIR6859-1"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ens2syn[ens2syn['gene_symbol'].isin(['DDX11L1','MIR6859-1'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 使用正则表达式选取符合要求的行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gene_symbol</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000223972.5</th>\n",
       "      <td>DDX11L1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000227232.5</th>\n",
       "      <td>WASH7P</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000222623.1</th>\n",
       "      <td>RNU6-1100P</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000228463.9</th>\n",
       "      <td>AP006222.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000224813.3</th>\n",
       "      <td>SEPT14P13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  gene_symbol\n",
       "gene_id                      \n",
       "ENSG00000223972.5     DDX11L1\n",
       "ENSG00000227232.5      WASH7P\n",
       "ENSG00000222623.1  RNU6-1100P\n",
       "ENSG00000228463.9  AP006222.2\n",
       "ENSG00000224813.3   SEPT14P13"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# head: 只展示部分数据\n",
    "ens2syn[ens2syn.index.str.contains(r'ENSG0000022')].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 读取多列文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`gzip`, `bzip`压缩的文件也可以直接读取，但是需要保证文件后缀的正确。\n",
    "`read_table`默认参数可以自动检测文件的格式，根据文件的后缀 '.gz', '.bz2', '.zip', or 'xz'分别使用 gzip, bz2, zip or xz读取。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['pandas_data/ENCFF289HGQ.tsv',\n",
       " 'pandas_data/ENCFF262OBL.tsv',\n",
       " 'pandas_data/ENCFF673KYR.tsv',\n",
       " 'pandas_data/ENCFF060LPA.tsv']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tsvL = glob(dir_1+'ENC*.tsv')\n",
    "tsvL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>transcript_id(s)</th>\n",
       "      <th>length</th>\n",
       "      <th>effective_length</th>\n",
       "      <th>expected_count</th>\n",
       "      <th>TPM</th>\n",
       "      <th>FPKM</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>ENST00000373020.8,ENST00000494424.1,ENST000004...</td>\n",
       "      <td>2198.69</td>\n",
       "      <td>1939.24</td>\n",
       "      <td>2827.0</td>\n",
       "      <td>1.03</td>\n",
       "      <td>10.84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000005.5</th>\n",
       "      <td>ENST00000373031.4,ENST00000485971.1</td>\n",
       "      <td>940.50</td>\n",
       "      <td>681.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>ENST00000371582.8,ENST00000371584.8,ENST000003...</td>\n",
       "      <td>1079.84</td>\n",
       "      <td>820.38</td>\n",
       "      <td>1680.0</td>\n",
       "      <td>1.45</td>\n",
       "      <td>15.23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     transcript_id(s)  \\\n",
       "gene_id                                                                 \n",
       "ENSG00000000003.14  ENST00000373020.8,ENST00000494424.1,ENST000004...   \n",
       "ENSG00000000005.5                 ENST00000373031.4,ENST00000485971.1   \n",
       "ENSG00000000419.12  ENST00000371582.8,ENST00000371584.8,ENST000003...   \n",
       "\n",
       "                     length  effective_length  expected_count   TPM   FPKM  \n",
       "gene_id                                                                     \n",
       "ENSG00000000003.14  2198.69           1939.24          2827.0  1.03  10.84  \n",
       "ENSG00000000005.5    940.50            681.07             0.0  0.00   0.00  \n",
       "ENSG00000000419.12  1079.84            820.38          1680.0  1.45  15.23  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index = 0\n",
    "tsvFile = tsvL[index]\n",
    "expr = pd.read_table(tsvFile, header=0, index_col=0)\n",
    "expr.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 选取多列数据\n",
    "列的输出顺序与给定的列名字的顺序一致"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FPKM</th>\n",
       "      <th>TPM</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>10.84</td>\n",
       "      <td>1.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000005.5</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>15.23</td>\n",
       "      <td>1.45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     FPKM   TPM\n",
       "gene_id                        \n",
       "ENSG00000000003.14  10.84  1.03\n",
       "ENSG00000000005.5    0.00  0.00\n",
       "ENSG00000000419.12  15.23  1.45"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "expr[['FPKM','TPM']].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 重命名列名字\n",
    "从Dataframe中只选取一列时，数据框会被转换成Series，因此需要使用`pd.loc[:,[column_name]]`(虽然内部的方括号内只有一个值，但写法是必须的)索引。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ENCFF289HGQ\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>1.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000005.5</th>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>1.45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF289HGQ\n",
       "gene_id                        \n",
       "ENSG00000000003.14         1.03\n",
       "ENSG00000000005.5          0.00\n",
       "ENSG00000000419.12         1.45"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 因为要把多个文件的同一类型表达值合并到一个文件，我们使用文件名作为列的名字\n",
    "name = os.path.split(tsvFile)[-1][:-4]\n",
    "print name\n",
    "expr_tpm = expr.loc[:,['TPM']] # 取出所有的行和名字为TPM的列\n",
    "expr_tpm.columns=[name]  \n",
    "expr_tpm[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 合并矩阵"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 定义函数简化文件读取"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 为了读取多个文件，定义一个函数简化操作\n",
    "def readExpr_1(tsvFileL, typeL=['TPM','FPKM']):\n",
    "    '''\n",
    "    tsvFileL: lists of files waiting for reading\n",
    "    resultD: a dictionary to save data matrix\n",
    "            {'TPM':[mat1, mat2,...]\n",
    "             'FPKM':[mat1, mat2, ...]}\n",
    "    typeL; list of names for columns to be extracted\n",
    "    '''\n",
    "    resultD = {}\n",
    "    for _type in typeL: resultD[_type] = []\n",
    "    \n",
    "    for tsvFile in tsvFileL:\n",
    "        expr = pd.read_table(tsvFile, header=0, index_col=0)\n",
    "        name = os.path.split(tsvFile)[-1][:-4]  #this options is very arbitary\n",
    "        for _type in typeL: # add _ to type to avoid override Python inner function `type` \n",
    "            expr_type = expr.loc[:,[_type]]\n",
    "            expr_type.columns = [name]\n",
    "            resultD[_type].append(expr_type)\n",
    "    return resultD\n",
    "#-----------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "exprD = readExpr_1(tsvL)\n",
    "TPM_mat = exprD['TPM']\n",
    "FPKM_mat = exprD['FPKM']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 使用pd.merge合并矩阵示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "先从刚才读取的矩阵中选出2个测试下pandas中的矩阵合并方法和效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>1.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000005.5</th>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>1.45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000457.13</th>\n",
       "      <td>0.24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF289HGQ\n",
       "gene_id                        \n",
       "ENSG00000000003.14         1.03\n",
       "ENSG00000000005.5          0.00\n",
       "ENSG00000000419.12         1.45\n",
       "ENSG00000000457.13         0.24"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取第一个矩阵\n",
    "_idL = ['ENSG00000000003.14', 'ENSG00000000005.5','ENSG00000000419.12','ENSG00000000457.13']\n",
    "mat1 = TPM_mat[0]\n",
    "mat1 = mat1[mat1.index.isin(_idL)]\n",
    "mat1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF262OBL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>17.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>18.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001036.13</th>\n",
       "      <td>10.34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001561.6</th>\n",
       "      <td>2.47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF262OBL\n",
       "gene_id                        \n",
       "ENSG00000000003.14        17.13\n",
       "ENSG00000000419.12        18.86\n",
       "ENSG00000001036.13        10.34\n",
       "ENSG00000001561.6          2.47"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取第二个矩阵\n",
    "_idL = ['ENSG00000001561.6','ENSG00000000003.14', 'ENSG00000000419.12','ENSG00000001036.13']\n",
    "mat2 = TPM_mat[1]\n",
    "mat2 = mat2[mat2.index.isin(_idL)]\n",
    "mat2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "基于索引(index)的合并\n",
    "    * outer: 合并所有的索引，缺失值填充NA\n",
    "    * inner：保留共有的索引\n",
    "    * left：使用第一个矩阵的索引\n",
    "    * right：使用第二个矩阵的索引"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <th>ENCFF262OBL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>1.03</td>\n",
       "      <td>17.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000005.5</th>\n",
       "      <td>0.00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>1.45</td>\n",
       "      <td>18.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000457.13</th>\n",
       "      <td>0.24</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001036.13</th>\n",
       "      <td>NaN</td>\n",
       "      <td>10.34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001561.6</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2.47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF289HGQ  ENCFF262OBL\n",
       "gene_id                                     \n",
       "ENSG00000000003.14         1.03        17.13\n",
       "ENSG00000000005.5          0.00          NaN\n",
       "ENSG00000000419.12         1.45        18.86\n",
       "ENSG00000000457.13         0.24          NaN\n",
       "ENSG00000001036.13          NaN        10.34\n",
       "ENSG00000001561.6           NaN         2.47"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.merge(mat1, mat2, left_index=True, right_index=True, how=\"outer\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <th>ENCFF262OBL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>1.03</td>\n",
       "      <td>17.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>1.45</td>\n",
       "      <td>18.86</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF289HGQ  ENCFF262OBL\n",
       "gene_id                                     \n",
       "ENSG00000000003.14         1.03        17.13\n",
       "ENSG00000000419.12         1.45        18.86"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.merge(mat1, mat2, left_index=True, right_index=True, how=\"inner\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <th>ENCFF262OBL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>1.03</td>\n",
       "      <td>17.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000005.5</th>\n",
       "      <td>0.00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>1.45</td>\n",
       "      <td>18.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000457.13</th>\n",
       "      <td>0.24</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF289HGQ  ENCFF262OBL\n",
       "gene_id                                     \n",
       "ENSG00000000003.14         1.03        17.13\n",
       "ENSG00000000005.5          0.00          NaN\n",
       "ENSG00000000419.12         1.45        18.86\n",
       "ENSG00000000457.13         0.24          NaN"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.merge(mat1, mat2, left_index=True, right_index=True, how=\"left\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 使用pd.concat合并矩阵示例 \n",
    "对于较多的数据表合并操作时，`concat`比`merge`要简单快速很多。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <th>ENCFF262OBL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>1.03</td>\n",
       "      <td>17.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000005.5</th>\n",
       "      <td>0.00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>1.45</td>\n",
       "      <td>18.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000457.13</th>\n",
       "      <td>0.24</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001036.13</th>\n",
       "      <td>NaN</td>\n",
       "      <td>10.34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001561.6</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2.47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF289HGQ  ENCFF262OBL\n",
       "ENSG00000000003.14         1.03        17.13\n",
       "ENSG00000000005.5          0.00          NaN\n",
       "ENSG00000000419.12         1.45        18.86\n",
       "ENSG00000000457.13         0.24          NaN\n",
       "ENSG00000001036.13          NaN        10.34\n",
       "ENSG00000001561.6           NaN         2.47"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([mat1, mat2], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <th>ENCFF262OBL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>1.03</td>\n",
       "      <td>17.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>1.45</td>\n",
       "      <td>18.86</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF289HGQ  ENCFF262OBL\n",
       "gene_id                                     \n",
       "ENSG00000000003.14         1.03        17.13\n",
       "ENSG00000000419.12         1.45        18.86"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([mat1, mat2], axis=1, join=\"inner\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 使用pd.join合并矩阵示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <th>ENCFF262OBL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>1.03</td>\n",
       "      <td>17.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000005.5</th>\n",
       "      <td>0.00</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>1.45</td>\n",
       "      <td>18.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000457.13</th>\n",
       "      <td>0.24</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001036.13</th>\n",
       "      <td>NaN</td>\n",
       "      <td>10.34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001561.6</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2.47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF289HGQ  ENCFF262OBL\n",
       "gene_id                                     \n",
       "ENSG00000000003.14         1.03        17.13\n",
       "ENSG00000000005.5          0.00          NaN\n",
       "ENSG00000000419.12         1.45        18.86\n",
       "ENSG00000000457.13         0.24          NaN\n",
       "ENSG00000001036.13          NaN        10.34\n",
       "ENSG00000001561.6           NaN         2.47"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mat3 = mat1.join(mat2, how=\"outer\")\n",
    "mat3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "替换NA值为0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <th>ENCFF262OBL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>1.03</td>\n",
       "      <td>17.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000005.5</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>1.45</td>\n",
       "      <td>18.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000457.13</th>\n",
       "      <td>0.24</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001036.13</th>\n",
       "      <td>0.00</td>\n",
       "      <td>10.34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001561.6</th>\n",
       "      <td>0.00</td>\n",
       "      <td>2.47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF289HGQ  ENCFF262OBL\n",
       "gene_id                                     \n",
       "ENSG00000000003.14         1.03        17.13\n",
       "ENSG00000000005.5          0.00         0.00\n",
       "ENSG00000000419.12         1.45        18.86\n",
       "ENSG00000000457.13         0.24         0.00\n",
       "ENSG00000001036.13         0.00        10.34\n",
       "ENSG00000001561.6          0.00         2.47"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mat3 = mat3.fillna(0)\n",
    "mat3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "去除所有值都为0的行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <th>ENCFF262OBL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>1.03</td>\n",
       "      <td>17.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>1.45</td>\n",
       "      <td>18.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000457.13</th>\n",
       "      <td>0.24</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001036.13</th>\n",
       "      <td>0.00</td>\n",
       "      <td>10.34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001561.6</th>\n",
       "      <td>0.00</td>\n",
       "      <td>2.47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF289HGQ  ENCFF262OBL\n",
       "gene_id                                     \n",
       "ENSG00000000003.14         1.03        17.13\n",
       "ENSG00000000419.12         1.45        18.86\n",
       "ENSG00000000457.13         0.24         0.00\n",
       "ENSG00000001036.13         0.00        10.34\n",
       "ENSG00000001561.6          0.00         2.47"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Both works well here\n",
    "#mat3[(mat3>0).any(axis=1)]\n",
    "mat3.loc[(mat3>0).any(axis=1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 测试三种方法使用的内存和速度比较"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "速度：`concat`>`join`>>`merge`\n",
    "\n",
    "内存：相当"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100 loops, best of 3: 3.36 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit test_merge = reduce(lambda left,right: pd.merge(left,right,left_index=True,right_index=True,how='outer'), TPM_mat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000 loops, best of 3: 1.21 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit test_merge = pd.concat(TPM_mat, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000 loops, best of 3: 1.25 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%timeit TPM_mat[0].join(TPM_mat[1:], how=\"outer\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%load_ext memory_profiler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "peak memory: 107.32 MiB, increment: 0.01 MiB\n"
     ]
    }
   ],
   "source": [
    "%memit test_merge = reduce(lambda left,right: pd.merge(left,right,left_index=True,right_index=True,how='outer'), TPM_mat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "peak memory: 107.30 MiB, increment: 0.10 MiB\n"
     ]
    }
   ],
   "source": [
    "%memit test_merge = pd.concat(TPM_mat, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "peak memory: 107.32 MiB, increment: 0.00 MiB\n"
     ]
    }
   ],
   "source": [
    "%memit TPM_mat[0].join(TPM_mat[1:], how=\"outer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 重写函数完成文件的读写和矩阵的合并"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 读取多个文件，并且合并矩阵，定义一个函数简化操作\n",
    "def concatExpr(tsvFileL, typeL=['TPM','FPKM']):\n",
    "    '''\n",
    "    tsvFileL: lists of files waiting for reading\n",
    "    resultD: a dictionary to save data matrix\n",
    "            {'TPM':[mat1, mat2,...]\n",
    "             'FPKM':[mat1, mat2, ...]}\n",
    "    typeL; list of names for columns to be extracted\n",
    "    '''\n",
    "    resultD = {}\n",
    "    for _type in typeL: resultD[_type] = []\n",
    "    \n",
    "    for tsvFile in tsvFileL:\n",
    "        expr = pd.read_table(tsvFile, header=0, index_col=0)\n",
    "        name = os.path.split(tsvFile)[-1][:-4]  #this options is very arbitary\n",
    "        for _type in typeL: # add _ to type to avoid override Python inner function `type` \n",
    "            expr_type = expr.loc[:,[_type]]\n",
    "            expr_type.columns = [name]\n",
    "            resultD[_type].append(expr_type)\n",
    "    #-------------------------------------------\n",
    "    mergeD = {}\n",
    "    for _type in typeL:\n",
    "        mergeM = pd.concat(resultD[_type], axis=1)\n",
    "        mergeM = mergeM.fillna(0) # Substitute all NA with 0\n",
    "        mergeM = mergeM.loc[(mergeM>0).any(axis=1)] # Delete aoo zero rows.\n",
    "        mergeD[_type] = mergeM\n",
    "    return mergeD\n",
    "#-----------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "typeL = ['TPM','FPKM']\n",
    "exprD = concatExpr(tsvL, typeL)\n",
    "TPM_mat = exprD['TPM']\n",
    "FPKM_mat = exprD['FPKM']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <th>ENCFF262OBL</th>\n",
       "      <th>ENCFF673KYR</th>\n",
       "      <th>ENCFF060LPA</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>1.03</td>\n",
       "      <td>17.13</td>\n",
       "      <td>2.42</td>\n",
       "      <td>6.64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>1.45</td>\n",
       "      <td>18.86</td>\n",
       "      <td>1.80</td>\n",
       "      <td>9.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000457.13</th>\n",
       "      <td>0.24</td>\n",
       "      <td>2.48</td>\n",
       "      <td>0.38</td>\n",
       "      <td>0.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000460.16</th>\n",
       "      <td>0.26</td>\n",
       "      <td>5.36</td>\n",
       "      <td>0.16</td>\n",
       "      <td>1.51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000938.12</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF289HGQ  ENCFF262OBL  ENCFF673KYR  ENCFF060LPA\n",
       "gene_id                                                               \n",
       "ENSG00000000003.14         1.03        17.13         2.42         6.64\n",
       "ENSG00000000419.12         1.45        18.86         1.80         9.91\n",
       "ENSG00000000457.13         0.24         2.48         0.38         0.86\n",
       "ENSG00000000460.16         0.26         5.36         0.16         1.51\n",
       "ENSG00000000938.12         0.00         0.05         0.00         0.01"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TPM_mat.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 矩阵数据提取"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "只保留表达矩阵中存储的基因的`ID`和`Symbol`对照表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gene_symbol</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000223972.5</th>\n",
       "      <td>DDX11L1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000227232.5</th>\n",
       "      <td>WASH7P</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000278267.1</th>\n",
       "      <td>MIR6859-1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  gene_symbol\n",
       "gene_id                      \n",
       "ENSG00000223972.5     DDX11L1\n",
       "ENSG00000227232.5      WASH7P\n",
       "ENSG00000278267.1   MIR6859-1"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 回顾下数据格式\n",
    "ens2syn.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60725, 1)"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ens2syn.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ens2syn = ens2syn[ens2syn.index.isin(TPM_mat.index)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(48, 1)"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ens2syn.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gene_symbol</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000001460.17</th>\n",
       "      <td>STPG1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001461.16</th>\n",
       "      <td>NIPAL3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000938.12</th>\n",
       "      <td>FGR</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   gene_symbol\n",
       "gene_id                       \n",
       "ENSG00000001460.17       STPG1\n",
       "ENSG00000001461.16      NIPAL3\n",
       "ENSG00000000938.12         FGR"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ens2syn.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 读取META data文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>File format</th>\n",
       "      <th>Output type</th>\n",
       "      <th>Experiment accession</th>\n",
       "      <th>Assay</th>\n",
       "      <th>Biosample term id</th>\n",
       "      <th>Biosample term name</th>\n",
       "      <th>Biosample type</th>\n",
       "      <th>Biosample life stage</th>\n",
       "      <th>Biosample sex</th>\n",
       "      <th>Biosample organism</th>\n",
       "      <th>...</th>\n",
       "      <th>md5sum</th>\n",
       "      <th>File download URL</th>\n",
       "      <th>Assembly</th>\n",
       "      <th>Platform</th>\n",
       "      <th>Controlled by</th>\n",
       "      <th>File Status</th>\n",
       "      <th>Audit WARNING</th>\n",
       "      <th>Audit INTERNAL_ACTION</th>\n",
       "      <th>Audit NOT_COMPLIANT</th>\n",
       "      <th>Audit ERROR</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>File accession</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENCFF120PLK</th>\n",
       "      <td>tsv</td>\n",
       "      <td>gene quantifications</td>\n",
       "      <td>ENCSR198TKA</td>\n",
       "      <td>RNA-seq</td>\n",
       "      <td>CL:0000650</td>\n",
       "      <td>mesangial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>unknown, fetal</td>\n",
       "      <td>unknown, female</td>\n",
       "      <td>Homo sapiens</td>\n",
       "      <td>...</td>\n",
       "      <td>1e9a3db25f5361b2ca454d1df427f430</td>\n",
       "      <td>https://www.encodeproject.org/files/ENCFF120PL...</td>\n",
       "      <td>hg19</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF805BVE</th>\n",
       "      <td>tsv</td>\n",
       "      <td>gene quantifications</td>\n",
       "      <td>ENCSR198TKA</td>\n",
       "      <td>RNA-seq</td>\n",
       "      <td>CL:0000650</td>\n",
       "      <td>mesangial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>unknown, fetal</td>\n",
       "      <td>unknown, female</td>\n",
       "      <td>Homo sapiens</td>\n",
       "      <td>...</td>\n",
       "      <td>ee0e94d6795ed7c2ef69c61b1d29eb02</td>\n",
       "      <td>https://www.encodeproject.org/files/ENCFF805BV...</td>\n",
       "      <td>hg19</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF850RHD</th>\n",
       "      <td>tsv</td>\n",
       "      <td>gene quantifications</td>\n",
       "      <td>ENCSR198TKA</td>\n",
       "      <td>RNA-seq</td>\n",
       "      <td>CL:0000650</td>\n",
       "      <td>mesangial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>unknown, fetal</td>\n",
       "      <td>unknown, female</td>\n",
       "      <td>Homo sapiens</td>\n",
       "      <td>...</td>\n",
       "      <td>22f948135c0935516f19f6b995ccc30c</td>\n",
       "      <td>https://www.encodeproject.org/files/ENCFF850RH...</td>\n",
       "      <td>GRCh38</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 47 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               File format           Output type Experiment accession  \\\n",
       "File accession                                                          \n",
       "ENCFF120PLK            tsv  gene quantifications          ENCSR198TKA   \n",
       "ENCFF805BVE            tsv  gene quantifications          ENCSR198TKA   \n",
       "ENCFF850RHD            tsv  gene quantifications          ENCSR198TKA   \n",
       "\n",
       "                  Assay Biosample term id Biosample term name Biosample type  \\\n",
       "File accession                                                                 \n",
       "ENCFF120PLK     RNA-seq        CL:0000650      mesangial cell   primary cell   \n",
       "ENCFF805BVE     RNA-seq        CL:0000650      mesangial cell   primary cell   \n",
       "ENCFF850RHD     RNA-seq        CL:0000650      mesangial cell   primary cell   \n",
       "\n",
       "               Biosample life stage    Biosample sex Biosample organism  \\\n",
       "File accession                                                            \n",
       "ENCFF120PLK          unknown, fetal  unknown, female       Homo sapiens   \n",
       "ENCFF805BVE          unknown, fetal  unknown, female       Homo sapiens   \n",
       "ENCFF850RHD          unknown, fetal  unknown, female       Homo sapiens   \n",
       "\n",
       "                   ...                                md5sum  \\\n",
       "File accession     ...                                         \n",
       "ENCFF120PLK        ...      1e9a3db25f5361b2ca454d1df427f430   \n",
       "ENCFF805BVE        ...      ee0e94d6795ed7c2ef69c61b1d29eb02   \n",
       "ENCFF850RHD        ...      22f948135c0935516f19f6b995ccc30c   \n",
       "\n",
       "                                                File download URL  Assembly  \\\n",
       "File accession                                                                \n",
       "ENCFF120PLK     https://www.encodeproject.org/files/ENCFF120PL...      hg19   \n",
       "ENCFF805BVE     https://www.encodeproject.org/files/ENCFF805BV...      hg19   \n",
       "ENCFF850RHD     https://www.encodeproject.org/files/ENCFF850RH...    GRCh38   \n",
       "\n",
       "                Platform  Controlled by  File Status Audit WARNING  \\\n",
       "File accession                                                       \n",
       "ENCFF120PLK          NaN            NaN     released           NaN   \n",
       "ENCFF805BVE          NaN            NaN     released           NaN   \n",
       "ENCFF850RHD          NaN            NaN     released           NaN   \n",
       "\n",
       "               Audit INTERNAL_ACTION Audit NOT_COMPLIANT Audit ERROR  \n",
       "File accession                                                        \n",
       "ENCFF120PLK                      NaN                 NaN         NaN  \n",
       "ENCFF805BVE                      NaN                 NaN         NaN  \n",
       "ENCFF850RHD                      NaN                 NaN         NaN  \n",
       "\n",
       "[3 rows x 47 columns]"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meta = \"pandas_data/meta.tsv\"\n",
    "metaM = pd.read_table(meta, header=0, index_col=0)\n",
    "# 重名了列的名字\n",
    "oriColnames = metaM.columns.values\n",
    "nameD = dict([(i,i.replace(' ','_')) for i in oriColnames])\n",
    "metaM.rename(columns=nameD, inplace=True)\n",
    "metaM.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 只保留前面提到的4个样品的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Biosample term id</th>\n",
       "      <th>Biosample term name</th>\n",
       "      <th>Biosample type</th>\n",
       "      <th>Biosample life stage</th>\n",
       "      <th>Biosample sex</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>File accession</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENCFF673KYR</th>\n",
       "      <td>CL:0000650</td>\n",
       "      <td>mesangial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>unknown, fetal</td>\n",
       "      <td>unknown, female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF262OBL</th>\n",
       "      <td>CL:1001568</td>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF060LPA</th>\n",
       "      <td>CL:1001568</td>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <td>CL:0002558</td>\n",
       "      <td>fibroblast of villous mesenchyme</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>newborn</td>\n",
       "      <td>male, female</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Biosample term id                Biosample term name  \\\n",
       "File accession                                                        \n",
       "ENCFF673KYR           CL:0000650                     mesangial cell   \n",
       "ENCFF262OBL           CL:1001568  pulmonary artery endothelial cell   \n",
       "ENCFF060LPA           CL:1001568  pulmonary artery endothelial cell   \n",
       "ENCFF289HGQ           CL:0002558   fibroblast of villous mesenchyme   \n",
       "\n",
       "               Biosample type Biosample life stage    Biosample sex  \n",
       "File accession                                                       \n",
       "ENCFF673KYR      primary cell       unknown, fetal  unknown, female  \n",
       "ENCFF262OBL      primary cell                adult             male  \n",
       "ENCFF060LPA      primary cell                adult             male  \n",
       "ENCFF289HGQ      primary cell              newborn     male, female  "
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampleL = TPM_mat.columns.values\n",
    "metaM = metaM[metaM.index.isin(sampleL)]\n",
    "# 同时索引行和列\n",
    "metaM.ix[:4,:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  提取目标列信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Biosample term id</th>\n",
       "      <th>Biosample term name</th>\n",
       "      <th>Biosample type</th>\n",
       "      <th>Biosample life stage</th>\n",
       "      <th>Biosample sex</th>\n",
       "      <th>Biosample organism</th>\n",
       "      <th>Biosample treatments</th>\n",
       "      <th>Biosample subcellular fraction term name</th>\n",
       "      <th>Biosample phase</th>\n",
       "      <th>Biosample synchronization stage</th>\n",
       "      <th>Biosample Age</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>File accession</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENCFF673KYR</th>\n",
       "      <td>CL:0000650</td>\n",
       "      <td>mesangial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>unknown, fetal</td>\n",
       "      <td>unknown, female</td>\n",
       "      <td>Homo sapiens</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF262OBL</th>\n",
       "      <td>CL:1001568</td>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "      <td>Homo sapiens</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>52 year</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF060LPA</th>\n",
       "      <td>CL:1001568</td>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "      <td>Homo sapiens</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23 year</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <td>CL:0002558</td>\n",
       "      <td>fibroblast of villous mesenchyme</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>newborn</td>\n",
       "      <td>male, female</td>\n",
       "      <td>Homo sapiens</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Biosample term id                Biosample term name  \\\n",
       "File accession                                                        \n",
       "ENCFF673KYR           CL:0000650                     mesangial cell   \n",
       "ENCFF262OBL           CL:1001568  pulmonary artery endothelial cell   \n",
       "ENCFF060LPA           CL:1001568  pulmonary artery endothelial cell   \n",
       "ENCFF289HGQ           CL:0002558   fibroblast of villous mesenchyme   \n",
       "\n",
       "               Biosample type Biosample life stage    Biosample sex  \\\n",
       "File accession                                                        \n",
       "ENCFF673KYR      primary cell       unknown, fetal  unknown, female   \n",
       "ENCFF262OBL      primary cell                adult             male   \n",
       "ENCFF060LPA      primary cell                adult             male   \n",
       "ENCFF289HGQ      primary cell              newborn     male, female   \n",
       "\n",
       "               Biosample organism  Biosample treatments  \\\n",
       "File accession                                            \n",
       "ENCFF673KYR          Homo sapiens                   NaN   \n",
       "ENCFF262OBL          Homo sapiens                   NaN   \n",
       "ENCFF060LPA          Homo sapiens                   NaN   \n",
       "ENCFF289HGQ          Homo sapiens                   NaN   \n",
       "\n",
       "                Biosample subcellular fraction term name  Biosample phase  \\\n",
       "File accession                                                              \n",
       "ENCFF673KYR                                          NaN              NaN   \n",
       "ENCFF262OBL                                          NaN              NaN   \n",
       "ENCFF060LPA                                          NaN              NaN   \n",
       "ENCFF289HGQ                                          NaN              NaN   \n",
       "\n",
       "                Biosample synchronization stage Biosample Age  \n",
       "File accession                                                 \n",
       "ENCFF673KYR                                 NaN           NaN  \n",
       "ENCFF262OBL                                 NaN       52 year  \n",
       "ENCFF060LPA                                 NaN       23 year  \n",
       "ENCFF289HGQ                                 NaN           NaN  "
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 假如只提取`Biosample`开头的列\n",
    "#meta_colL = ['Biosample term id', 'Biosample term name']\n",
    "\n",
    "# Extract columns matching specific patterns\n",
    "# Both works well, filter is more simple\n",
    "#metaM.loc[:,metaM.columns.str.contains(r'^Biosample')]\n",
    "metaM = metaM.filter(regex=(\"^Biosample\"))\n",
    "metaM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Biosample term id</th>\n",
       "      <th>Biosample term name</th>\n",
       "      <th>Biosample type</th>\n",
       "      <th>Biosample life stage</th>\n",
       "      <th>Biosample sex</th>\n",
       "      <th>Biosample organism</th>\n",
       "      <th>Biosample treatments</th>\n",
       "      <th>Biosample subcellular fraction term name</th>\n",
       "      <th>Biosample phase</th>\n",
       "      <th>Biosample synchronization stage</th>\n",
       "      <th>Biosample Age</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>File accession</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENCFF673KYR</th>\n",
       "      <td>CL:0000650</td>\n",
       "      <td>mesangial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>unknown, fetal</td>\n",
       "      <td>unknown, female</td>\n",
       "      <td>Homo sapiens</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF262OBL</th>\n",
       "      <td>CL:1001568</td>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "      <td>Homo sapiens</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>52 year</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF060LPA</th>\n",
       "      <td>CL:1001568</td>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "      <td>Homo sapiens</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>23 year</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <td>CL:0002558</td>\n",
       "      <td>fibroblast of villous mesenchyme</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>newborn</td>\n",
       "      <td>male, female</td>\n",
       "      <td>Homo sapiens</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Biosample term id                Biosample term name  \\\n",
       "File accession                                                        \n",
       "ENCFF673KYR           CL:0000650                     mesangial cell   \n",
       "ENCFF262OBL           CL:1001568  pulmonary artery endothelial cell   \n",
       "ENCFF060LPA           CL:1001568  pulmonary artery endothelial cell   \n",
       "ENCFF289HGQ           CL:0002558   fibroblast of villous mesenchyme   \n",
       "\n",
       "               Biosample type Biosample life stage    Biosample sex  \\\n",
       "File accession                                                        \n",
       "ENCFF673KYR      primary cell       unknown, fetal  unknown, female   \n",
       "ENCFF262OBL      primary cell                adult             male   \n",
       "ENCFF060LPA      primary cell                adult             male   \n",
       "ENCFF289HGQ      primary cell              newborn     male, female   \n",
       "\n",
       "               Biosample organism Biosample treatments  \\\n",
       "File accession                                           \n",
       "ENCFF673KYR          Homo sapiens                        \n",
       "ENCFF262OBL          Homo sapiens                        \n",
       "ENCFF060LPA          Homo sapiens                        \n",
       "ENCFF289HGQ          Homo sapiens                        \n",
       "\n",
       "               Biosample subcellular fraction term name Biosample phase  \\\n",
       "File accession                                                            \n",
       "ENCFF673KYR                                                               \n",
       "ENCFF262OBL                                                               \n",
       "ENCFF060LPA                                                               \n",
       "ENCFF289HGQ                                                               \n",
       "\n",
       "               Biosample synchronization stage Biosample Age  \n",
       "File accession                                                \n",
       "ENCFF673KYR                                                   \n",
       "ENCFF262OBL                                          52 year  \n",
       "ENCFF060LPA                                          23 year  \n",
       "ENCFF289HGQ                                                   "
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metaM.fillna('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Pandas写入文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 写入文本文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "metaM.to_csv(\"pandas_data/meta2.tsv\", sep=\"\\t\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ens2syn.to_csv(\"pandas_data/gencode.v24.ENS2SYN\", sep=\"\\t\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "TPM_mat.to_csv(\"pandas_data/TPM\", sep=b'\\t', float_format=\"%.2f\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 写入HDF5文件\n",
    "\n",
    "HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data. HDF5 is portable and is extensible, allowing applications to evolve in their use of HDF5. The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format. <https://support.hdfgroup.org/HDF5/>\n",
    "\n",
    "使用优势是把处理好的数据以二进制文件存取，既可以减少文件数目、压缩使用空间，又可以方便多次快速读取，并且可以在不同的程序语言如Python与R中共同使用。\n",
    "\n",
    "HDF5文件的写入形式上类似于字典操作，其读取也是。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/MPATHB/soft/anacond/lib/python2.7/site-packages/IPython/core/interactiveshell.py:3035: PerformanceWarning: \n",
      "your performance may suffer as PyTables will pickle object types that it cannot\n",
      "map directly to c-types [inferred_type->mixed,key->block0_values] [items->['Biosample term id', 'Biosample term name', 'Biosample type', 'Biosample life stage', 'Biosample sex', 'Biosample organism', 'Biosample Age']]\n",
      "\n",
      "  exec(code_obj, self.user_global_ns, self.user_ns)\n"
     ]
    }
   ],
   "source": [
    "# 写入模式打开一个HDF5文件,使用压缩格式以节省空间\n",
    "store = pd.HDFStore(\"pandas_data/ENCODE.hdf5\", \"w\", complib=str(\"zlib\"), complevel=9)\n",
    "\n",
    "# 写入表达矩阵\n",
    "store[\"TPM\"] = TPM_mat\n",
    "store[\"FPKM\"] = FPKM_mat\n",
    "\n",
    "# 写入注释文件\n",
    "store['ens2syn'] = ens2syn\n",
    "store['meta'] = metaM\n",
    "\n",
    "# 关闭HDF5句柄\n",
    "store.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当数据中存在混合数据模式时，会出现上面的Warning，对于我们的数据只要把`metaM`中的`NaN`值替换掉就可以。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 写入模式打开一个HDF5文件,使用压缩格式已节省空间\n",
    "store = pd.HDFStore(\"pandas_data/ENCODE.hdf5\", \"w\", complib=str(\"zlib\"), complevel=9)\n",
    "\n",
    "# 写入表达矩阵\n",
    "store[\"TPM\"] = TPM_mat\n",
    "store[\"FPKM\"] = FPKM_mat\n",
    "\n",
    "# 写入注释文件\n",
    "store['ens2syn'] = ens2syn\n",
    "store['meta'] = metaM.fillna('')\n",
    "\n",
    "# 关闭HDF5句柄\n",
    "store.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 读取HDF5文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store = pd.HDFStore(\"pandas_data/ENCODE.hdf5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/FPKM', '/TPM', '/ens2syn', '/meta']"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 列出HDF5文件的索引名字\n",
    "store.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <th>ENCFF262OBL</th>\n",
       "      <th>ENCFF673KYR</th>\n",
       "      <th>ENCFF060LPA</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000000003.14</th>\n",
       "      <td>1.03</td>\n",
       "      <td>17.13</td>\n",
       "      <td>2.42</td>\n",
       "      <td>6.64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000419.12</th>\n",
       "      <td>1.45</td>\n",
       "      <td>18.86</td>\n",
       "      <td>1.80</td>\n",
       "      <td>9.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000000457.13</th>\n",
       "      <td>0.24</td>\n",
       "      <td>2.48</td>\n",
       "      <td>0.38</td>\n",
       "      <td>0.86</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF289HGQ  ENCFF262OBL  ENCFF673KYR  ENCFF060LPA\n",
       "gene_id                                                               \n",
       "ENSG00000000003.14         1.03        17.13         2.42         6.64\n",
       "ENSG00000000419.12         1.45        18.86         1.80         9.91\n",
       "ENSG00000000457.13         0.24         2.48         0.38         0.86"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TPM_mat = store['TPM']\n",
    "TPM_mat.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ens2syn = store['ens2syn']\n",
    "meta = store['meta']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "store.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PANDAS矩阵的小应用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "利用上面的矩阵操作，选取这两个基因相关的信息并绘制表达谱"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "targetL = ['KRIT1','AK2']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gene_symbol转换为Gene_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gene_symbol</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000004455.16</th>\n",
       "      <td>AK2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000001631.14</th>\n",
       "      <td>KRIT1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   gene_symbol\n",
       "gene_id                       \n",
       "ENSG00000004455.16         AK2\n",
       "ENSG00000001631.14       KRIT1"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ensID = ens2syn[ens2syn[\"gene_symbol\"].isin(targetL)]\n",
    "ensID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "提取目标基因的表达"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <th>ENCFF262OBL</th>\n",
       "      <th>ENCFF673KYR</th>\n",
       "      <th>ENCFF060LPA</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENSG00000001631.14</th>\n",
       "      <td>1.15</td>\n",
       "      <td>13.36</td>\n",
       "      <td>1.37</td>\n",
       "      <td>6.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENSG00000004455.16</th>\n",
       "      <td>2.31</td>\n",
       "      <td>37.62</td>\n",
       "      <td>8.95</td>\n",
       "      <td>15.57</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ENCFF289HGQ  ENCFF262OBL  ENCFF673KYR  ENCFF060LPA\n",
       "gene_id                                                               \n",
       "ENSG00000001631.14         1.15        13.36         1.37         6.21\n",
       "ENSG00000004455.16         2.31        37.62         8.95        15.57"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "targetExpr = TPM_mat[TPM_mat.index.isin(ensID.index)]\n",
    "targetExpr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "重命名矩阵的索引"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'gene_symbol': {'ENSG00000001631.14': 'KRIT1', 'ENSG00000004455.16': 'AK2'}}"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ensID_dict = ensID.to_dict()\n",
    "ensID_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <th>ENCFF262OBL</th>\n",
       "      <th>ENCFF673KYR</th>\n",
       "      <th>ENCFF060LPA</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gene_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>KRIT1</th>\n",
       "      <td>1.15</td>\n",
       "      <td>13.36</td>\n",
       "      <td>1.37</td>\n",
       "      <td>6.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AK2</th>\n",
       "      <td>2.31</td>\n",
       "      <td>37.62</td>\n",
       "      <td>8.95</td>\n",
       "      <td>15.57</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         ENCFF289HGQ  ENCFF262OBL  ENCFF673KYR  ENCFF060LPA\n",
       "gene_id                                                    \n",
       "KRIT1           1.15        13.36         1.37         6.21\n",
       "AK2             2.31        37.62         8.95        15.57"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "targetExpr = targetExpr.rename(index=ensID_dict['gene_symbol'])\n",
    "targetExpr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "转置矩阵以增加META信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>gene_id</th>\n",
       "      <th>KRIT1</th>\n",
       "      <th>AK2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <td>1.15</td>\n",
       "      <td>2.31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF262OBL</th>\n",
       "      <td>13.36</td>\n",
       "      <td>37.62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF673KYR</th>\n",
       "      <td>1.37</td>\n",
       "      <td>8.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF060LPA</th>\n",
       "      <td>6.21</td>\n",
       "      <td>15.57</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "gene_id      KRIT1    AK2\n",
       "ENCFF289HGQ   1.15   2.31\n",
       "ENCFF262OBL  13.36  37.62\n",
       "ENCFF673KYR   1.37   8.95\n",
       "ENCFF060LPA   6.21  15.57"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "targetExpr_t = targetExpr.T\n",
    "targetExpr_t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从meta矩阵中提取4列信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "meta_type = [\"Biosample term name\",\"Biosample type\", \"Biosample life stage\", \"Biosample sex\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Biosample term name</th>\n",
       "      <th>Biosample type</th>\n",
       "      <th>Biosample life stage</th>\n",
       "      <th>Biosample sex</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>File accession</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENCFF673KYR</th>\n",
       "      <td>mesangial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>unknown, fetal</td>\n",
       "      <td>unknown, female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF262OBL</th>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF060LPA</th>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <td>fibroblast of villous mesenchyme</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>newborn</td>\n",
       "      <td>male, female</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              Biosample term name Biosample type  \\\n",
       "File accession                                                     \n",
       "ENCFF673KYR                        mesangial cell   primary cell   \n",
       "ENCFF262OBL     pulmonary artery endothelial cell   primary cell   \n",
       "ENCFF060LPA     pulmonary artery endothelial cell   primary cell   \n",
       "ENCFF289HGQ      fibroblast of villous mesenchyme   primary cell   \n",
       "\n",
       "               Biosample life stage    Biosample sex  \n",
       "File accession                                        \n",
       "ENCFF673KYR          unknown, fetal  unknown, female  \n",
       "ENCFF262OBL                   adult             male  \n",
       "ENCFF060LPA                   adult             male  \n",
       "ENCFF289HGQ                 newborn     male, female  "
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meta = meta[meta_type]\n",
    "meta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "修改下矩阵信息，去除`unknow,`字符串（只是为了展示方便）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Biosample term name</th>\n",
       "      <th>Biosample type</th>\n",
       "      <th>Biosample life stage</th>\n",
       "      <th>Biosample sex</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>File accession</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENCFF673KYR</th>\n",
       "      <td>mesangial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>fetal</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF262OBL</th>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF060LPA</th>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <td>fibroblast of villous mesenchyme</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>newborn</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              Biosample term name Biosample type  \\\n",
       "File accession                                                     \n",
       "ENCFF673KYR                        mesangial cell   primary cell   \n",
       "ENCFF262OBL     pulmonary artery endothelial cell   primary cell   \n",
       "ENCFF060LPA     pulmonary artery endothelial cell   primary cell   \n",
       "ENCFF289HGQ      fibroblast of villous mesenchyme   primary cell   \n",
       "\n",
       "               Biosample life stage Biosample sex  \n",
       "File accession                                     \n",
       "ENCFF673KYR                   fetal        female  \n",
       "ENCFF262OBL                   adult          male  \n",
       "ENCFF060LPA                   adult          male  \n",
       "ENCFF289HGQ                 newborn        female  "
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meta.loc['ENCFF673KYR',\"Biosample life stage\"] = \"fetal\"\n",
    "# Much faster\n",
    "meta = meta.set_value('ENCFF673KYR','Biosample sex','female')\n",
    "meta = meta.set_value('ENCFF289HGQ','Biosample sex','female')\n",
    "meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>gene_id</th>\n",
       "      <th>KRIT1</th>\n",
       "      <th>AK2</th>\n",
       "      <th>Biosample term name</th>\n",
       "      <th>Biosample type</th>\n",
       "      <th>Biosample life stage</th>\n",
       "      <th>Biosample sex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <td>1.15</td>\n",
       "      <td>2.31</td>\n",
       "      <td>fibroblast of villous mesenchyme</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>newborn</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF262OBL</th>\n",
       "      <td>13.36</td>\n",
       "      <td>37.62</td>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF673KYR</th>\n",
       "      <td>1.37</td>\n",
       "      <td>8.95</td>\n",
       "      <td>mesangial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>fetal</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF060LPA</th>\n",
       "      <td>6.21</td>\n",
       "      <td>15.57</td>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "gene_id      KRIT1    AK2                Biosample term name Biosample type  \\\n",
       "ENCFF289HGQ   1.15   2.31   fibroblast of villous mesenchyme   primary cell   \n",
       "ENCFF262OBL  13.36  37.62  pulmonary artery endothelial cell   primary cell   \n",
       "ENCFF673KYR   1.37   8.95                     mesangial cell   primary cell   \n",
       "ENCFF060LPA   6.21  15.57  pulmonary artery endothelial cell   primary cell   \n",
       "\n",
       "gene_id     Biosample life stage Biosample sex  \n",
       "ENCFF289HGQ              newborn        female  \n",
       "ENCFF262OBL                adult          male  \n",
       "ENCFF673KYR                fetal        female  \n",
       "ENCFF060LPA                adult          male  "
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_expr_meta = targetExpr_t.join(meta, how=\"left\")\n",
    "target_expr_meta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "重名了列名字(替换掉名字中的空格)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>gene_id</th>\n",
       "      <th>KRIT1</th>\n",
       "      <th>AK2</th>\n",
       "      <th>Biosample_term_name</th>\n",
       "      <th>Biosample_type</th>\n",
       "      <th>Biosample_life_stage</th>\n",
       "      <th>Biosample_sex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ENCFF289HGQ</th>\n",
       "      <td>1.15</td>\n",
       "      <td>2.31</td>\n",
       "      <td>fibroblast of villous mesenchyme</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>newborn</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF262OBL</th>\n",
       "      <td>13.36</td>\n",
       "      <td>37.62</td>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF673KYR</th>\n",
       "      <td>1.37</td>\n",
       "      <td>8.95</td>\n",
       "      <td>mesangial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>fetal</td>\n",
       "      <td>female</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ENCFF060LPA</th>\n",
       "      <td>6.21</td>\n",
       "      <td>15.57</td>\n",
       "      <td>pulmonary artery endothelial cell</td>\n",
       "      <td>primary cell</td>\n",
       "      <td>adult</td>\n",
       "      <td>male</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "gene_id      KRIT1    AK2                Biosample_term_name Biosample_type  \\\n",
       "ENCFF289HGQ   1.15   2.31   fibroblast of villous mesenchyme   primary cell   \n",
       "ENCFF262OBL  13.36  37.62  pulmonary artery endothelial cell   primary cell   \n",
       "ENCFF673KYR   1.37   8.95                     mesangial cell   primary cell   \n",
       "ENCFF060LPA   6.21  15.57  pulmonary artery endothelial cell   primary cell   \n",
       "\n",
       "gene_id     Biosample_life_stage Biosample_sex  \n",
       "ENCFF289HGQ              newborn        female  \n",
       "ENCFF262OBL                adult          male  \n",
       "ENCFF673KYR                fetal        female  \n",
       "ENCFF060LPA                adult          male  "
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "oriColnames = target_expr_meta.columns.values\n",
    "nameD = dict([(i,i.replace(' ','_')) for i in oriColnames])\n",
    "target_expr_meta.rename(columns=nameD, inplace=True)\n",
    "target_expr_meta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "绘制散点图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fbcaefc0c10>"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/MPATHB/soft/anacond/lib/python2.7/site-packages/matplotlib/collections.py:590: FutureWarning:\n",
      "\n",
      "elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAG8FJREFUeJzt3XtwVOXBx/HfhiREwibrSigmGi4qwiwNgQHsKG0gWsdq\nGKNTttN4x+oYcKR0ikQ6AlapUBoDHYNYdbyOI9tOTbHjIJ1CAkWnJDYIXYRKa9J2MhJDkk0ChGSz\n+/7hy74CTy7LS87Jge9nxulezp7zWzvmt8959jzrikajUQEAcIYEuwMAAIYmCgIAYERBAACMKAgA\ngBEFAQAwoiAAAEaJVh4sEomopKREXq9XJSUl6ujoUFlZmZqampSRkaElS5YoNTXVykgAgF5YOoJ4\n//33dcUVV8jlckmSKioqlJOTow0bNmjKlCmqqKgY0H6CweBgxhx0Ts7v5OwS+e1GfnvFm9+ygjh6\n9Khqa2uVn5+vU9fm1dTUKC8vT5I0Z84cVVdXD2hfF9v/SUOJk7NL5Lcb+e01ZAvi9ddf1913362E\nhP87ZCgUksfjkSSlp6crFApZFQcA0A9LCuLjjz9WWlqaxo8fr95W9jh12gkAMDS4rFiL6e2339au\nXbuUkJCg7u5unThxQrNmzdI///lPrVq1Sh6PRy0tLXrqqae0fv36s14fDAZPGxr5/f7BjgwAF6RA\nIBC77fP55PP5et3WkoL4ugMHDmjLli0qKSnRW2+9pZEjR6qwsFAVFRU6duyY7rrrrgHtp6GhYZCT\nDh6326329na7Y5wTJ2eXyG838tsrMzMzru1tuQ7i1OmkwsJC7d+/X4sXL9bf//53FRYW2hEHAGBg\n+QjifGEEYQ8nZ5fIbzfy28sRIwgAwNBHQQAAjCgIAIARBQEAMKIgAABGFAQAwIiCAAAHaO+Oqr3b\n2qsSLP09CABA/A41d2tNVb0kqSRvrK71JllyXEYQADCEtXdHtaaqXs3Hw2o+HtaaqnrLRhIUBADA\niIIAgCHMneRSSd5YeUckyjsiUSV5Y+VOsubnEZiDAIAh7lpvkn5dcJUkWVYOEgUBAI5gZTGcwikm\nAIARBQEAMKIgAABGFAQAwIiCAAAYURAAACPLvuba1dWlVatWqbu7W5FIRNddd538fr8CgYC2b9+u\ntLQ0SVJRUZFyc3OtigUA6IVlBZGcnKyVK1dq+PDh6unp0YoVKzRt2jS5XC4VFBSooKDAqigAgAGw\n9BTT8OHDJUnhcFjhcFgu11cXfkSj1i5hCwDon6VXUkciES1btkxHjhzRLbfcoquvvlq1tbXaunWr\ndu7cqQkTJujee+9VamqqlbEAAAauqA0f348fP65169ZpwYIFSktLi80/bN68WS0tLSouLu53Hw0N\nDYMdc9C43W61t7fbHeOcODm7RH67kd9emZmZcW1vy1pMI0aMkM/n0969ezVv3rzY4/n5+Vq7du1Z\n2weDQQWDwdh9v98vt9ttSdbBkJyc7Nj8Ts4ukd9u5LdfIBCI3fb5fPL5fL1ua1lBtLW1adiwYUpN\nTVVXV5f279+v22+/Xa2trfJ4PJKkPXv2KDs7+6zXmt6Ek1vcyZ9CnJxdIr/dyG8vt9stv98/4O0t\nK4jW1laVl5crEokoEono+uuv1/Tp0/X888+rrq5OLpdLGRkZevjhh62KBADogy1zEOcDcxD2cHJ2\nifx2I7+94p2D4EpqAIARBQEAMKIgAABGFAQAwIiCAAAYURAAACMKAgBgREEAAIwoCACAEQUBADCi\nIAAARhQEAMCIggAAGFEQAAAjCgIAYERBAACMKAgAgBEFAQAwoiAAAEaJVhykq6tLq1atUnd3tyKR\niK677jr5/X51dHSorKxMTU1NysjI0JIlS5SammpFJABAPywpiOTkZK1cuVLDhw9XT0+PVqxYoWnT\npumvf/2rcnJydPvtt6uiokIVFRW66667rIgEAOiHZaeYhg8fLkkKh8MKh8NyuVyqqalRXl6eJGnO\nnDmqrq62Kg4AoB+WjCAkKRKJaNmyZTpy5IhuueUWXX311QqFQvJ4PJKk9PR0hUIhq+IAAPphWUEk\nJCRo3bp1On78uNatW6d///vfpz3vcrl6fW0wGFQwGIzd9/v9crvdg5Z1sCUnJzs2v5OzS+S3G/nt\nFwgEYrd9Pp98Pl+v21pWEKeMGDFCPp9Pn3zyidLT09Xa2iqPx6OWlhalp6cbX2N6E+3t7VbEHRRu\nt9ux+Z2cXSK/3chvL7fbLb/fP+DtLZmDaGtr07FjxyR99Y2m/fv3KysrSzNmzFBlZaUkqaqqSjNn\nzrQiDgBgACwZQbS2tqq8vFyRSESRSETXX3+9pk+frokTJ6qsrEw7duyIfc0VADA0uKLRaNTuEOei\noaHB7gjnzMnDVCdnl8hvN/LbKzMzM67tuZIaAGBEQQAAjCgIAIARBQEAMKIgAABGFAQAwIiCAAAY\nURAAACMKAgBgREEAAIwoCACAEQUBADCiIAAARhQEAMCIggAAGFEQAAAjCgIAYERBAACMKAgAgFGi\nVQdqampSeXm5QqGQXC6XbrzxRt16660KBALavn270tLSJElFRUXKzc21KhYAoBeWFURiYqLuu+8+\njRs3Tp2dnVq2bJlycnLkcrlUUFCggoICq6IAAAbAsoLweDzyeDySpJSUFGVlZam5uVmSFI1GrYoB\nABggW+YgGhsbVVdXp4kTJ0qStm7dqqVLl+qFF17QsWPH7IgEADiDK2rxx/fOzk6tWrVKd955p2bN\nmqVQKBSbf9i8ebNaWlpUXFx82muCwaCCwWDsvt/vV3t7u5Wxz6vk5GR1dXXZHeOcODm7RH67kd9e\nbrdbgUAgdt/n88nn8/W6vaUFEQ6HtXbtWuXm5uq222476/nGxkatXbtWpaWl/e6roaFhMCJawu12\nO7bgnJxdIr/dyG+vzMzMuLa37BRTNBrVpk2blJWVdVo5tLS0xG7v2bNH2dnZVkUCAPTBsknqQ4cO\nadeuXcrOztbjjz8uSfrhD3+o3bt3q66uTi6XSxkZGXr44YetigQA6INlBTFp0iRt3rz5rMenTZtm\nVQQAQBy4khoAYERBAACMKAgAgBEFAQAwoiAAAEYUBADAiIIAABhREAAAIwoCAGBEQQAAjCgIAIAR\nBQEAMKIgAABGFAQAwKjfgvjyyy+1Z88e4y+4/eUvfxmUUAAA+/X5exB79+5VWVmZRo8erYaGBs2Z\nM0cPPvigEhK+6pXf/OY3mj17tiVBAQDW6nME8fbbb2vx4sVat26dysvL9cUXX+iXv/yluru7rcoH\nALBJnwVx5MgRTZ8+XZLk8Xj0xBNPKCUlRc8++6w6OzstCQgAsEefBTFy5Eg1NTXF7icmJuqxxx7T\nqFGj9PTTTysSiQx6QACAPfqcg5gyZYp27Nih+fPnxx5LSEhQcXGxXnrpJR0+fHjAB2pqalJ5eblC\noZBcLpduvPFG3Xrrrero6FBZWZmampqUkZGhJUuWKDU19dzfEQDgvHBFo9Fob0+Gw2H19PRo+PDh\nxuebmpo0atSoAR2otbVVra2tGjdunDo7O7Vs2TItXbpUlZWVcrvduv3221VRUaFjx47prrvu6nd/\npm9VOYXb7VZ7e7vdMc6Jk7NL5Lcb+e2VmZkZ1/Z9nmLaunVrr+Vw8uRJvfDCCwM+kMfj0bhx4yRJ\nKSkpysrKUnNzs2pqapSXlydJmjNnjqqrqwe8TwDA4OmzILZs2aJt27ad9fiJEye0evXqcz5oY2Oj\n6urqdM011ygUCsnj8UiS0tPTFQqFznm/AIDzp885iCeffFI///nPlZiYqPz8fEnSsWPHtHr1arnd\nbv30pz+N+4CdnZ0qLS3V/fffr0suueS051wul/E1wWBQwWAwdt/v98vtdsd97KEiOTnZsfmdnF0i\nv93Ib79AIBC77fP55PP5et22z4K48sortXz5cj3zzDNKSkpSbm6unnnmGV122WX6yU9+osTEPl9+\nlnA4rNLSUn3nO9/RrFmzJH01amhtbZXH41FLS4vS09PPep3pTTj5PKCTz2M6ObtEfruR315ut1t+\nv3/A2/f7F378+PF64okn9Itf/EKpqakaP368Fi9erGHDhsUVLBqNatOmTcrKytJtt90We3zGjBmq\nrKxUYWGhqqqqNHPmzLj2CwAYHH0WxDvvvCOXy6VoNKoJEybos88+05gxY/S73/1O0WhULpdLP/jB\nDwZ0oEOHDmnXrl3Kzs7W448/LkkqKipSYWGhysrKtGPHjtjXXAEA9uuzII4ePRqbF/B6vZo1a5Za\nW1tjz8dzodykSZO0efNm43NPPvnkgPcDALBGnwWxaNEi4+P19fWqqqpiNVcAuIANeJa5ra1Nu3bt\nUlVVlerr6zVp0iQ98MADg5kNAGCjPgsiHA6rpqZGlZWV+uSTTzRmzBjdcMMN+vLLL7VkyZLY9QsA\ngAtPnwXx0EMPKSEhQXl5efL7/ZowYYIkadu2bb1eswAAuDD0eSX12LFj1dHRoc8++0yHDx9WR0eH\nVbkAADbrcwSxatUqNTY2aufOnXrvvff06quvaurUqers7FQ4HLYqIwDABn2u5nqmgwcPqrKyUh99\n9JGGDRumuXPn6p577hnMfL1iNVd7ODm7RH67kd9e8a7mGtdaGZMmTdKkSZO0YMEC7dmzRzt37ozr\nYAAA54hvMaX/lZycrNmzZ2v27NnnOw8AYIjoc5IaAHDxoiAAAEYUBADAiIIAABhREAAAIwoCAGBE\nQQAAjCgIAIARBYGLWnt3VO3dA15tBrionNOV1MCF4FBzt9ZU1UuSSvLG6lpvks2JgKHFsoLYuHGj\namtrlZaWptLSUklSIBDQ9u3blZaWJkkqKipSbm6uVZFwEWvvjmpNVb2aj3+1KvGaqnr9uuAquZP4\nnRPgFMsKYu7cufre976n559/PvaYy+VSQUGBCgoKrIoBABggy+YgJk+erNTU1LMej2O1ceC8cSe5\nVJI3Vt4RifKOSFRJ3lhGD8AZbJ+D2Lp1q3bu3KkJEybo3nvvNZYIMBiu9Sbp1wVXSRLlABjYWhA3\n33yzvv/970uSNm/erDfeeEPFxcVnbRcMBhUMBmP3/X6/3G63ZTnPt+TkZMfmd3J26ez8TnsnF9q/\nf6dxen7pq7nfU3w+n3w+X6/b2loQ6enpsdv5+flau3atcTvTm3Dyrzo5+VepnJxdIr/dyG8vt9st\nv98/4O1tvQ6ipaUldnvPnj3Kzs62MQ0A4OssG0GsX79en376qdra2lRcXKz58+frwIEDqqurk8vl\nUkZGhh5++GGr4gAA+mFZQfz4xz8+67H8/HyrDg8AiBNLbQAAjCgIAIARBQEAMKIgAABGFAQAwIiC\nAAAYURAAACMKAgBgREEAAIwoCACAEQUBADCiIAAARhQEAMCIggAAGFEQAAAjCgIAYERBAACMKAgA\ngBEFAQAwsuw3qTdu3Kja2lqlpaWptLRUktTR0aGysjI1NTUpIyNDS5YsUWpqqlWRAAB9sGwEMXfu\nXC1fvvy0xyoqKpSTk6MNGzZoypQpqqiosCoOAKAflhXE5MmTzxod1NTUKC8vT5I0Z84cVVdXWxUH\nANAPW+cgQqGQPB6PJCk9PV2hUMjOOACAr7FsDqI/Lper1+eCwaCCwWDsvt/vl9vttiLWoEhOTnZs\nfidnl8hvN/LbLxAIxG77fD75fL5et7W1INLT09Xa2iqPx6OWlhalp6cbtzO9ifb2disiDgq32+3Y\n/E7OLpHfbuS3l9vtlt/vH/D2tp5imjFjhiorKyVJVVVVmjlzpp1xAABfY9kIYv369fr000/V1tam\n4uJi+f1+FRYWqqysTDt27Ih9zdUJ2rujkiR3Uu+nxQDA6VzRaDRqd4hz0dDQYMtxDzV3a01VvSSp\nJG+srvUmxb0PJw9TnZxdIr/dyG+vzMzMuLbnSuo4tHdHtaaqXs3Hw2o+HtaaqvrYaAIALjQUBADA\niIKIgzvJpZK8sfKOSJR3RKJK8sYyDwHggjVkroNwimu9Sfp1wVWSmKQGcGGjIM4BxQDgYsApJgCA\nEQUBADCiIAAARhQEAMCIggAAGFEQAAAjCgIAYERBAACMKAgAgBEFAQAwoiAAAEYUBADAiIIAABhR\nEAAAoyGx3PeiRYt0ySWXKCEhQcOGDdOzzz5rdyQAuOgNiYKQpFWrVmnkyJF2xwAA/K8hc4opGo3a\nHQEA8DVDYgThcrn09NNPKyEhQTfddJNuuukmuyMBwEXPFR0CH91bWlp06aWXqq2tTU8//bQWLFig\nyZMnx54PBoMKBoOx+36/X+3t7XZEPS+Sk5PV1dVld4xz4uTsEvntRn57ud1uBQKB2H2fzyefz9fr\n9kOiIL7ut7/9rVJSUjRv3rw+t2toaLAo0fnndrsdW3BOzi6R327kt1dmZmZc29s+B3Hy5EmdOHFC\nktTZ2al9+/YpOzvb5lQAANvnIEKhkNatWydJikQimj17tqZOnWpzKgCA7QUxevToWEEAAIYO208x\nAQCGJgoCAGBEQQAAjCgIAIARBQEAMKIgAABGFAQAwIiC6EN7d1Tt3UNqJRIAsIztF8oNVYeau7Wm\nql6SVJI3Vtd6k2xOBADWYgRh0N4d1ZqqejUfD6v5eFhrquoZSQC46FAQAAAjCsLAneRSSd5YeUck\nyjsiUSV5Y+VOctkdCwAsxRxEL671JunXBVdJEuUA4KJEQfSBYgBwMeMUEwDAiIIAABhREAAAIwoC\nAGA0JCap9+7dq9dee02RSET5+fkqLCy0OxIAXPRsH0FEIhG98sorWr58uZ577jnt3r1b//3vf+2O\nBQAXPdsL4vDhwxozZoxGjx6txMRE3XDDDaqpqbE7FgBc9GwviObmZl122WWx+16vV83NzTYmAgBI\nQ6AgAABDk+2T1F6vV0ePHo3dP3r0qLxe72nbBINBBYPB2H2/36/MzEzLMg4Gt9ttd4Rz5uTsEvnt\nRn57BQKB2G2fzyefz9frtraPIK666ip98cUXamxsVDgc1ocffqgZM2acto3P55Pf74/98/U36ERO\nzu/k7BL57UZ+ewUCgdP+lvZVDtIQGEEMGzZMCxYs0OrVq2Nfc73iiivsjgUAFz3bC0KSpk2bpmnT\nptkdAwDwNbafYjoX/Q2Lhjon53dydon8diO/veLN74pGo/yWJgDgLI4cQQAABh8FAQAwGhKT1APl\n5EX9mpqaVF5erlAoJJfLpRtvvFG33nqr3bHiFolEVFJSIq/Xq5KSErvjxOXYsWPatGlTbK2v4uJi\nTZw40eZUA/fHP/5RO3bskCRlZ2dr4cKFSkpKsjmV2caNG1VbW6u0tDSVlpZKkjo6OlRWVqampiZl\nZGRoyZIlSk1NtTmpmSn/m2++qb/97W9KTEzUN77xDS1cuFAjRoywOamZKf8p7733nt566y298sor\nGjlyZJ/7ccwIwumL+iUmJuq+++7Tc889p9WrV+uDDz5wVP5T3n//fV1xxRVyuZz3c6yvvvqqpk2b\nprKyMv3qV79y1Nepm5ubtXXrVq1Zs0alpaWKRCLavXu33bF6NXfuXC1fvvy0xyoqKpSTk6MNGzZo\nypQpqqiosCld/0z5p06dqtLSUq1bt06XX3653n33XZvS9c+UX/rqg+q+ffs0atSoAe3HMQXh9EX9\nPB6Pxo0bJ0lKSUlRVlaWWlpa7A0Vp6NHj6q2tlb5+fly2ncbjh8/roMHDyo/P1/SV9ffDNVPf73p\n6enRyZMnY/975ooDQ8nkyZPPGh3U1NQoLy9PkjRnzhxVV1fbEW1ATPlzcnKUkPDVn8xrrrnmtBUg\nhhpTfkl64403dPfddw94P445xWRa1O/w4cM2Jjp3jY2Nqqur0zXXXGN3lLi8/vrruvvuu3XixAm7\no8StsbFRaWlp2rhxo+rr6zV+/Hg98MADGj58uN3RBsTr9WrevHlauHChkpOTNXXqVOXk5NgdKy6h\nUEgej0eSlJ6erlAoZHOic7d9+3bNnj3b7hhxqa6ultfr1dixYwf8GseMIC4UnZ2deu6553T//fcr\nJSXF7jgD9vHHHystLU3jx4933OhB+urT9+eff66bb75Za9euVUpKypA+xXGmjo4O1dTUqLy8XC++\n+KI6Ozu1a9cuu2OdMyeeojzl97//vRITEx1VECdPntS7774rv98fe2wg/x07piAGsqjfUBcOh1Va\nWqpvf/vbmjVrlt1x4nLo0CF9/PHHWrRokTZs2KBgMKjnn3/e7lgDdtlll8nr9erqq6+WJH3rW9/S\n559/bnOqgdu/f79Gjx4tt9utYcOG6brrrtOhQ4fsjhWX9PR0tba2SpJaWlqUnp5uc6L4VVZWqra2\nVo899pjdUeJy5MgRffnll1q6dKkWLVqk5uZmlZSU9DuKc8wppq8v6uf1evXhhx9q8eLFdscasGg0\nqk2bNikrK0u33Xab3XHiVlRUpKKiIknSgQMHtGXLFj366KM2pxo4j8ejUaNGqaGhQZmZmdq3b5+j\nJqkzMjL02WefqaurS0lJSdq3b1+s7JxixowZqqysVGFhoaqqqjRz5ky7I8Vl79692rJli1atWqXk\n5GS748QlOztbL730Uuz+okWLtHbt2n6/xeSoK6lra2tP+5rrHXfcYXekATt48KBWrlyp7Ozs2PC6\nqKhIubm5NieL34EDB/Tee+9p2bJldkeJS11dnV588UWFw+Eh/zVFk0AgoI8++kgJCQkaP368Hnnk\nESUmDs3PeOvXr9enn36qtrY2eTwe+f1+zZw50zFfcz0z//z581VRUaFwOBz7ozpx4kT96Ec/sjmp\n2an87e3tSk9Pl9/v19y5c2PPP/roo1qzZs2FVRAAAOs4Zg4CAGAtCgIAYERBAACMKAgAgBEFAQAw\noiAAAEYUBADAaGheZQPYZNGiRXrkkUf0zW9+U5K0e/duvfzyy1q6dKmeeuqp2OJ+brdb3/3ud0/7\nTZJTrz18+HBsKeienh719PTErrzNyMhQaWmp3nnnHVVXV6uhoUF33nmn5s+fb/E7BfpHQQBnOHWl\ne2Vlpd5880098cQTsVVIX3vtNSUkJOhf//qXVq5cqQkTJpy2qqrL5dIdd9wRu8q/srJSO3bs0FNP\nPXXaMS6//HLdc8892rZtm6MXrsOFjVNMwBmi0aj+9Kc/6c0339TPfvYz46/OTZgwQVdeeaXq6+sH\ntL8z5eXlKTc3V5dccokjV8fFxYGCAM7wwQcfKBAIxEYIJv/4xz/0n//8R2PGjLE4HWAdTjEBZ9i/\nf7+mTJmiK6+88qznHnzwQXV3d6u7u1vz5s1z3IqkQDwYQQBneOihh9TQ0KBNmzad9dwrr7yiN954\nQ/fcc4+CwaDC4bANCQFrUBDAGTwej1asWKGDBw/q5ZdfPuv5hIQEFRQUKCkpSdu2bft/H49JagxV\nFARgcOmll+rJJ5/U3r179frrrxu3KSws1B/+8Ad1d3fHvf+enh51dXUpEokoHA7HbgNDCXMQQC9G\njRqlFStWaOXKlbGfyvy66dOna+TIkfrzn/+sW265pdf9mEYImzZt0s6dO2P33333XS1cuFB5eXnn\nJzxwHvCDQQAAI04xAQCMKAgAgBEFAQAwoiAAAEYUBADAiIIAABhREAAAIwoCAGBEQQAAjP4HzHcw\nY8fIVDMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fbcaf1c5650>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "target_expr_meta.plot.scatter(x='KRIT1', y='AK2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "绘制箱线图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/MPATHB/soft/anacond/lib/python2.7/site-packages/IPython/kernel/__main__.py:1: FutureWarning:\n",
      "\n",
      "\n",
      "The default value for 'return_type' will change to 'axes' in a future release.\n",
      " To use the future behavior now, set return_type='axes'.\n",
      " To keep the previous behavior and silence this warning, set return_type='dict'.\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEECAYAAAAxqm/oAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFfhJREFUeJzt3X9sVfX9x/HXaZtbodwfdpZpaxrG1EmuqQX5EdEIVrMw\nrdINvd+MbkLMJOtMho1COwbahC1Lw7pKZgXn3MBt34VrwMoCYRkpBSlxFK2OXQwJIc0yiHalvbcU\nKKXcfv/w2ztqf9ze29N776d9Pv4699x7et69+fDi08855/Ox+vv7+wUAME5asgsAAMSHAAcAQxHg\nAGAoAhwADEWAA4ChCHAAMFTGWD4UDodVWVmp7OxsVVZWqru7W7W1tWpvb1dOTo7Ky8uVlZU10bUC\nAG4wph74/v37dfvtt8uyLElSfX29CgoKtHXrVt1zzz2qr6+f0CIxVCAQSHYJwIhon4kRNcAvXLig\nlpYWFRUVaeCZnxMnTmjJkiWSpKVLl6q5uXliq8QQ/ANBKqN9JkbUAN+5c6e+973vKS3tvx8NhULy\neDySJLfbrVAoNHEVAgCGNWqAf/jhh3K5XPra176mkZ64HxhWAQAk1qgXMU+fPq0PP/xQLS0tunbt\nmq5cuaJf//rXcrvdCgaD8ng86uzslNvtHvb4QCAw6E8pn89nb/VTGN8lUhnt035+vz+y7fV65fV6\nZY11MqtTp05p7969qqys1B//+EfNmDFDJSUlqq+v16VLl1RaWjqmIs6fPx9f9RjE6XTq4sWLyS4D\nGBbt0165ubnD7o/pPvCB4ZKSkhKdPHlSa9eu1T//+U+VlJSMv0IAQEzG3AO3Cz1we9DDQSqjfdrL\nlh44ACB1EOAAYCgCHAAMRYADgKEIcAAwFAEOAIYiwAHAUAQ4ABiKAAcAQ41pRR4AGE5eXl7Mx5w7\nd24CKpmaCHAAcRspjHmUPjEYQgFguyvv7Eh2CVMCAQ7Adld370x2CVMCAQ4AhiLAAcBQBDgAGIoA\nBwBDEeAAbJe5YlWyS5gSCHAAtpv29OpklzAlRH2Qp7e3V1VVVbp27ZrC4bAWLVokn88nv9+vhoYG\nuVwuSdLKlStVWFg44QUDAL4QNcAdDodeeeUVZWZm6vr163r55Zc1d+5cWZal4uJiFRcXJ6JOAMCX\njGkIJTMzU5LU19envr4+WZYlSUrwgvYAgBuMaS6UcDisiooKff7551q2bJnuuOMOtbS06MCBAzpy\n5Ihmz56tZ555RllZWRNdLwDg/1n9MXSjL1++rC1btujZZ5+Vy+WKjH/v2rVLnZ2dKisri/ozzp8/\nH3+1iGCyIKSyjAO71bdsRbLLmDRyc3OH3R/TbITTp0+X1+vVxx9/rCeeeCKyv6ioSNXV1UM+HwgE\nFAgEIq99Pp+cTmcsp8QIHA4H3yVSVnD3Tnm4E8VWfr8/su31euX1eqMHeFdXl9LT05WVlaXe3l6d\nPHlSy5cvVzAYlMfjkSQdP35c+fn5Q44dOMmN6DXagx44Uh3t0z5Op1M+n2/I/qgBHgwGVVdXp3A4\nrHA4rMWLF2vevHl67bXX1NraKsuylJOTozVr1kxI4QCA4cU0Bm4HxsDtQQ8cqez6c08q/c29yS5j\n0hhpDJwnMQHAUAQ4ANsxF0piEOAAbMdcKIlBgAOAoQhwADAUAQ4AhiLAAcBQBDgA2115Z0eyS5gS\nCHAAtru6e2eyS5gSCHAAMBQBDgCGIsABwFAEOAAYigAHYDvmQkkMAhyA7ZgLJTEIcAAwFAEOAIYi\nwAHAUAQ4ABhq1EWNe3t7VVVVpWvXrikcDmvRokXy+Xzq7u5WbW2t2tvblZOTo/LycmVlZSWqZgAp\n7so7O6RlK5JdxqQXdVHjq1evKjMzU9evX9fLL7+s1atX6+9//7ucTqeWL1+u+vp6Xbp0SaWlpWM6\nIYsa24NFjZHKWNTYXnEvapyZmSlJ6uvrU19fnyzL0okTJ7RkyRJJ0tKlS9Xc3GxjqQCAsRh1CEWS\nwuGwKioq9Pnnn2vZsmW64447FAqF5PF4JElut1uhUGjCCwUADBY1wNPS0rRlyxZdvnxZW7Zs0b/+\n9a9B71uWNeKxgUBAgUAg8trn88npdI6jXAxwOBx8l0hZQYn2aTO/3x/Z9nq98nq90QN8wPTp0+X1\nevXJJ5/I7XYrGAzK4/Gos7NTbrd72GMGTnIjxm3twRg4Uh3t0z5Op1M+n2/I/lHHwLu6unTp0iVJ\nX9yRcvLkSeXl5Wn+/PlqbGyUJB0+fFgLFiywv2IAxmIulMQYtQceDAZVV1encDiscDisxYsXa968\nebrrrrtUW1urQ4cORW4jBIAB055eTQ88AaLeRmg3biO0B0MoSGW0T3vFfRshACA1EeAAYCgCHAAM\nRYADsN2Vd3Yku4QpgQAHYLuru3cmu4QpgQAHAEMR4ABgKAIcAAxFgAOAoQhwALZjLpTEIMAB2G7a\n06uTXcKUQIADgKEIcAAwFAEOAIYiwAHAUAQ4ANsxF0piEOAAbMdcKIlBgAOAoQhwADDUqIsaS1J7\ne7vq6uoUCoVkWZYeeeQRPfbYY/L7/WpoaJDL5ZIkrVy5UoWFhRNeMADgC1EDPCMjQ6tWrdKsWbPU\n09OjiooKFRQUyLIsFRcXq7i4OBF1AgC+JGqAezweeTweSdJNN92kvLw8dXR0SJISvKA9AENkrlil\nvmQXMQXENAbe1tam1tZW3XXXXZKkAwcOaN26ddq2bZsuXbo0IQUCMA9zoSSG1T/GbnRPT4+qqqr0\nne98RwsXLlQoFIqMf+/atUudnZ0qKysbdEwgEFAgEIi89vl8unjxoo3lT10Oh0O9vb3JLgMYFu3T\nXk6nU36/P/La6/XK6/WOLcD7+vpUXV2twsJCPf7440Peb2trU3V1tWpqaqIWcv78+RhLx3CcTif/\nGSJl0T7tlZubO+z+qEMo/f392r59u/Ly8gaFd2dnZ2T7+PHjys/Pt6FMAMBYRb2Iefr0ab3//vvK\nz8/X+vXrJUnf/e531dTUpNbWVlmWpZycHK1Zs2bCiwUA/NeYx8DtwhCKPfgTFaks48Bu9S1bkewy\nJo24h1AAIFbMhZIYBDgAGIoABwBDEeAAYCgCHAAMRYADsF3milXJLmFKIMAB2I65UBKDAAcAQxHg\nAGAoAhwADEWAA4ChCHAAtrvyzo5klzAlEOAAbMdcKIlBgAOAoQhwADAUAQ4AhiLAAcBQUZdUAzC1\nXV+7UrrcHftxzz059g9Pn6H0rf8b8zmmOgIcwOgudyv9zb0xHRLrkn8xhT0iogZ4e3u76urqFAqF\nZFmWHnnkET322GPq7u5WbW2t2tvblZOTo/LycmVlZSWiZgCAxhDgGRkZWrVqlWbNmqWenh5VVFSo\noKBAjY2NKigo0PLly1VfX6/6+nqVlpYmomYAgMZwEdPj8WjWrFmSpJtuukl5eXnq6OjQiRMntGTJ\nEknS0qVL1dzcPKGFAgAGi+kulLa2NrW2turOO+9UKBSSx+ORJLndboVCoQkpEAAwvDFfxOzp6VFN\nTY1Wr16tadOmDXrPsqxhjwkEAgoEApHXPp9PTqczzlJxI4fDwXeJhAhKMbe1WNtnPOeYavx+f2Tb\n6/XK6/WOLcD7+vpUU1Ojhx56SAsXLpT0Ra87GAzK4/Gos7NTbrd7yHEDJ7lRLFemMbJYr/ID4xFr\nW4unfdKeR+Z0OuXz+YbsjzqE0t/fr+3btysvL0+PP/54ZP/8+fPV2NgoSTp8+LAWLFhgX7UAgKii\n9sBPnz6t999/X/n5+Vq/fr0kaeXKlSopKVFtba0OHToUuY0QAJA4UQP87rvv1q5du4Z9b9OmTbYX\nBAAYG+ZCAQBDEeAAYCgCHAAMRYADgKEIcAAwFAEOAIYiwAHAUAQ4ABiKAAcAQxHgAGAoAhwADEWA\nA4ChCHAAMBQBDgCGIsABwFAEOAAYigAHAEMR4ABgKAIcAAwVdU3M119/XS0tLXK5XKqpqZEk+f1+\nNTQ0yOVySfpikePCwsKJrRQAMEjUAH/44Yf1rW99S6+99lpkn2VZKi4uVnFx8YQWBwAYWdQhlDlz\n5igrK2vI/v7+/gkpCAAwNlF74CM5cOCAjhw5otmzZ+uZZ54ZNuQBABMnrgD/5je/qaeeekqStGvX\nLr399tsqKysb8rlAIKBAIBB57fP55HQ64ywVN3I4HHyXSIigFHNbi7V9xnOOqcbv90e2vV6vvF5v\nfAHudrsj20VFRaqurh72cwMnudHFixfjOSW+xOl08l0iYWJta/G0T9rzyJxOp3w+35D9cd1G2NnZ\nGdk+fvy48vPz468MABCXqD3wV199VZ9++qm6urpUVlamp59+WqdOnVJra6ssy1JOTo7WrFmTiFoB\nADeIGuAvvPDCkH1FRUUTUgwAYOx4EhMADEWAA4ChCHAAMBQBDgCGivtJTCRGXl5eXMedO3fO5koA\npBoCPMWNFMTXn3tS6W/uTXA1AFIJQygAYCgCHAAMRYADgKEIcAAwFBcxDZW5YpX6kl0EpoT9j74t\n7QrGeFSMn3/0bT0R4xkgWf0JXlrn/PnziTzdpMV0skiUeO54irV9clfV6HJzc4fdzxAKABiKAAcA\nQxHgAGAoAhwADEWAG+rKOzuSXQKAJCPADXV1985klwAgyQhwADBU1Ad5Xn/9dbW0tMjlcqmmpkaS\n1N3drdraWrW3tysnJ0fl5eXKysqa8GIBAP8VtQf+8MMPa8OGDYP21dfXq6CgQFu3btU999yj+vr6\nCSsQADC8qAE+Z86cIb3rEydOaMmSJZKkpUuXqrm5eWKqAwCMKK4x8FAoJI/HI0lyu90KhUK2FoXo\nMlesSnYJAJJs3JNZWZY14nuBQECBQCDy2ufzyel0jveUkOQoXaPe3t5kl4EpICjF/O/W4XDEdEw8\n55hq/H5/ZNvr9crr9cYX4G63W8FgUB6PR52dnXK73cN+buAkN2ICJnswmRUSKda2Fk/7pD2PzOl0\nyufzDdkf1xDK/Pnz1djYKEk6fPiwFixYMK7iAACxi9oDf/XVV/Xpp5+qq6tLZWVl8vl8KikpUW1t\nrQ4dOhS5jRAAkFhRA/yFF14Ydv+mTZtsLwYAMHY8iWko5kIBQIAbirlQABDgAGAoAhwADEWAA4Ch\nCHAAMNS4H6WHPa6vXSld7o7tmOeejO0k02cofev/xnYMgJRFgKeKy91Kf3PvmD8ez6PKMQc+gJTG\nEAoAGIoABwBDMYQCIKpYh9+CsZ5g+oxYj4AIcABRxHJtZsD1556M6zjEhiEUADAUAQ4AhiLAAcBQ\nBDgAGIoAB2C7zBWrkl3ClECAA7DdtKdXJ7uEKYEABwBDjes+8Oeff17Tpk1TWlqa0tPT9Ytf/MKu\nugAAUYz7QZ6qqirNmMFTVACQaOMeQunv77ejDgBAjMYV4JZlafPmzaqsrNTBgwftqgmA4a68syPZ\nJUwJ4xpC2bx5s26++WZ1dXVp8+bNysvL05w5cyLvBwIBBQKByGufzyen0zmeU05aQSmm78bhcMT8\nXcZ6DiBewd075eFOFFv5/f7IttfrldfrHV+A33zzzZIkl8ulhQsX6syZM4MCfOAkN4p1EYKpJJbv\nJp4FHWI9BzAetDX7OJ1O+Xy+IfvjHkK5evWqrly5Iknq6enRP/7xD+Xn58dfIQAgJnH3wEOhkLZs\n2SJJCofDevDBB3XvvffaVhgAYHRxB/jMmTMjAQ4ASDyexARgO+ZCSQwCHIDtmAslMVhSLUXsf/Rt\naVcsKwnGvOqg9OjbeiL2owCkKAI8RTx28JmY1hCM5zbC6889Kf0P6xQCkwVDKABgKAIcAAxFgAOw\nHXOhJAYBDsB2V3fvTHYJUwIBDgCGIsABwFDcRgggbnl5eaO9Oezuc+fOTVA1Uw8BDiBuI4VxvNMd\nIzYMoQCAoQhwADAUAQ4AhmIMPIVcf+7JMX82jqmspOkz4jkKQIoiwFPESBNZjXqVfxRc6QcmPwI8\nxXGVH8BIxhXgH3/8sXbs2KFwOKyioiKVlJTYVRcAIIq4L2KGw2G99dZb2rBhg371q1+pqalJ//73\nv+2sDQAwirgD/MyZM7r11ls1c+ZMZWRk6IEHHtCJEyfsrA0AMIq4A7yjo0Nf+cpXIq+zs7PV0dFh\nS1EAgOi4DxwADBX3Rczs7GxduHAh8vrChQvKzs4e9JlAIKBAIBB57fP5lJubG+8p8SVOpzPZJQAj\non3ay+/3R7a9Xq+8Xm/8Af71r39dn332mdra2pSdna1jx45p7dq1gz4zcBLYz+/3y+fzJbsMYFi0\nT/sN933GHeDp6el69tln9fOf/zxyG+Htt98+rgIBAGM3rvvA586dq7lz59pVCwAgBlzENBRDU0hl\ntM/EsPr7+/uTXQQAIHb0wAHAUAQ4ABiK2QhTwPe//3394Q9/kCR99NFH2rlzpzZt2qSGhgY1NDTI\n5XKpr69PK1as0AMPPCBJqqur03333aejR4+qra1NPT096urq0syZMyVJP/jBD3T27Fnt27dPbW1t\neuuttzRjBvOBwz7Hjx9XTU2NamtrlZubq7a2NlVXV6umpkaSdPDgQR08eFCbNm3Snj179NFHHykj\nI0Nf/epX9aMf/UjTp09P8m9gPgI8BViWJUk6efKkduzYoY0bN+qWW26RZVkqLi5WcXGxPvvsM1VU\nVOj+++9XWlqaLMuSZVl66aWXJEmnTp3S3r17VVlZGfm5DodD9913n6qqqpLxa2GSa2pq0t13362j\nR48OuUf5yJEj+utf/6pXXnlFWVlZuvfee1VaWqq0tDT96U9/0rvvvqvS0tIkVT55MISSIk6dOqXf\n/OY3qqysjPSiJWngGvOtt96qzMxMdXd3D3nvy9sDZs2apZycnAmsGlNVT0+PTp8+rR/+8Ic6duzY\noPeOHTum9957Txs3boz81VdQUKC0tC/i5s477xz0FDfiR4CngGvXrumXv/yl1q1bN+JUA2fPntVt\nt90ml8uV4OqAoZqbm1VYWKjbbrtNTqdTZ8+elST95z//0e9//3tt3LhRbrd72GMbGho0b968RJY7\naRHgKSAjI0Pf+MY31NDQMGh/f3+/9u3bpxdffFE//elP9e1vfztJFQKDNTU1afHixZKk+++/X01N\nTbIsS263W7fccsuQXvmAPXv2KCMjQw8++GAiy520CPAUYFmWysvLdebMGb377ruD9hcXF6umpkYv\nvviitm/frr6+viRWCkjd3d0KBAJ644039Pzzz+svf/mLPvjgA/X398vhcOgnP/mJ/va3v+no0aOD\njmtsbFRLS4t+/OMfJ6nyyYcATxEOh0OVlZU6evTooJ74wNj2/PnzNXv2bDU2Nsb183leC3b54IMP\n9NBDD6murk51dXXatm2bcnJy1N7eLklyuVzasGGD/vznP+uTTz6R9MXyi3v37tW6devkcDiSWf6k\nQoCngIG7UGbMmKENGzZoz549kdWNBt6TpKeeekr79u2LhPGN7w3clXKj/fv3q6ysTB0dHXrppZf0\nxhtvTPSvgimgqalJCxcuHLRv0aJFeu+99yJtcObMmVq/fr22bdumM2fO6He/+516enr0s5/9TOvX\nr9dvf/vbZJQ+6fAoPQAYih44ABiKAAcAQxHgAGAoAhwADEWAA4ChCHAAMBQBDgCGIsABwFD/B8Y5\nMxXMPIFJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fbc9ec7e610>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "a = target_expr_meta.boxplot([\"KRIT1\", \"AK2\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用Plotly绘制交互图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"77e25d92-10f1-4f66-99d1-e2cb8400f276\" style=\"height: 500px; width: 600px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"77e25d92-10f1-4f66-99d1-e2cb8400f276\", [{\"name\": \"Legend name\", \"text\": [\"fibroblast of villous mesenchyme\", \"pulmonary artery endothelial cell\", \"mesangial cell\", \"pulmonary artery endothelial cell\"], \"y\": [2.31, 37.62, 8.95, 15.57], \"mode\": \"markers+texts\", \"marker\": {\"opacity\": \"0.9\", \"symbol\": \"circle\"}, \"x\": [1.15, 13.36, 1.37, 6.21]}, {\"showlegend\": false, \"name\": \"Diagonal line\", \"text\": \"Diagonal\", \"mode\": \"lines\", \"y\": [0, 40], \"x\": [0, 40], \"line\": {\"color\": \"rgb(192,192,192)\"}}], {\"showlegend\": true, \"yaxis\": {\"range\": [0, 40], \"title\": \"AK2 expression value\"}, \"height\": 500, \"width\": 600, \"xaxis\": {\"range\": [0, 40], \"title\": \"KRIT1 expression value\"}, \"legend\": {\"y\": 1, \"x\": 0.6}}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = {\n",
    "    'data': [\n",
    "        {\n",
    "            'x': target_expr_meta.KRIT1,\n",
    "            'y': target_expr_meta.AK2,\n",
    "            'text': target_expr_meta.Biosample_term_name,\n",
    "            'mode': 'markers+texts',\n",
    "            'name': 'Legend name',\n",
    "            'marker': {\n",
    "                'symbol':\"circle\",\n",
    "                \"opacity\": \"0.9\"\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            'x': [0,40],\n",
    "            'y': [0,40],\n",
    "            'text': \"Diagonal\",\n",
    "            'mode': 'lines',\n",
    "            'name': 'Diagonal line',\n",
    "            'showlegend': False,\n",
    "            'line': {\n",
    "                'color': ('rgb(192,192,192)')\n",
    "            }\n",
    "        }\n",
    "    ],\n",
    "    'layout': {\n",
    "        'xaxis': {'title':'KRIT1 expression value','range':[0,40]},\n",
    "        'yaxis': {'title':'AK2 expression value','range':[0,40]},\n",
    "        'height':500,\n",
    "        'width':600,\n",
    "        'showlegend': True,\n",
    "         \"legend\": {\n",
    "            \"x\": 0.6,\n",
    "            \"y\": 1\n",
    "        }\n",
    "    }\n",
    "}\n",
    "plotly.offline.iplot(fig)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"128b2b20-5d14-4df5-b156-a9b7ea2f5dea\" style=\"height: 500px; width: 600px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"128b2b20-5d14-4df5-b156-a9b7ea2f5dea\", [{\"name\": \"female\", \"text\": [\"fibroblast of villous mesenchyme\", \"mesangial cell\"], \"y\": [2.31, 8.95], \"mode\": \"markers+texts\", \"marker\": {\"opacity\": \"0.9\", \"symbol\": \"circle\"}, \"x\": [1.15, 1.37]}, {\"name\": \"female\", \"text\": [\"pulmonary artery endothelial cell\", \"pulmonary artery endothelial cell\"], \"y\": [37.62, 15.57], \"mode\": \"markers+texts\", \"marker\": {\"opacity\": \"0.9\", \"symbol\": \"circle\"}, \"x\": [13.36, 6.21]}], {\"showlegend\": true, \"yaxis\": {\"range\": [0, 40], \"title\": \"AK2 expression value\"}, \"height\": 500, \"width\": 600, \"xaxis\": {\"range\": [0, 40], \"title\": \"KRIT1 expression value\"}, \"legend\": {\"y\": 1, \"x\": 0.6}}, {\"linkText\": \"Export to plot.ly\", \"showLink\": true})});</script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = {\n",
    "    'data': [\n",
    "        {\n",
    "            'x': target_expr_meta[target_expr_meta['Biosample_sex']==atype].KRIT1,\n",
    "            'y': target_expr_meta[target_expr_meta['Biosample_sex']==atype].AK2,\n",
    "            'text': target_expr_meta[target_expr_meta['Biosample_sex']==atype].Biosample_term_name,\n",
    "            'mode': 'markers+texts',\n",
    "            'name': _type,\n",
    "            'marker': {\n",
    "                'symbol':\"circle\",\n",
    "                \"opacity\": \"0.9\"\n",
    "            }\n",
    "        } for atype in ['female','male']\n",
    "    ],\n",
    "    'layout': {\n",
    "        'xaxis': {'title':'KRIT1 expression value','range':[0,40]},\n",
    "        'yaxis': {'title':'AK2 expression value','range':[0,40]},\n",
    "        'height':500,\n",
    "        'width':600,\n",
    "        'showlegend': True,\n",
    "         \"legend\": {\n",
    "            \"x\": 0.6,\n",
    "            \"y\": 1\n",
    "        }\n",
    "    }\n",
    "}\n",
    "plotly.offline.iplot(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用R读取HDF5文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "##R code for reading hdf5\n",
    "\n",
    "> h5ls('test.hdf5')\n",
    "      group          name       otype  dclass      dim\n",
    "0         /          FPKM   H5I_GROUP                 \n",
    "1     /FPKM         axis0 H5I_DATASET  STRING        3\n",
    "2     /FPKM         axis1 H5I_DATASET  STRING    25135\n",
    "3     /FPKM  block0_items H5I_DATASET  STRING        3\n",
    "4     /FPKM block0_values H5I_DATASET   FLOAT  x 25135\n",
    "5         /           TPM   H5I_GROUP                 \n",
    "6      /TPM         axis0 H5I_DATASET  STRING        3\n",
    "7      /TPM         axis1 H5I_DATASET  STRING    24025\n",
    "8      /TPM  block0_items H5I_DATASET  STRING        3\n",
    "9      /TPM block0_values H5I_DATASET   FLOAT  x 24025\n",
    "10        /       ens2syn   H5I_GROUP                 \n",
    "11 /ens2syn         axis0 H5I_DATASET  STRING        1\n",
    "12 /ens2syn         axis1 H5I_DATASET  STRING    60725\n",
    "13 /ens2syn  block0_items H5I_DATASET  STRING        1\n",
    "14 /ens2syn block0_values H5I_DATASET    VLEN        1\n",
    "15        /          meta   H5I_GROUP                 \n",
    "16    /meta         axis0 H5I_DATASET  STRING       47\n",
    "17    /meta         axis1 H5I_DATASET  STRING        3\n",
    "18    /meta  block0_items H5I_DATASET  STRING       19\n",
    "19    /meta block0_values H5I_DATASET   FLOAT      x 3\n",
    "20    /meta  block1_items H5I_DATASET  STRING        2\n",
    "21    /meta block1_values H5I_DATASET INTEGER      x 3\n",
    "22    /meta  block2_items H5I_DATASET  STRING       26\n",
    "23    /meta block2_values H5I_DATASET    VLEN        1\n",
    "> TPM = h5read(\"test.hdf5\", \"/TPM\")\n",
    "> str(TPM)\n",
    "List of 4\n",
    " $ axis0        : chr [1:3(1d)] \"ENCFF673KYR\" \"ENCFF805ZGF\" \"ENCFF581ZEU\"\n",
    " $ axis1        : chr [1:24025(1d)] \"ENSG00000000003.14\" \"ENSG00000000005.5\" \"ENSG00000000419.12\" \"ENSG00000000457.13\" ...\n",
    " $ block0_items : chr [1:3(1d)] \"ENCFF673KYR\" \"ENCFF805ZGF\" \"ENCFF581ZEU\"\n",
    " $ block0_values: num [1:3, 1:24025] 2.42 1.64 5.69 0 0 0.11 1.8 3.82 6.38 0.38 ...\n",
    "> d <- TPM$block0_values\n",
    "> rownames(d) <- TPM$axis1\n",
    "Error in `rownames<-`(`*tmp*`, value = c(\"ENSG00000000003.14\", \"ENSG00000000005.5\",  : \n",
    "  length of 'dimnames' [1] not equal to array extent\n",
    "> d <- as.data.frame(TPM$block0_values)\n",
    "> rownames(d) <- TPM$axis1\n",
    "Error in `row.names<-.data.frame`(`*tmp*`, value = value) : \n",
    "  invalid 'row.names' length\n",
    "> dims(d)\n",
    "Error: could not find function \"dims\"\n",
    "> dim(d)\n",
    "[1]     3 24025\n",
    "> d <- t(as.data.frame(TPM$block0_values))\n",
    "> dim(d)\n",
    "[1] 24025     3\n",
    "> rownames(d) <- TPM$axis1\n",
    "> colnames(d) <- TPM$axis0\n",
    "> hed(d)\n",
    "Error: could not find function \"hed\"\n",
    "> head(d)\n",
    "                   ENCFF673KYR ENCFF805ZGF ENCFF581ZEU\n",
    "ENSG00000000003.14        2.42        1.64        5.69\n",
    "ENSG00000000005.5         0.00        0.00        0.11\n",
    "ENSG00000000419.12        1.80        3.82        6.38\n",
    "ENSG00000000457.13        0.38        0.57        1.17\n",
    "ENSG00000000460.16        0.16        0.31        0.14\n",
    "ENSG00000000938.12        0.00        0.03        0.00\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandas矩阵生成"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>first</th>\n",
       "      <th>second</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.417022</td>\n",
       "      <td>0.326645</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0.720324</td>\n",
       "      <td>0.527058</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0.000114</td>\n",
       "      <td>0.885942</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0.302333</td>\n",
       "      <td>0.357270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0.146756</td>\n",
       "      <td>0.908535</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   class     first    second\n",
       "0      0  0.417022  0.326645\n",
       "1      0  0.720324  0.527058\n",
       "2      1  0.000114  0.885942\n",
       "3      1  0.302333  0.357270\n",
       "4      1  0.146756  0.908535"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "df = pd.DataFrame({\"first\": np.random.rand(100),\n",
    "                   \"second\": np.random.rand(100),\n",
    "                   \"class\": np.random.randint(0, 2, (100,))},\n",
    "                   index=range(100))\n",
    "df.head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}