Kubernetes-Installation-Report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 写在前面\n",
    "\n",
    "最近一周在清华装Kubernetes，真的是一把辛酸泪，前几天都快绝望了，还好在csdn上找到一篇博客详细地介绍了安装Kubernetes的相关事宜，我在这篇文章的指导下才渐渐将一个双节点的Kubernets集群在阿里云上搭建起来，虽然教程里面也有说的不清楚的地方，但是还是给了我从无到有的启发，现在就把安装的细节写下来，以供参考。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 安装环境\n",
    "\n",
    "清华提供了一个`6`节点的集群，计划搭建一个6节点的Kubernetes集群，各个节点的配置如下所示。\n",
    "\n",
    "\n",
    "|IP地址|操作系统|角色|\n",
    "|-------|----------|--------------|\n",
    "|192.168.10.10|Ubuntu 16.04.3 LTS|Master|\n",
    "|192.168.10.11|Ubuntu 16.04.3 LTS|Slave1|\n",
    "|192.168.10.12|Ubuntu 16.04.3 LTS|Slave2|\n",
    "|192.168.10.13|Ubuntu 16.04.3 LTS|Slave3|\n",
    "|192.168.10.14|Ubuntu 16.04.3 LTS|Slave4|\n",
    "|192.168.10.15|Ubuntu 16.04.3 LTS|Slave5|\n",
    "\n",
    "把主节点的`/etc/hosts`文件展示如下：\n",
    "\n",
    "```shell\n",
    "zdyfjh@Master:~$ cat /etc/hosts\n",
    "127.0.0.1 localhost\n",
    "\n",
    "# The following lines are desirable for IPv6 capable hosts\n",
    "::1     ip6-localhost ip6-loopback\n",
    "fe00::0 ip6-localnet\n",
    "ff00::0 ip6-mcastprefix\n",
    "ff02::1 ip6-allnodes\n",
    "ff02::2 ip6-allrouters\n",
    "101.6.6.177 mirrors.tuna.tsinghua.edu.cn\n",
    "192.168.10.10 Master\n",
    "192.168.10.11 Slave1\n",
    "192.168.10.12 Slave2\n",
    "192.168.10.13 Slave3\n",
    "192.168.10.14 Slave4\n",
    "192.168.10.15 Slave5\n",
    "151.101.24.175 bootstrap.pypa.io\n",
    "151.101.26.49 curl.haxx.se\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 组件下载\n",
    "\n",
    "由于装系统这种事，有一定的玄学成分，常常是同一个东西A版本就很正常，B版本就呵呵了，因此为了保证安装的正确性，将用到的组件下载地址在这里给出，推荐下载这里给出的组件。\n",
    "\n",
    "1. Etcd: [https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz](https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz)\n",
    "\n",
    "2. Kubernetes: [https://dl.k8s.io/v1.9.6/kubernetes-server-linux-amd64.tar.gz](https://dl.k8s.io/v1.9.6/kubernetes-server-linux-amd64.tar.gz)\n",
    "\n",
    "3. Flannel: [https://github.com/coreos/flannel/releases/download/v0.9.0/flannel-v0.9.0-linux-amd64.tar.gz](https://github.com/coreos/flannel/releases/download/v0.9.0/flannel-v0.9.0-linux-amd64.tar.gz)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 配置CA证书\n",
    "\n",
    "该步骤主要是为了使集群上能够使用`https`协议，这里使用 CloudFlare 的 PKI 工具集 cfssl 来生成 Certificate Authority (CA) 证书和秘钥文件。\n",
    "\n",
    "## Master上操作\n",
    "\n",
    "* ### 安装`cfssl`\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh# mkdir -p /opt/local/cfssl\n",
    "root@Master:/home/zdyfjh# cd /opt/local/cfssl\n",
    "root@Master:/opt/local/cfssl# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64\n",
    "root@Master:/opt/local/cfssl# mv cfssl_linux-amd64 cfssl\n",
    "root@Master:/opt/local/cfssl# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64\n",
    "root@Master:/opt/local/cfssl# mv cfssljson_linux-amd64 cfssljson\n",
    "root@Master:/opt/local/cfssl# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64\n",
    "root@Master:/opt/local/cfssl# mv cfssl-certinfo_linux-amd64 cfssl-certinfo\n",
    "root@Master:/opt/local/cfssl# chmod +x *\n",
    "root@Master:/opt/local/cfssl# ll\n",
    "total 18816\n",
    "drwxr-xr-x 2 root root     4096 Oct 22 21:34 ./\n",
    "drwxr-xr-x 3 root root     4096 Oct 22 21:30 ../\n",
    "-rwxr-xr-x 1 root root 10376657 Mar 30  2016 cfssl*\n",
    "-rwxr-xr-x 1 root root  6595195 Mar 30  2016 cfssl-certinfo*\n",
    "-rwxr-xr-x 1 root root  2277873 Mar 30  2016 cfssljson*\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 创建CA证书配置文件\n",
    "\n",
    "```shell\n",
    "root@Master:/opt/local/cfssl# mkdir /opt/ssl\n",
    "root@Master:/opt/local/cfssl# cd /opt/ssl\n",
    "root@Master:/opt/ssl# vim config.json\n",
    "root@Master:/opt/ssl# cat config.json \n",
    "{\n",
    "  \"signing\": {\n",
    "    \"default\": {\n",
    "      \"expiry\": \"87600h\"\n",
    "    },\n",
    "    \"profiles\": {\n",
    "      \"kubernetes\": {\n",
    "        \"usages\": [\n",
    "            \"signing\",\n",
    "            \"key encipherment\",\n",
    "            \"server auth\",\n",
    "            \"client auth\"\n",
    "        ],\n",
    "        \"expiry\": \"87600h\"\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "root@Master:/opt/ssl# vim csr.json\n",
    "root@Master:/opt/ssl# cat csr.json \n",
    "{\n",
    "  \"CN\": \"kubernetes\",\n",
    "  \"key\": {\n",
    "    \"algo\": \"rsa\",\n",
    "    \"size\": 2048\n",
    "  },\n",
    "  \"names\": [\n",
    "    {\n",
    "      \"C\": \"CN\",\n",
    "      \"ST\": \"Beijing\",\n",
    "      \"L\": \"Beijing\",\n",
    "      \"O\": \"k8s\",\n",
    "      \"OU\": \"System\"\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "root@Master:/opt/ssl# ls\n",
    "config.json  csr.json\n",
    "root@Master:/opt/ssl# /opt/local/cfssl/cfssl gencert -initca csr.json | /opt/local/cfssl/cfssljson -bare ca\n",
    "2018/10/22 21:43:35 [INFO] generating a new CA key and certificate from CSR\n",
    "2018/10/22 21:43:35 [INFO] generate received request\n",
    "2018/10/22 21:43:35 [INFO] received CSR\n",
    "2018/10/22 21:43:35 [INFO] generating key: rsa-2048\n",
    "2018/10/22 21:43:36 [INFO] encoded CSR\n",
    "2018/10/22 21:43:36 [INFO] signed certificate with serial number 52404311838458003012572611311024350631102884515\n",
    "root@Master:/opt/ssl# ls -l\n",
    "total 20\n",
    "-rw-r--r-- 1 root root 1001 Oct 22 21:43 ca.csr\n",
    "-rw------- 1 root root 1675 Oct 22 21:43 ca-key.pem\n",
    "-rw-r--r-- 1 root root 1359 Oct 22 21:43 ca.pem\n",
    "-rw-r--r-- 1 root root  292 Oct 22 21:37 config.json\n",
    "-rw-r--r-- 1 root root  208 Oct 22 21:39 csr.json\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 向主节点及从节点分发证书（依然只在主节点上操作）\n",
    "\n",
    "```shell\n",
    "root@Master:/opt/ssl# mkdir -p /etc/kubernetes/ssl\n",
    "root@Master:/opt/ssl# cp *.pem /etc/kubernetes/ssl\n",
    "root@Master:/opt/ssl# cp ca.csr /etc/kubernetes/ssl\n",
    "```\n",
    "**注意：一下操作要通过`scp`命令向各个从节点发送证书，证书存放的路径都为`/etc/kubernetes/ss/`，因此要确保各个从节点上均存在这个目录**\n",
    "\n",
    "```shell\n",
    "root@Master:/opt/ssl# scp * zdyfjh@192.168.10.11:/etc/kubernetes/ssl\n",
    "root@Master:/opt/ssl# scp * zdyfjh@192.168.10.12:/etc/kubernetes/ssl\n",
    "root@Master:/opt/ssl# scp * zdyfjh@192.168.10.13:/etc/kubernetes/ssl\n",
    "root@Master:/opt/ssl# scp * zdyfjh@192.168.10.14:/etc/kubernetes/ssl\n",
    "root@Master:/opt/ssl# scp * zdyfjh@192.168.10.15:/etc/kubernetes/ssl\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 安装Docker（该步骤各个从节点也要进行）\n",
    "\n",
    "## Master节点上操作\n",
    "\n",
    "```shell\n",
    "root@Master:/opt/ssl# apt-get update\n",
    "root@Master:/opt/ssl# apt-get remove docker docker-engine docker.io\n",
    "root@Master:/opt/ssl# apt-get install docker.io\n",
    "root@Master:/opt/ssl# docker -v\n",
    "Docker version 17.03.2-ce, build f5ec1e2\n",
    "```\n",
    "\n",
    "## Slave节点上操作\n",
    "\n",
    "```shell\n",
    "root@Slave:/opt/ssl# apt-get update\n",
    "root@Slave:/opt/ssl# apt-get remove docker docker-engine docker.io\n",
    "root@Slave:/opt/ssl# apt-get install docker.io\n",
    "root@Slave:/opt/ssl# docker -v\n",
    "Docker version 17.03.2-ce, build f5ec1e2\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 安装Etcd（该步骤需要从节点操作）\n",
    "\n",
    "## Master节点上操作\n",
    "\n",
    "* ### 下载Etcd源代码\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh/download# wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz\n",
    "root@Master:/home/zdyfjh/download# ls\n",
    "etcd-v3.3.10-linux-amd64.tar.gz\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 向各个从节点分发Etcd源代码（**注意：各个从节点上要建立好相应的目录**）\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh/download# scp etcd-v3.3.10-linux-amd64.tar.gz zdyfjh@192.168.10.11:/home/zdyfjh/download   \n",
    "root@Master:/home/zdyfjh/download# scp etcd-v3.3.10-linux-amd64.tar.gz zdyfjh@192.168.10.12:/home/zdyfjh/download   \n",
    "root@Master:/home/zdyfjh/download# scp etcd-v3.3.10-linux-amd64.tar.gz zdyfjh@192.168.10.13:/home/zdyfjh/download \n",
    "root@Master:/home/zdyfjh/download# scp etcd-v3.3.10-linux-amd64.tar.gz zdyfjh@192.168.10.14:/home/zdyfjh/download    \n",
    "root@Master:/home/zdyfjh/download# scp etcd-v3.3.10-linux-amd64.tar.gz zdyfjh@192.168.10.15:/home/zdyfjh/download\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 安装Etcd\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh/download# tar -zxvf etcd-v3.3.10-linux-amd64.tar.gz\n",
    "root@Master:/home/zdyfjh/download# cd ./etcd-v3.3.10-linux-amd64\n",
    "root@Master:/home/zdyfjh/download/etcd-v3.3.10-linux-amd64# ls\n",
    "Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md\n",
    "root@Master:/home/zdyfjh/download/etcd-v3.3.10-linux-amd64# mv etcd* /usr/bin/\n",
    "root@Master:/home/zdyfjh/download/etcd-v3.3.10-linux-amd64# ll /usr/bin/ | grep etcd\n",
    "-rwxr-xr-x  1 6810230 users   19237536 Oct 11 01:32 etcd*\n",
    "-rwxr-xr-x  1 6810230 users   15817472 Oct 11 01:32 etcdctl*\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 为Etcd配置秘钥（证书）并创建数据目录\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh/download/etcd-v3.3.10-linux-amd64# cd /opt/ssl/\n",
    "root@Master:/opt/ssl# vim etcd-csr.json\n",
    "root@Master:/opt/ssl# cat etcd-csr.json \n",
    "{\n",
    "  \"CN\": \"etcd\",\n",
    "  \"hosts\": [\n",
    "    \"192.168.10.10\",\n",
    "    \"192.168.10.11\",\n",
    "    \"192.168.10.12\",\n",
    "    \"192.168.10.13\",\n",
    "    \"192.168.10.14\",\n",
    "    \"192.168.10.15\"\n",
    "  ],\n",
    "  \"key\": {\n",
    "    \"algo\": \"rsa\",\n",
    "    \"size\": 2048\n",
    "  },\n",
    "  \"names\": [\n",
    "    {\n",
    "      \"C\": \"CN\",\n",
    "      \"ST\": \"Beijing\",\n",
    "      \"L\": \"Beijing\",\n",
    "      \"O\": \"k8s\",\n",
    "      \"OU\": \"System\"\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "root@Master:/opt/ssl# /opt/local/cfssl/cfssl gencert -ca=/opt/ssl/ca.pem \\\n",
    ">   -ca-key=/opt/ssl/ca-key.pem \\\n",
    ">   -config=/opt/ssl/config.json \\\n",
    ">   -profile=kubernetes etcd-csr.json | /opt/local/cfssl/cfssljson -bare etcd\n",
    "2018/10/23 10:15:19 [INFO] generate received request\n",
    "2018/10/23 10:15:19 [INFO] received CSR\n",
    "2018/10/23 10:15:19 [INFO] generating key: rsa-2048\n",
    "2018/10/23 10:15:20 [INFO] encoded CSR\n",
    "2018/10/23 10:15:20 [INFO] signed certificate with serial number 160692721133621926504549778955227969855580110303\n",
    "2018/10/23 10:15:20 [WARNING] This certificate lacks a \"hosts\" field. This makes it unsuitable for\n",
    "websites. For more information see the Baseline Requirements for the Issuance and Management\n",
    "of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);\n",
    "specifically, section 10.2.3 (\"Information Requirements\").\n",
    "root@Master:/opt/ssl# ls etcd*\n",
    "etcd.csr  etcd-csr.json  etcd-key.pem  etcd.pem\n",
    "root@Master:/opt/ssl# cp etcd*.pem /etc/kubernetes/ssl/\n",
    "root@Master:/opt/ssl# scp etcd*.pem zdyfjh@192.168.10.11:/etc/kubernetes/ssl/ \n",
    "root@Master:/opt/ssl# scp etcd*.pem zdyfjh@192.168.10.12:/etc/kubernetes/ssl/   \n",
    "root@Master:/opt/ssl# scp etcd*.pem zdyfjh@192.168.10.13:/etc/kubernetes/ssl/    \n",
    "root@Master:/opt/ssl# scp etcd*.pem zdyfjh@192.168.10.14:/etc/kubernetes/ssl/   \n",
    "root@Master:/opt/ssl# scp etcd*.pem zdyfjh@192.168.10.15:/etc/kubernetes/ssl/\n",
    "root@Master:/opt/ssl# chmod 644 /etc/kubernetes/ssl/etcd-key.pem\n",
    "root@Master:/home/zdyfjh# useradd etcd\n",
    "root@Master:/home/zdyfjh# mkdir -p /opt/etcd\n",
    "root@Master:/home/zdyfjh# chown -R etcd:etcd /opt/etcd             \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 创建service文件\n",
    "\n",
    "```shell\n",
    "root@Master:/opt/ssl# vim /etc/systemd/system/etcd.service\n",
    "root@Master:/opt/ssl# cat /etc/systemd/system/etcd.service \n",
    "[Unit]\n",
    "Description=Etcd Server\n",
    "After=network.target\n",
    "After=network-online.target\n",
    "Wants=network-online.target\n",
    "\n",
    "[Service]\n",
    "Type=notify\n",
    "WorkingDirectory=/opt/etcd/\n",
    "User=etcd\n",
    "# set GOMAXPROCS to number of processors\n",
    "ExecStart=/usr/bin/etcd \\\n",
    "  --name=etcd1 \\ # 注意：这里要根据实际填入etcd1，etcd2等名称，具体的填入内容参照--initial-cluster中的内容名\n",
    "  --cert-file=/etc/kubernetes/ssl/etcd.pem \\\n",
    "  --key-file=/etc/kubernetes/ssl/etcd-key.pem \\\n",
    "  --peer-cert-file=/etc/kubernetes/ssl/etcd.pem \\\n",
    "  --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \\\n",
    "  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\\n",
    "  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\\n",
    "  --initial-advertise-peer-urls=https://192.168.10.10:2380 \\ #这里的ip要改成实际机器的IP\n",
    "  --listen-peer-urls=https://192.168.10.10:2380 \\ #这里的ip要改成实际机器的IP\n",
    "  --listen-client-urls=https://192.168.10.10:2379,http://127.0.0.1:2379 \\ #这里的ip要改成实际机器的IP\n",
    "  --advertise-client-urls=https://192.168.10.10:2379 \\ #这里的ip要改成实际机器的IP\n",
    "  --initial-cluster-token=k8s-etcd-cluster \\\n",
    "  --initial-cluster=etcd1=https://192.168.10.10:2380,etcd2=https://192.168.10.11:2380,etcd3=https://192.168.10.12:2380,etcd4=https://192.168.10.13:2380,etcd5=https://192.168.10.14:2380,etcd6=https://192.168.10.15:2380 \\\n",
    "  --initial-cluster-state=new \\\n",
    "  --data-dir=/opt/etcd/\n",
    "Restart=on-failure\n",
    "RestartSec=5\n",
    "LimitNOFILE=65536\n",
    "\n",
    "[Install]\n",
    "WantedBy=multi-user.target\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slave节点的操作\n",
    "\n",
    "* ### 安装Etcd\n",
    "\n",
    "```shell\n",
    "root@Slave:/home/zdyfjh/download# tar -zxvf etcd-v3.3.10-linux-amd64.tar.gz\n",
    "root@Slave:/home/zdyfjh/download# cd ./etcd-v3.3.10-linux-amd64\n",
    "root@Slave:/home/zdyfjh/download/etcd-v3.3.10-linux-amd64# ls\n",
    "Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md\n",
    "root@Slave:/home/zdyfjh/download/etcd-v3.3.10-linux-amd64# mv etcd* /usr/bin/\n",
    "root@Slave:/home/zdyfjh/download/etcd-v3.3.10-linux-amd64# ll /usr/bin/ | grep etcd\n",
    "-rwxr-xr-x  1 6810230 users   19237536 Oct 11 01:32 etcd*\n",
    "-rwxr-xr-x  1 6810230 users   15817472 Oct 11 01:32 etcdctl*\n",
    "```\n",
    "\n",
    "* ### 更改权限并创建数据目录\n",
    "\n",
    "```shell\n",
    "root@Slave:/home/zdyfjh# chmod 644 /etc/kubernetes/ssl/etcd-key.pem\n",
    "root@Slave:/home/zdyfjh# useradd etcd\n",
    "root@Slave:/home/zdyfjh# mkdir -p /opt/etcd\n",
    "root@Slave:/home/zdyfjh# chown -R etcd:etcd /opt/etcd\n",
    "```\n",
    "\n",
    "* ### 创建service文件\n",
    "\n",
    "```shell\n",
    "root@Slave:/opt/ssl# vim /etc/systemd/system/etcd.service\n",
    "root@Slave:/opt/ssl# cat /etc/systemd/system/etcd.service \n",
    "[Unit]\n",
    "Description=Etcd Server\n",
    "After=network.target\n",
    "After=network-online.target\n",
    "Wants=network-online.target\n",
    "\n",
    "[Service]\n",
    "Type=notify\n",
    "WorkingDirectory=/opt/etcd/\n",
    "User=etcd\n",
    "# set GOMAXPROCS to number of processors\n",
    "ExecStart=/usr/bin/etcd \\\n",
    "  --name=etcd1 \\ # 注意：这里要根据实际填入etcd1，etcd2等名称，具体的填入内容参照--initial-cluster中的内容名\n",
    "  --cert-file=/etc/kubernetes/ssl/etcd.pem \\\n",
    "  --key-file=/etc/kubernetes/ssl/etcd-key.pem \\\n",
    "  --peer-cert-file=/etc/kubernetes/ssl/etcd.pem \\\n",
    "  --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \\\n",
    "  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\\n",
    "  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\\n",
    "  --initial-advertise-peer-urls=https://192.168.10.10:2380 \\ #这里的ip要改成实际机器的IP\n",
    "  --listen-peer-urls=https://192.168.10.10:2380 \\ #这里的ip要改成实际机器的IP\n",
    "  --listen-client-urls=https://192.168.10.10:2379,http://127.0.0.1:2379 \\ #这里的ip要改成实际机器的IP\n",
    "  --advertise-client-urls=https://192.168.10.10:2379 \\ #这里的ip要改成实际机器的IP\n",
    "  --initial-cluster-token=k8s-etcd-cluster \\\n",
    "  --initial-cluster=etcd1=https://192.168.10.10:2380,etcd2=https://192.168.10.11:2380,etcd3=https://192.168.10.12:2380,etcd4=https://192.168.10.13:2380,etcd5=https://192.168.10.14:2380,etcd6=https://192.168.10.15:2380 \\\n",
    "  --initial-cluster-state=new \\\n",
    "  --data-dir=/opt/etcd/\n",
    "Restart=on-failure\n",
    "RestartSec=5\n",
    "LimitNOFILE=65536\n",
    "\n",
    "[Install]\n",
    "WantedBy=multi-user.target\n",
    "```\n",
    "\n",
    "* ### 启动Etcd（所有节点上都要执行）\n",
    "\n",
    "**注意：Etcd是一个分布式数据，因此一定要多个节点同时启动，如果间隔时间过长可能会出现错误**\n",
    "\n",
    "```shell\n",
    "root@Slave:/home/zdyfjh# systemctl daemon-reload\n",
    "root@Slave:/home/zdyfjh# systemctl enable etcd\n",
    "root@Slave:/home/zdyfjh# systemctl start etcd\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Master节点上的操作\n",
    "\n",
    "* ### 启动Etcd（所有节点都要执行）\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh# systemctl daemon-reload\n",
    "root@Master:/home/zdyfjh# systemctl enable etcd\n",
    "root@Master:/home/zdyfjh# systemctl start etcd\n",
    "root@Master:/home/zdyfjh# systemctl status etcd\n",
    "● etcd.service - Etcd Server\n",
    "   Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)\n",
    "   Active: active (running) since Tue 2018-10-23 11:12:02 CST; 13s ago\n",
    " Main PID: 27682 (etcd)\n",
    "    Tasks: 48\n",
    "   Memory: 31.2M\n",
    "      CPU: 1.307s\n",
    "   CGroup: /system.slice/etcd.service\n",
    "           └─27682 /usr/bin/etcd --name=etcd1 --cert-file=/etc/kubernetes/ssl/etcd.pem --key-file=/etc/kubernetes/ssl/etcd-key.pem --peer-cer\n",
    "\n",
    "Oct 23 11:12:06 Master etcd[27682]: health check for peer 5f9042fd5c3dd434 could not connect: dial tcp 192.168.10.11:2380: connect: connectio\n",
    "Oct 23 11:12:07 Master etcd[27682]: peer 5f9042fd5c3dd434 became active\n",
    "Oct 23 11:12:07 Master etcd[27682]: established a TCP streaming connection with peer 5f9042fd5c3dd434 (stream MsgApp v2 reader)\n",
    "Oct 23 11:12:07 Master etcd[27682]: established a TCP streaming connection with peer 5f9042fd5c3dd434 (stream Message reader)\n",
    "Oct 23 11:12:07 Master etcd[27682]: established a TCP streaming connection with peer 5f9042fd5c3dd434 (stream MsgApp v2 writer)\n",
    "Oct 23 11:12:07 Master etcd[27682]: established a TCP streaming connection with peer 5f9042fd5c3dd434 (stream Message writer)\n",
    "Oct 23 11:12:10 Master etcd[27682]: updated the cluster version from 3.0 to 3.3\n",
    "Oct 23 11:12:10 Master etcd[27682]: enabled capabilities for version 3.3\n",
    "Oct 23 11:12:11 Master etcd[27682]: health check for peer 5f9042fd5c3dd434 could not connect: dial tcp 192.168.10.11:2380: connect: connectio\n",
    "Oct 23 11:12:11 Master etcd[27682]: health check for peer 5f9042fd5c3dd434 could not connect: dial tcp 192.168.10.11:2380: connect: connectio\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 查看Etcd的运行状态，同时鉴定证书可读性\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh# etcdctl --endpoints=https://192.168.10.10:2379,https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379,https://192.168.10.15:2379\\\n",
    ">         --cert-file=/etc/kubernetes/ssl/etcd.pem \\\n",
    ">         --ca-file=/etc/kubernetes/ssl/ca.pem \\\n",
    ">         --key-file=/etc/kubernetes/ssl/etcd-key.pem \\\n",
    ">         cluster-health\n",
    "member ebec49f7cca334c is healthy: got healthy result from https://192.168.10.13:2379\n",
    "member fe8d75804df3c84 is healthy: got healthy result from https://192.168.10.14:2379\n",
    "member 2da67ab84f1e0230 is healthy: got healthy result from https://192.168.10.12:2379\n",
    "member 4ae14a68da42d271 is healthy: got healthy result from https://192.168.10.10:2379\n",
    "member 5f9042fd5c3dd434 is healthy: got healthy result from https://192.168.10.11:2379\n",
    "member b81b08d39ad7e16f is healthy: got healthy result from https://192.168.10.15:2379\n",
    "cluster is healthy\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 查看Etcd成员\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh# etcdctl --endpoints=https://192.168.10.10:2379,https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379,https://192.168.10.15:2379\\\n",
    ">         --cert-file=/etc/kubernetes/ssl/etcd.pem \\\n",
    ">         --ca-file=/etc/kubernetes/ssl/ca.pem \\\n",
    ">         --key-file=/etc/kubernetes/ssl/etcd-key.pem \\\n",
    ">         member list\n",
    "ebec49f7cca334c: name=etcd4 peerURLs=https://192.168.10.13:2380 clientURLs=https://192.168.10.13:2379 isLeader=true\n",
    "fe8d75804df3c84: name=etcd5 peerURLs=https://192.168.10.14:2380 clientURLs=https://192.168.10.14:2379 isLeader=false\n",
    "2da67ab84f1e0230: name=etcd3 peerURLs=https://192.168.10.12:2380 clientURLs=https://192.168.10.12:2379 isLeader=false\n",
    "4ae14a68da42d271: name=etcd1 peerURLs=https://192.168.10.10:2380 clientURLs=https://192.168.10.10:2379 isLeader=false\n",
    "5f9042fd5c3dd434: name=etcd2 peerURLs=https://192.168.10.11:2380 clientURLs=https://192.168.10.11:2379 isLeader=false\n",
    "b81b08d39ad7e16f: name=etcd6 peerURLs=https://192.168.10.15:2380 clientURLs=https://192.168.10.15:2379 isLeader=false\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 安装Kubernetes1.9.6\n",
    "\n",
    "## Master上操作\n",
    "\n",
    "* ### 下载源代码\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh/download# wget https://dl.k8s.io/v1.9.6/kubernetes-server-linux-amd64.tar.gz\n",
    "root@Master:/home/zdyfjh/download# ls -a | grep kube\n",
    "kubernetes-server-linux-amd64.tar.gz\n",
    "root@Master:/home/zdyfjh/download# scp kubernetes-server-linux-amd64.tar.gz zdyfjh@192.168.10.11:/home/zdyfjh/download \n",
    "root@Master:/home/zdyfjh/download# scp kubernetes-server-linux-amd64.tar.gz zdyfjh@192.168.10.12:/home/zdyfjh/download\n",
    "root@Master:/home/zdyfjh/download# scp kubernetes-server-linux-amd64.tar.gz zdyfjh@192.168.10.13:/home/zdyfjh/download\n",
    "root@Master:/home/zdyfjh/download# scp kubernetes-server-linux-amd64.tar.gz zdyfjh@192.168.10.14:/home/zdyfjh/download\n",
    "root@Master:/home/zdyfjh/download# scp kubernetes-server-linux-amd64.tar.gz zdyfjh@192.168.10.15:/home/zdyfjh/download\n",
    "root@Master:/home/zdyfjh/download# tar -zxf kubernetes-server-linux-amd64.tar.gz ^C\n",
    "root@Master:/home/zdyfjh/download# cd ./kubernetes\n",
    "root@Master:/home/zdyfjh/download/kubernetes# cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kubelet,kube-proxy} /usr/local/bin/\n",
    "root@Master:/home/zdyfjh/download/kubernetes# ls /usr/local/bin/ | grep kube\n",
    "kube-apiserver\n",
    "kube-controller-manager\n",
    "kubectl\n",
    "kubelet\n",
    "kube-proxy\n",
    "kube-scheduler\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 配置证书\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh/download/kubernetes# cd /opt/ssl/\n",
    "root@Master:/opt/ssl# vim admin-csr.json\n",
    "root@Master:/opt/ssl# cat admin-csr.json \n",
    "{\n",
    "  \"CN\": \"admin\",\n",
    "  \"hosts\": [],\n",
    "  \"key\": {\n",
    "    \"algo\": \"rsa\",\n",
    "    \"size\": 2048\n",
    "  },\n",
    "  \"names\": [\n",
    "    {\n",
    "      \"C\": \"CN\",\n",
    "      \"ST\": \"Beijing\",\n",
    "      \"L\": \"Beijing\",\n",
    "      \"O\": \"system:masters\",\n",
    "      \"OU\": \"System\"\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "root@Master:/opt/ssl# /opt/local/cfssl/cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \\\n",
    ">   -ca-key=/etc/kubernetes/ssl/ca-key.pem \\\n",
    ">   -config=/opt/ssl/config.json \\\n",
    ">   -profile=kubernetes admin-csr.json | /opt/local/cfssl/cfssljson -bare admin\n",
    "2018/10/23 14:41:32 [INFO] generate received request\n",
    "2018/10/23 14:41:32 [INFO] received CSR\n",
    "2018/10/23 14:41:32 [INFO] generating key: rsa-2048\n",
    "2018/10/23 14:41:32 [INFO] encoded CSR\n",
    "2018/10/23 14:41:32 [INFO] signed certificate with serial number 695643913881607257811736010743672424384952693292\n",
    "2018/10/23 14:41:32 [WARNING] This certificate lacks a \"hosts\" field. This makes it unsuitable for\n",
    "websites. For more information see the Baseline Requirements for the Issuance and Management\n",
    "of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);\n",
    "specifically, section 10.2.3 (\"Information Requirements\").\n",
    "root@Master:/opt/ssl# ls admin*\n",
    "admin.csr  admin-csr.json  admin-key.pem  admin.pem\n",
    "root@Master:/opt/ssl# cp admin*.pem /etc/kubernetes/ssl/\n",
    "root@Master:/opt/ssl# scp admin*.pem zdyfjh@192.168.10.11:/etc/kubernetes/ssl/   \n",
    "root@Master:/opt/ssl# scp admin*.pem zdyfjh@192.168.10.12:/etc/kubernetes/ssl/   \n",
    "root@Master:/opt/ssl# scp admin*.pem zdyfjh@192.168.10.13:/etc/kubernetes/ssl/  \n",
    "root@Master:/opt/ssl# scp admin*.pem zdyfjh@192.168.10.14:/etc/kubernetes/ssl/ \n",
    "root@Master:/opt/ssl# scp admin*.pem zdyfjh@192.168.10.15:/etc/kubernetes/ssl/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 配置Kubernetes集群\n",
    "\n",
    "```shell\n",
    "root@Master:/opt/ssl# kubectl config set-cluster kubernetes \\\n",
    ">   --certificate-authority=/etc/kubernetes/ssl/ca.pem \\\n",
    ">   --embed-certs=true \\\n",
    ">   --server=https://192.168.10.10:6443\n",
    "Cluster \"kubernetes\" set.\n",
    "root@Master:/opt/ssl# kubectl config set-credentials admin \\\n",
    ">   --client-certificate=/etc/kubernetes/ssl/admin.pem \\\n",
    ">   --embed-certs=true \\\n",
    ">   --client-key=/etc/kubernetes/ssl/admin-key.pem\n",
    "User \"admin\" set.\n",
    "root@Master:/opt/ssl# kubectl config set-context kubernetes \\\n",
    ">   --cluster=kubernetes \\\n",
    ">   --user=admin\n",
    "Context \"kubernetes\" created.\n",
    "root@Master:/opt/ssl# kubectl config use-context kubernetes\n",
    "Switched to context \"kubernetes\".\n",
    "root@Master:/opt/ssl# cd ~/.kube\n",
    "root@Master:~/.kube# ls\n",
    "config\n",
    "root@Master:~/.kube# scp * zdyfjh@192.168.10.11:~/download    \n",
    "root@Master:~/.kube# scp * zdyfjh@192.168.10.12:~/download    \n",
    "root@Master:~/.kube# scp * zdyfjh@192.168.10.13:~/download  \n",
    "root@Master:~/.kube# scp * zdyfjh@192.168.10.14:~/download   \n",
    "root@Master:~/.kube# scp * zdyfjh@192.168.10.15:~/download\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 创建SSL证书\n",
    "\n",
    "```shell\n",
    "root@Master:~/.kube# cd /opt/ssl\n",
    "root@Master:/opt/ssl# vim kubernetes-csr.json\n",
    "root@Master:/opt/ssl# cat kubernetes-csr.json \n",
    "{\n",
    "  \"CN\": \"kubernetes\",\n",
    "  \"hosts\": [\n",
    "    \"192.168.10.10\",\n",
    "    \"10.254.0.1\",\n",
    "    \"kubernetes\",\n",
    "    \"kubernetes.default\",\n",
    "    \"kubernetes.default.svc\",\n",
    "    \"kubernetes.default.svc.cluster\",\n",
    "    \"kubernetes.default.svc.cluster.local\"\n",
    "  ],\n",
    "  \"key\": {\n",
    "    \"algo\": \"rsa\",\n",
    "    \"size\": 2048\n",
    "  },\n",
    "  \"names\": [\n",
    "    {\n",
    "      \"C\": \"CN\",\n",
    "      \"ST\": \"Beijing\",\n",
    "      \"L\": \"Beijing\",\n",
    "      \"O\": \"k8s\",\n",
    "      \"OU\": \"System\"\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "root@Master:/opt/ssl# /opt/local/cfssl/cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \\\n",
    ">   -ca-key=/etc/kubernetes/ssl/ca-key.pem \\\n",
    ">   -config=/opt/ssl/config.json \\\n",
    ">   -profile=kubernetes kubernetes-csr.json | /opt/local/cfssl/cfssljson -bare kubernetes\n",
    "2018/10/23 15:21:03 [INFO] generate received request\n",
    "2018/10/23 15:21:03 [INFO] received CSR\n",
    "2018/10/23 15:21:03 [INFO] generating key: rsa-2048\n",
    "2018/10/23 15:21:03 [INFO] encoded CSR\n",
    "2018/10/23 15:21:03 [INFO] signed certificate with serial number 80697406592929186537048906789746052100450546141\n",
    "2018/10/23 15:21:03 [WARNING] This certificate lacks a \"hosts\" field. This makes it unsuitable for\n",
    "websites. For more information see the Baseline Requirements for the Issuance and Management\n",
    "of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);\n",
    "specifically, section 10.2.3 (\"Information Requirements\").\n",
    "root@Master:/opt/ssl# ls -l kubernetes*\n",
    "-rw-r--r-- 1 root root 1236 Oct 23 15:21 kubernetes.csr\n",
    "-rw-r--r-- 1 root root  420 Oct 23 15:20 kubernetes-csr.json\n",
    "-rw------- 1 root root 1675 Oct 23 15:21 kubernetes-key.pem\n",
    "-rw-r--r-- 1 root root 1602 Oct 23 15:21 kubernetes.pem\n",
    "root@Master:/opt/ssl# cp kubernetes*.pem /etc/kubernetes/ssl/\n",
    "root@Master:/opt/ssl# scp kubernetes*.pem zdyfjh@192.168.10.11:/etc/kubernetes/ssl/    \n",
    "root@Master:/opt/ssl# scp kubernetes*.pem zdyfjh@192.168.10.12:/etc/kubernetes/ssl/   \n",
    "root@Master:/opt/ssl# scp kubernetes*.pem zdyfjh@192.168.10.13:/etc/kubernetes/ssl/   \n",
    "root@Master:/opt/ssl# scp kubernetes*.pem zdyfjh@192.168.10.14:/etc/kubernetes/ssl/   \n",
    "root@Master:/opt/ssl# scp kubernetes*.pem zdyfjh@192.168.10.15:/etc/kubernetes/ssl/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 生成Token\n",
    "\n",
    "```shell\n",
    "root@Master:/opt/ssl# head -c 16 /dev/urandom | od -An -t x | tr -d ' '\n",
    "f27f606aa0bb8bea34b1e22aabd3aab8\n",
    "root@Master:/opt/ssl# vim token.csv\n",
    "root@Master:/opt/ssl# cat token.csv \n",
    "f27f606aa0bb8bea34b1e22aabd3aab8,kubelet-bootstrap,10001,\"system:kubelet-bootstrap\"\n",
    "root@Master:/opt/ssl# cp token.csv /etc/kubernetes/\n",
    "root@Master:/opt/ssl# scp token.csv zdyfjh@192.168.10.11:/etc/kubernetes/ssl/   \n",
    "root@Master:/opt/ssl# scp token.csv zdyfjh@192.168.10.12:/etc/kubernetes/ssl/   \n",
    "root@Master:/opt/ssl# scp token.csv zdyfjh@192.168.10.13:/etc/kubernetes/ssl/    \n",
    "root@Master:/opt/ssl# scp token.csv zdyfjh@192.168.10.14:/etc/kubernetes/ssl/   \n",
    "root@Master:/opt/ssl# scp token.csv zdyfjh@192.168.10.15:/etc/kubernetes/ssl/ \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 生成高级审核文件\n",
    "\n",
    "```shell\n",
    "root@Master:/opt/ssl# cd /etc/kubernetes\n",
    "root@Master:/etc/kubernetes# cat >> audit-policy.yaml <<EOF\n",
    "> # Log all requests at the Metadata level.\n",
    "> apiVersion: audit.k8s.io/v1beta1\n",
    "> kind: Policy\n",
    "> rules:\n",
    "> - level: Metadata\n",
    "> EOF\n",
    "root@Master:/etc/kubernetes# scp audit-policy.yaml zdyfjh@192.168.10.11:/etc/kubernetes/ssl/  \n",
    "root@Master:/etc/kubernetes# scp audit-policy.yaml zdyfjh@192.168.10.12:/etc/kubernetes/ssl/    \n",
    "root@Master:/etc/kubernetes# scp audit-policy.yaml zdyfjh@192.168.10.13:/etc/kubernetes/ssl/   \n",
    "root@Master:/etc/kubernetes# scp audit-policy.yaml zdyfjh@192.168.10.14:/etc/kubernetes/ssl/  \n",
    "root@Master:/etc/kubernetes# scp audit-policy.yaml zdyfjh@192.168.10.15:/etc/kubernetes/ssl/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 配置APIServer的service文件并启动APIServer\n",
    "\n",
    "```shell\n",
    "root@Master:/etc/kubernetes# vim /etc/systemd/system/kube-apiserver.service\n",
    "root@Master:/etc/kubernetes# cat /etc/systemd/system/kube-apiserver.service\n",
    "[Unit]\n",
    "Description=Kubernetes API Server\n",
    "Documentation=https://github.com/GoogleCloudPlatform/kubernetes\n",
    "After=network.target\n",
    "\n",
    "[Service]\n",
    "User=root\n",
    "ExecStart=/usr/local/bin/kube-apiserver \\\n",
    "  --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \\\n",
    "  --advertise-address=192.168.10.10 \\\n",
    "  --allow-privileged=true \\\n",
    "  --apiserver-count=1 \\\n",
    "  --audit-policy-file=/etc/kubernetes/audit-policy.yaml \\\n",
    "  --audit-log-maxage=30 \\\n",
    "  --audit-log-maxbackup=3 \\\n",
    "  --audit-log-maxsize=100 \\\n",
    "  --audit-log-path=/var/log/kubernetes/audit.log \\\n",
    "  --authorization-mode=Node,RBAC \\\n",
    "  --bind-address=0.0.0.0 \\\n",
    "  --secure-port=6443 \\\n",
    "  --client-ca-file=/etc/kubernetes/ssl/ca.pem \\\n",
    "  --enable-swagger-ui=true \\\n",
    "  --etcd-cafile=/etc/kubernetes/ssl/ca.pem \\\n",
    "  --etcd-certfile=/etc/kubernetes/ssl/etcd.pem \\\n",
    "  --etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem \\\n",
    "  --etcd-servers=https://192.168.10.10:2379,https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379,https://192.168.10.15:2379 \\\n",
    "  --event-ttl=1h \\\n",
    "  --kubelet-https=true \\\n",
    "  --insecure-bind-address=192.168.10.10 \\\n",
    "  --insecure-port=8080 \\\n",
    "  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \\\n",
    "  --service-cluster-ip-range=10.254.0.0/18 \\\n",
    "  --service-node-port-range=30000-32000 \\\n",
    "  --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \\\n",
    "  --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\\n",
    "  --enable-bootstrap-token-auth \\\n",
    "  --token-auth-file=/etc/kubernetes/token.csv \\\n",
    "  --v=1\n",
    "Restart=on-failure\n",
    "RestartSec=5\n",
    "Type=notify\n",
    "LimitNOFILE=65536\n",
    "\n",
    "[Install]\n",
    "WantedBy=multi-user.target\n",
    "root@Master:/etc/kubernetes# systemctl daemon-reload\n",
    "root@Master:/etc/kubernetes# systemctl enable kube-apiserver\n",
    "Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /etc/systemd/system/kube-apiserver.service.\n",
    "root@Master:/etc/kubernetes# systemctl start kube-apiserver\n",
    "root@Master:/etc/kubernetes# systemctl status kube-apiserver\n",
    "● kube-apiserver.service - Kubernetes API Server\n",
    "   Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled)\n",
    "   Active: active (running) since Tue 2018-10-23 15:49:49 CST; 8s ago\n",
    "     Docs: https://github.com/GoogleCloudPlatform/kubernetes\n",
    " Main PID: 38924 (kube-apiserver)\n",
    "    Tasks: 55\n",
    "   Memory: 227.7M\n",
    "      CPU: 8.712s\n",
    "   CGroup: /system.slice/kube-apiserver.service\n",
    "           └─38924 /usr/local/bin/kube-apiserver --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,Resour\n",
    "\n",
    "Oct 23 15:49:52 Master kube-apiserver[38924]: I1023 15:49:52.011284   38924 storage_rbac.go:267] created role.rbac.authorization.k8s.io/syste\n",
    "Oct 23 15:49:52 Master kube-apiserver[38924]: I1023 15:49:52.047667   38924 storage_rbac.go:267] created role.rbac.authorization.k8s.io/syste\n",
    "Oct 23 15:49:52 Master kube-apiserver[38924]: I1023 15:49:52.091192   38924 storage_rbac.go:267] created role.rbac.authorization.k8s.io/syste\n",
    "Oct 23 15:49:52 Master kube-apiserver[38924]: I1023 15:49:52.125045   38924 controller.go:538] quota admission added evaluator for: {rbac.aut\n",
    "Oct 23 15:49:52 Master kube-apiserver[38924]: I1023 15:49:52.128525   38924 storage_rbac.go:297] created rolebinding.rbac.authorization.k8s.i\n",
    "Oct 23 15:49:52 Master kube-apiserver[38924]: I1023 15:49:52.167760   38924 storage_rbac.go:297] created rolebinding.rbac.authorization.k8s.i\n",
    "Oct 23 15:49:52 Master kube-apiserver[38924]: I1023 15:49:52.208015   38924 storage_rbac.go:297] created rolebinding.rbac.authorization.k8s.i\n",
    "Oct 23 15:49:52 Master kube-apiserver[38924]: I1023 15:49:52.247990   38924 storage_rbac.go:297] created rolebinding.rbac.authorization.k8s.i\n",
    "Oct 23 15:49:52 Master kube-apiserver[38924]: I1023 15:49:52.288011   38924 storage_rbac.go:297] created rolebinding.rbac.authorization.k8s.i\n",
    "Oct 23 15:49:52 Master kube-apiserver[38924]: I1023 15:49:52.327823   38924 storage_rbac.go:297] created rolebinding.rbac.authorization.k8s.i\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 配置kube-controller的service文件并启动\n",
    "\n",
    "```shell\n",
    "root@Master:/etc/kubernetes# vim /etc/systemd/system/kube-controller-manager.service\n",
    "root@Master:/etc/kubernetes# cat /etc/systemd/system/kube-controller-manager.service \n",
    "[Unit]\n",
    "Description=Kubernetes Controller Manager\n",
    "Documentation=https://github.com/GoogleCloudPlatform/kubernetes\n",
    "\n",
    "[Service]\n",
    "ExecStart=/usr/local/bin/kube-controller-manager \\\n",
    "  --address=0.0.0.0 \\\n",
    "  --master=http://192.168.10.10:8080 \\\n",
    "  --allocate-node-cidrs=true \\\n",
    "  --service-cluster-ip-range=10.254.0.0/18 \\\n",
    "  --cluster-cidr=10.254.64.0/18 \\\n",
    "  --cluster-name=kubernetes \\\n",
    "  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \\\n",
    "  --root-ca-file=/etc/kubernetes/ssl/ca.pem \\\n",
    "  --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \\\n",
    "  --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \\\n",
    "  --leader-elect=true \\\n",
    "  --v=1\n",
    "Restart=on-failure\n",
    "RestartSec=5\n",
    "\n",
    "[Install]\n",
    "WantedBy=multi-user.target\n",
    "root@Master:/etc/kubernetes# systemctl daemon-reload\n",
    "root@Master:/etc/kubernetes# systemctl enable kube-controller-manager\n",
    "Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /etc/systemd/system/kube-controller-manager.service.\n",
    "root@Master:/etc/kubernetes# systemctl start kube-controller-manager\n",
    "root@Master:/etc/kubernetes# systemctl status kube-controller-manager\n",
    "● kube-controller-manager.service - Kubernetes Controller Manager\n",
    "   Loaded: loaded (/etc/systemd/system/kube-controller-manager.service; enabled; vendor preset: enabled)\n",
    "   Active: active (running) since Tue 2018-10-23 15:53:31 CST; 6s ago\n",
    "     Docs: https://github.com/GoogleCloudPlatform/kubernetes\n",
    " Main PID: 39196 (kube-controller)\n",
    "    Tasks: 30\n",
    "   Memory: 12.3M\n",
    "      CPU: 180ms\n",
    "   CGroup: /system.slice/kube-controller-manager.service\n",
    "           └─39196 /usr/local/bin/kube-controller-manager --address=0.0.0.0 --master=http://192.168.10.10:8080 --allocate-node-cidrs=true --s\n",
    "\n",
    "Oct 23 15:53:31 Master systemd[1]: Started Kubernetes Controller Manager.\n",
    "Oct 23 15:53:31 Master kube-controller-manager[39196]: I1023 15:53:31.803429   39196 controllermanager.go:108] Version: v1.9.6\n",
    "Oct 23 15:53:31 Master kube-controller-manager[39196]: I1023 15:53:31.804194   39196 leaderelection.go:174] attempting to acquire leader leas\n",
    "Oct 23 15:53:31 Master kube-controller-manager[39196]: E1023 15:53:31.805463   39196 leaderelection.go:224] error retrieving resource lock ku\n",
    "Oct 23 15:53:35 Master kube-controller-manager[39196]: E1023 15:53:35.257755   39196 leaderelection.go:224] error retrieving resource lock ku\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 配置scheduler的service文件并启动\n",
    "\n",
    "```shell\n",
    "root@Master:/etc/kubernetes# vim /etc/systemd/system/kube-scheduler.service\n",
    "root@Master:/etc/kubernetes# cat /etc/systemd/system/kube-scheduler.service \n",
    "[Unit]\n",
    "Description=Kubernetes Scheduler\n",
    "Documentation=https://github.com/GoogleCloudPlatform/kubernetes\n",
    "\n",
    "[Service]\n",
    "ExecStart=/usr/local/bin/kube-scheduler \\\n",
    "  --address=0.0.0.0 \\\n",
    "  --master=http://192.168.10.10:8080 \\\n",
    "  --leader-elect=true \\\n",
    "  --v=1\n",
    "Restart=on-failure\n",
    "RestartSec=5\n",
    "\n",
    "[Install]\n",
    "WantedBy=multi-user.target\n",
    "root@Master:/etc/kubernetes# systemctl daemon-reload\n",
    "root@Master:/etc/kubernetes# systemctl enable kube-scheduler\n",
    "Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /etc/systemd/system/kube-scheduler.service.\n",
    "root@Master:/etc/kubernetes# systemctl start kube-scheduler\n",
    "root@Master:/etc/kubernetes# systemctl status kube-scheduler\n",
    "● kube-scheduler.service - Kubernetes Scheduler\n",
    "   Loaded: loaded (/etc/systemd/system/kube-scheduler.service; enabled; vendor preset: enabled)\n",
    "   Active: active (running) since Tue 2018-10-23 15:57:13 CST; 6s ago\n",
    "     Docs: https://github.com/GoogleCloudPlatform/kubernetes\n",
    " Main PID: 39435 (kube-scheduler)\n",
    "    Tasks: 33\n",
    "   Memory: 14.5M\n",
    "      CPU: 253ms\n",
    "   CGroup: /system.slice/kube-scheduler.service\n",
    "           └─39435 /usr/local/bin/kube-scheduler --address=0.0.0.0 --master=http://192.168.10.10:8080 --leader-elect=true --v=1\n",
    "\n",
    "Oct 23 15:57:18 Master kube-scheduler[39435]: E1023 15:57:18.155660   39435 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/infor\n",
    "Oct 23 15:57:19 Master kube-scheduler[39435]: E1023 15:57:19.146464   39435 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/infor\n",
    "Oct 23 15:57:19 Master kube-scheduler[39435]: E1023 15:57:19.147634   39435 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/infor\n",
    "Oct 23 15:57:19 Master kube-scheduler[39435]: E1023 15:57:19.148800   39435 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/infor\n",
    "Oct 23 15:57:19 Master kube-scheduler[39435]: E1023 15:57:19.149888   39435 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/infor\n",
    "Oct 23 15:57:19 Master kube-scheduler[39435]: E1023 15:57:19.151873   39435 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/infor\n",
    "Oct 23 15:57:19 Master kube-scheduler[39435]: E1023 15:57:19.152995   39435 reflector.go:205] k8s.io/kubernetes/plugin/cmd/kube-scheduler/app\n",
    "Oct 23 15:57:19 Master kube-scheduler[39435]: E1023 15:57:19.154028   39435 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/infor\n",
    "Oct 23 15:57:19 Master kube-scheduler[39435]: E1023 15:57:19.155108   39435 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/infor\n",
    "Oct 23 15:57:19 Master kube-scheduler[39435]: E1023 15:57:19.156241   39435 reflector.go:205] k8s.io/kubernetes/vendor/k8s.io/client-go/infor\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 验证安装工作\n",
    "\n",
    "```shell\n",
    "root@Master:/etc/kubernetes# kubectl get componentstatuses\n",
    "NAME                 STATUS    MESSAGE             ERROR\n",
    "scheduler            Healthy   ok                  \n",
    "controller-manager   Healthy   ok                  \n",
    "etcd-2               Healthy   {\"health\":\"true\"}   \n",
    "etcd-1               Healthy   {\"health\":\"true\"}   \n",
    "etcd-5               Healthy   {\"health\":\"true\"}   \n",
    "etcd-0               Healthy   {\"health\":\"true\"}   \n",
    "etcd-4               Healthy   {\"health\":\"true\"}   \n",
    "etcd-3               Healthy   {\"health\":\"true\"}   \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 配置kubelet\n",
    "\n",
    "```shell\n",
    "root@Master:/etc/kubernetes# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap\n",
    "clusterrolebinding \"kubelet-bootstrap\" created\n",
    "root@Master:/etc/kubernetes# kubectl config set-cluster kubernetes \\\n",
    ">   --certificate-authority=/etc/kubernetes/ssl/ca.pem \\\n",
    ">   --embed-certs=true \\\n",
    ">   --server=https://192.168.10.10:6443 \\\n",
    ">   --kubeconfig=bootstrap.kubeconfig\n",
    "Cluster \"kubernetes\" set.\n",
    "root@Master:/etc/kubernetes# kubectl config set-credentials kubelet-bootstrap \\\n",
    ">   --token=f27f606aa0bb8bea34b1e22aabd3aab8 \\\n",
    ">   --kubeconfig=bootstrap.kubeconfig\n",
    "User \"kubelet-bootstrap\" set.\n",
    "root@Master:/etc/kubernetes# kubectl config set-context default \\\n",
    ">   --cluster=kubernetes \\\n",
    ">   --user=kubelet-bootstrap \\\n",
    ">   --kubeconfig=bootstrap.kubeconfig\n",
    "Context \"default\" created.\n",
    "root@Master:/etc/kubernetes# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig\n",
    "Switched to context \"default\".\n",
    "root@Master:/etc/kubernetes# ls\n",
    "audit-policy.yaml  bootstrap.kubeconfig  ssl  token.csv\n",
    "root@Master:/etc/kubernetes# mv bootstrap.kubeconfig /etc/kubernetes/\n",
    "mv: 'bootstrap.kubeconfig' and '/etc/kubernetes/bootstrap.kubeconfig' are the same file\n",
    "root@Master:/etc/kubernetes# scp bootstrap.kubeconfig zdyfjh@192.168.10.11:/etc/kubernetes/ssl/  \n",
    "root@Master:/etc/kubernetes# scp bootstrap.kubeconfig zdyfjh@192.168.10.12:/etc/kubernetes/ssl/   \n",
    "root@Master:/etc/kubernetes# scp bootstrap.kubeconfig zdyfjh@192.168.10.13:/etc/kubernetes/ssl/  \n",
    "root@Master:/etc/kubernetes# scp bootstrap.kubeconfig zdyfjh@192.168.10.14:/etc/kubernetes/ssl/   \n",
    "root@Master:/etc/kubernetes# scp bootstrap.kubeconfig zdyfjh@192.168.10.15:/etc/kubernetes/ssl/ \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 创建kubelet的service文件并启动\n",
    "\n",
    "```shell\n",
    "root@Master:/etc/kubernetes# mkdir /var/lib/kubelet\n",
    "root@Master:/etc/kubernetes# vim /etc/systemd/system/kubelet.service\n",
    "root@Master:/etc/kubernetes# cat /etc/systemd/system/kubelet.service \n",
    "[Unit]\n",
    "Description=Kubernetes Kubelet\n",
    "Documentation=https://github.com/GoogleCloudPlatform/kubernetes\n",
    "After=docker.service\n",
    "Requires=docker.service\n",
    "\n",
    "[Service]\n",
    "WorkingDirectory=/var/lib/kubelet\n",
    "ExecStart=/usr/local/bin/kubelet \\\n",
    "  --cgroup-driver=cgroupfs \\\n",
    "  --hostname-override=Master \\ #改成自己的hostname\n",
    "  --pod-infra-container-image=jicki/pause-amd64:3.0 \\ #需要预先从仓库里把这个镜像pull到本地\n",
    "  --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \\\n",
    "  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\\n",
    "  --cert-dir=/etc/kubernetes/ssl \\\n",
    "  --cluster_dns=10.254.0.2 \\\n",
    "  --cluster_domain=cluster.local \\\n",
    "  --hairpin-mode=promiscuous-bridge \\\n",
    "  --allow-privileged=true \\\n",
    "  --fail-swap-on=false \\\n",
    "  --serialize-image-pulls=false \\\n",
    "  --logtostderr=true \\\n",
    "  --max-pods=512 \\\n",
    "  --v=1\n",
    "\n",
    "[Install]\n",
    "WantedBy=multi-user.target\n",
    "root@Master:/etc/kubernetes# docker pull jicki/pause-amd64:3.0\n",
    "3.0: Pulling from jicki/pause-amd64\n",
    "a3ed95caeb02: Pull complete \n",
    "f11233434377: Pull complete \n",
    "Digest: sha256:3b3a29e3c90ae7762bdf587d19302e62485b6bef46e114b741f7d75dba023bd3\n",
    "Status: Downloaded newer image for jicki/pause-amd64:3.0\n",
    "root@Master:/etc/kubernetes# systemctl daemon-reload\n",
    "root@Master:/etc/kubernetes# systemctl enable kubelet\n",
    "Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.\n",
    "root@Master:/etc/kubernetes# systemctl start kubelet\n",
    "root@Master:/etc/kubernetes# systemctl status kubelet\n",
    "● kubelet.service - Kubernetes Kubelet\n",
    "   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)\n",
    "   Active: active (running) since Tue 2018-10-23 16:19:40 CST; 8s ago\n",
    "     Docs: https://github.com/GoogleCloudPlatform/kubernetes\n",
    " Main PID: 40677 (kubelet)\n",
    "    Tasks: 30\n",
    "   Memory: 16.8M\n",
    "      CPU: 620ms\n",
    "   CGroup: /system.slice/kubelet.service\n",
    "           └─40677 /usr/local/bin/kubelet --cgroup-driver=cgroupfs --hostname-override=Master --pod-infra-container-image=jicki/pause-amd64:3\n",
    "\n",
    "Oct 23 16:19:40 Master systemd[1]: Started Kubernetes Kubelet.\n",
    "Oct 23 16:19:40 Master kubelet[40677]: I1023 16:19:40.198672   40677 feature_gate.go:226] feature gates: &{{} map[]}\n",
    "Oct 23 16:19:40 Master kubelet[40677]: I1023 16:19:40.198855   40677 controller.go:114] kubelet config controller: starting controller\n",
    "Oct 23 16:19:40 Master kubelet[40677]: I1023 16:19:40.198872   40677 controller.go:118] kubelet config controller: validating combination of \n",
    "Oct 23 16:19:40 Master kubelet[40677]: W1023 16:19:40.615114   40677 cni.go:171] Unable to update cni config: No networks found in /etc/cni/n\n",
    "Oct 23 16:19:40 Master kubelet[40677]: I1023 16:19:40.620213   40677 server.go:182] Version: v1.9.6\n",
    "Oct 23 16:19:40 Master kubelet[40677]: I1023 16:19:40.620268   40677 feature_gate.go:226] feature gates: &{{} map[]}\n",
    "Oct 23 16:19:40 Master kubelet[40677]: I1023 16:19:40.620378   40677 plugins.go:101] No cloud provider specified.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 配置TLS认证\n",
    "\n",
    "```shell\n",
    "root@Master:/etc/kubernetes/ssl# kubectl get csr\n",
    "NAME                                                   AGE       REQUESTOR           CONDITION\n",
    "node-csr-eufLGPzV7r2HVCmqCAanrltdfrcf1c96ixfn0gKACL0   1m        kubelet-bootstrap   Pending\n",
    "root@Master:/etc/kubernetes/ssl# kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve\n",
    "certificatesigningrequest \"node-csr-eufLGPzV7r2HVCmqCAanrltdfrcf1c96ixfn0gKACL0\" approved\n",
    "root@Master:/etc/kubernetes/ssl# kubectl get csr\n",
    "NAME                                                   AGE       REQUESTOR           CONDITION\n",
    "node-csr-eufLGPzV7r2HVCmqCAanrltdfrcf1c96ixfn0gKACL0   1m        kubelet-bootstrap   Approved,Issued\n",
    "root@Master:/etc/kubernetes/ssl# kubectl get node\n",
    "NAME      STATUS    ROLES     AGE       VERSION\n",
    "master    Ready     <none>    1h        v1.9.6\n",
    "root@Master:/etc/kubernetes# ls /etc/kubernetes/kubelet.kubeconfig \n",
    "/etc/kubernetes/kubelet.kubeconfig\n",
    "root@Master:/etc/kubernetes# ls /etc/kubernetes/ssl/kubelet*\n",
    "/etc/kubernetes/ssl/kubelet-client.crt  /etc/kubernetes/ssl/kubelet.crt\n",
    "/etc/kubernetes/ssl/kubelet-client.key  /etc/kubernetes/ssl/kubelet.key\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 配置kube-proxy证书\n",
    "\n",
    "```shell\n",
    "root@Master:/etc/kubernetes# cd /opt/ssl\n",
    "root@Master:/opt/ssl# vim kube-proxy-csr.json\n",
    "root@Master:/opt/ssl# cat kube-proxy-csr.json \n",
    "{\n",
    "  \"CN\": \"system:kube-proxy\",\n",
    "  \"hosts\": [],\n",
    "  \"key\": {\n",
    "    \"algo\": \"rsa\",\n",
    "    \"size\": 2048\n",
    "  },\n",
    "  \"names\": [\n",
    "    {\n",
    "      \"C\": \"CN\",\n",
    "      \"ST\": \"Beijing\",\n",
    "      \"L\": \"Beijing\",\n",
    "      \"O\": \"k8s\",\n",
    "      \"OU\": \"System\"\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "root@Master:/opt/ssl# /opt/local/cfssl/cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \\\n",
    ">   -ca-key=/etc/kubernetes/ssl/ca-key.pem \\\n",
    ">   -config=/opt/ssl/config.json \\\n",
    ">   -profile=kubernetes  kube-proxy-csr.json | /opt/local/cfssl/cfssljson -bare kube-proxy\n",
    "2018/10/23 20:06:17 [INFO] generate received request\n",
    "2018/10/23 20:06:17 [INFO] received CSR\n",
    "2018/10/23 20:06:17 [INFO] generating key: rsa-2048\n",
    "2018/10/23 20:06:17 [INFO] encoded CSR\n",
    "2018/10/23 20:06:17 [INFO] signed certificate with serial number 36413578466703637328292670519988450625987607631\n",
    "2018/10/23 20:06:17 [WARNING] This certificate lacks a \"hosts\" field. This makes it unsuitable for\n",
    "websites. For more information see the Baseline Requirements for the Issuance and Management\n",
    "of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);\n",
    "specifically, section 10.2.3 (\"Information Requirements\").\n",
    "root@Master:/opt/ssl# ls kube-proxy*\n",
    "kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem\n",
    "root@Master:/opt/ssl# cp kube-proxy* /etc/kubernetes/ssl/\n",
    "root@Master:/opt/ssl# scp kube-proxy* zdyfjh@192.168.10.11:/etc/kubernetes/ssl/ \n",
    "root@Master:/opt/ssl# scp kube-proxy* zdyfjh@192.168.10.12:/etc/kubernetes/ssl/   \n",
    "root@Master:/opt/ssl# scp kube-proxy* zdyfjh@192.168.10.13:/etc/kubernetes/ssl/  \n",
    "root@Master:/opt/ssl# scp kube-proxy* zdyfjh@192.168.10.14:/etc/kubernetes/ssl/    \n",
    "root@Master:/opt/ssl# scp kube-proxy* zdyfjh@192.168.10.15:/etc/kubernetes/ssl/\n",
    "root@Master:/opt/ssl# kubectl config set-cluster kubernetes \\\n",
    ">   --certificate-authority=/etc/kubernetes/ssl/ca.pem \\\n",
    ">   --embed-certs=true \\\n",
    ">   --server=https://192.168.10.10:6443 \\\n",
    ">   --kubeconfig=kube-proxy.kubeconfig\n",
    "Cluster \"kubernetes\" set.\n",
    "root@Master:/opt/ssl# kubectl config set-credentials kube-proxy \\\n",
    ">   --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \\\n",
    ">   --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \\\n",
    ">   --embed-certs=true \\\n",
    ">   --kubeconfig=kube-proxy.kubeconfig\n",
    "User \"kube-proxy\" set.\n",
    "root@Master:/opt/ssl# kubectl config set-context default \\\n",
    ">   --cluster=kubernetes \\\n",
    ">   --user=kube-proxy \\\n",
    ">   --kubeconfig=kube-proxy.kubeconfig\n",
    "Context \"default\" created.\n",
    "root@Master:/opt/ssl# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig\n",
    "Switched to context \"default\".\n",
    "root@Master:/opt/ssl# cp kube-proxy.kubeconfig /etc/kubernetes/\n",
    "root@Master:/opt/ssl# cd /etc/kubernetes/\n",
    "root@Master:/etc/kubernetes# ls\n",
    "audit-policy.yaml  bootstrap.kubeconfig  kubelet.kubeconfig  kube-proxy.kubeconfig  ssl  token.csv\n",
    "root@Master:/etc/kubernetes# scp kube-proxy.kubeconfig zdyfjh@192.168.10.11:/etc/kubernetes/ssl    \n",
    "root@Master:/etc/kubernetes# scp kube-proxy.kubeconfig zdyfjh@192.168.10.12:/etc/kubernetes/ssl   \n",
    "root@Master:/etc/kubernetes# scp kube-proxy.kubeconfig zdyfjh@192.168.10.13:/etc/kubernetes/ssl   \n",
    "root@Master:/etc/kubernetes# scp kube-proxy.kubeconfig zdyfjh@192.168.10.14:/etc/kubernetes/ssl   \n",
    "root@Master:/etc/kubernetes# scp kube-proxy.kubeconfig zdyfjh@192.168.10.15:/etc/kubernetes/ssl                   \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 配置kube-proxy的service文件并启动\n",
    "\n",
    "```shell\n",
    "root@Master:/etc/kubernetes# mkdir -p /var/lib/kube-proxy\n",
    "root@Master:/etc/kubernetes# vim /etc/systemd/system/kube-proxy.service\n",
    "root@Master:/etc/kubernetes# cat /etc/systemd/system/kube-proxy.service \n",
    "[Unit]\n",
    "Description=Kubernetes Kube-Proxy Server\n",
    "Documentation=https://github.com/GoogleCloudPlatform/kubernetes\n",
    "After=network.target\n",
    "\n",
    "[Service]\n",
    "WorkingDirectory=/var/lib/kube-proxy\n",
    "ExecStart=/usr/local/bin/kube-proxy \\\n",
    "  --bind-address=192.168.10.10 \\\n",
    "  --hostname-override=k8s-master-47 \\\n",
    "  --cluster-cidr=10.254.64.0/18 \\\n",
    "  --masquerade-all \\\n",
    "  --feature-gates=SupportIPVSProxyMode=true \\\n",
    "  --proxy-mode=ipvs \\\n",
    "  --ipvs-min-sync-period=5s \\\n",
    "  --ipvs-sync-period=5s \\\n",
    "  --ipvs-scheduler=rr \\\n",
    "  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \\\n",
    "  --logtostderr=true \\\n",
    "  --v=1\n",
    "Restart=on-failure\n",
    "RestartSec=5\n",
    "LimitNOFILE=65536\n",
    "\n",
    "[Install]\n",
    "WantedBy=multi-user.target\n",
    "root@Master:/etc/kubernetes# systemctl daemon-reload\n",
    "root@Master:/etc/kubernetes# systemctl enable kube-proxy\n",
    "Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /etc/systemd/system/kube-proxy.service.\n",
    "root@Master:/etc/kubernetes# systemctl start kube-proxy\n",
    "root@Master:/etc/kubernetes# systemctl status kube-proxy\n",
    "● kube-proxy.service - Kubernetes Kube-Proxy Server\n",
    "   Loaded: loaded (/etc/systemd/system/kube-proxy.service; enabled; vendor preset: enabled)\n",
    "   Active: active (running) since Tue 2018-10-23 20:21:56 CST; 37s ago\n",
    "     Docs: https://github.com/GoogleCloudPlatform/kubernetes\n",
    " Main PID: 13444 (kube-proxy)\n",
    "    Tasks: 0\n",
    "   Memory: 13.6M\n",
    "      CPU: 261ms\n",
    "   CGroup: /system.slice/kube-proxy.service\n",
    "           ‣ 13444 /usr/local/bin/kube-proxy --bind-address=192.168.10.10 --hostname-override=k8s-master-47 --cluster-cidr=10.254.64.0/18 --m\n",
    "\n",
    "Oct 23 20:21:56 Master kube-proxy[13444]: I1023 20:21:56.990793   13444 conntrack.go:83] Setting conntrack hashsize to 393216\n",
    "Oct 23 20:21:57 Master kube-proxy[13444]: I1023 20:21:57.045973   13444 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_e\n",
    "Oct 23 20:21:57 Master kube-proxy[13444]: I1023 20:21:57.046196   13444 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_c\n",
    "Oct 23 20:21:57 Master kube-proxy[13444]: I1023 20:21:57.046457   13444 config.go:102] Starting endpoints config controller\n",
    "Oct 23 20:21:57 Master kube-proxy[13444]: I1023 20:21:57.046495   13444 controller_utils.go:1019] Waiting for caches to sync for endpoints co\n",
    "Oct 23 20:21:57 Master kube-proxy[13444]: I1023 20:21:57.046592   13444 config.go:202] Starting service config controller\n",
    "Oct 23 20:21:57 Master kube-proxy[13444]: I1023 20:21:57.046674   13444 controller_utils.go:1019] Waiting for caches to sync for service conf\n",
    "Oct 23 20:21:57 Master kube-proxy[13444]: I1023 20:21:57.146696   13444 controller_utils.go:1026] Caches are synced for endpoints config cont\n",
    "Oct 23 20:21:57 Master kube-proxy[13444]: I1023 20:21:57.146855   13444 controller_utils.go:1026] Caches are synced for service config contro\n",
    "Oct 23 20:21:57 Master kube-proxy[13444]: I1023 20:21:57.146960   13444 proxier.go:329] Adding new service port \"default/kubernetes:https\" at\n",
    "root@Master:/etc/kubernetes# ipvsadm -L -n\n",
    "IP Virtual Server version 1.2.1 (size=4096)\n",
    "Prot LocalAddress:Port Scheduler Flags\n",
    "  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn\n",
    "root@Master:/etc/kubernetes# cd /etc/kubernetes/ssl/\n",
    "root@Master:/etc/kubernetes/ssl# scp ca.pem kube-proxy.pem kube-proxy-key.pem zdyfjh@192.168.10.11:/etc/kubernetes/ssl \n",
    "root@Master:/etc/kubernetes/ssl# scp ca.pem kube-proxy.pem kube-proxy-key.pem zdyfjh@192.168.10.12:/etc/kubernetes/ssl \n",
    "root@Master:/etc/kubernetes/ssl# scp ca.pem kube-proxy.pem kube-proxy-key.pem zdyfjh@192.168.10.13:/etc/kubernetes/ssl\n",
    "root@Master:/etc/kubernetes/ssl# scp ca.pem kube-proxy.pem kube-proxy-key.pem zdyfjh@192.168.10.14:/etc/kubernetes/ssl\n",
    "root@Master:/etc/kubernetes/ssl# scp ca.pem kube-proxy.pem kube-proxy-key.pem zdyfjh@192.168.10.15:/etc/kubernetes/ssl\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slave节点上\n",
    "\n",
    "* ### 安装Kubernetes相关组件及配置\n",
    "\n",
    "```shell\n",
    "root@Slave:/home/zdyfjh/download# tar -zxvf kubernetes-server-linux-amd64.tar.gz \n",
    "root@Slave:/home/zdyfjh/download# ls\n",
    "etcd-v3.3.10-linux-amd64         kubernetes\n",
    "etcd-v3.3.10-linux-amd64.tar.gz  kubernetes-server-linux-amd64.tar.gz\n",
    "root@Slave:/home/zdyfjh/download# cd ./kubernetes\n",
    "root@Slave:/home/zdyfjh/download/kubernetes# ls\n",
    "LICENSES  addons  kubernetes-src.tar.gz  server\n",
    "root@Slave:/home/zdyfjh/download/kubernetes# cp server/bin/{kube-proxy,kubelet,kubectl} /usr/local/bin/\n",
    "root@Slave:/home/zdyfjh/download/kubernetes# cd ~\n",
    "root@Slave:~# mkdir -p .kube\n",
    "root@Slave:~# mv /home/zdyfjh/download/config .kube\n",
    "root@Slave:~# cd .kube\n",
    "root@Slave:~/.kube# chown -R root:root config \n",
    "root@Slave:~/.kube# ll\n",
    "total 16\n",
    "drwxr-xr-x 2 root root 4096 Oct 23 15:09 ./\n",
    "drwx------ 6 root root 4096 Oct 23 15:09 ../\n",
    "-rw------- 1 root root 6287 Oct 23 15:09 config\n",
    "```\n",
    "\n",
    "* ### 移动Token文件及高级审核文件\n",
    "\n",
    "```shell\n",
    "root@Slave:/home/zdyfjh# cd /etc/kubernetes/ssl/\n",
    "root@Slave:/etc/kubernetes/ssl# ls\n",
    "admin-key.pem      ca-key.pem  config.json   etcd.pem            token.csv\n",
    "admin.pem          ca.csr      csr.json      kubernetes-key.pem\n",
    "audit-policy.yaml  ca.pem      etcd-key.pem  kubernetes.pem\n",
    "root@Slave:/etc/kubernetes/ssl# mv token.csv ../\n",
    "root@Slave:/etc/kubernetes/ssl# mv audit-policy.yaml ../\n",
    "root@Slave:/etc/kubernetes/ssl# cd ../\n",
    "root@Slave:/etc/kubernetes# chown -R root:root audit-policy.yaml \n",
    "root@Slave:/etc/kubernetes# chown -R root:root token.csv \n",
    "root@Slave:/etc/kubernetes# ll\n",
    "total 28\n",
    "drwxr-xr-x   3 root   root    4096 Oct 23 15:35 ./\n",
    "drwxr-xr-x 138 root   root   12288 Oct 23 10:27 ../\n",
    "-rw-r--r--   1 root   root     113 Oct 23 15:33 audit-policy.yaml\n",
    "drwxr-xr-x   2 zdyfjh zdyfjh  4096 Oct 23 15:35 ssl/\n",
    "-rw-r--r--   1 root   root      84 Oct 23 15:28 token.csv\n",
    "```\n",
    "\n",
    "* ### 移动配置文件并创建工作目录\n",
    "\n",
    "```shell\n",
    "root@Slave:/home/zdyfjh# cd /etc/kubernetes/ssl/\n",
    "root@Slave:/etc/kubernetes/ssl# mv bootstrap.kubeconfig ../\n",
    "root@Slave:/etc/kubernetes/ssl# cd ../                \n",
    "root@Slave:/etc/kubernetes# chown -R root:root bootstrap.kubeconfig \n",
    "root@Slave:/etc/kubernetes# ll\n",
    "total 32\n",
    "drwxr-xr-x   3 root   root    4096 Oct 23 16:10 ./\n",
    "drwxr-xr-x 138 root   root   12288 Oct 23 10:27 ../\n",
    "-rw-r--r--   1 root   root     113 Oct 23 15:33 audit-policy.yaml\n",
    "-rw-------   1 root   root    2189 Oct 23 16:05 bootstrap.kubeconfig\n",
    "drwxr-xr-x   2 zdyfjh zdyfjh  4096 Oct 23 16:10 ssl/\n",
    "-rw-r--r--   1 root   root      84 Oct 23 15:28 token.csv\n",
    "```\n",
    "\n",
    "* ### 移动kube-proxy配置文件\n",
    "\n",
    "```shell\n",
    "root@Slave:/home/zdyfjh# cd /etc/kubernetes/ssl\n",
    "root@Slave:/etc/kubernetes/ssl# mv kube-proxy.kubeconfig ../\n",
    "root@Slave:/etc/kubernetes/ssl# cd ../\n",
    "root@Slave:/etc/kubernetes# chown -R root:root kube-proxy.kubeconfig \n",
    "root@Slave:/etc/kubernetes# ll\n",
    "total 40\n",
    "drwxr-xr-x   3 root   root    4096 Oct 23 20:15 ./\n",
    "drwxr-xr-x 135 root   root   12288 Oct 23 10:30 ../\n",
    "-rw-r--r--   1 root   root     113 Oct 23 15:34 audit-policy.yaml\n",
    "-rw-------   1 root   root    2185 Oct 23 19:41 bootstrap.kubeconfig\n",
    "-rw-------   1 root   root    6295 Oct 23 20:12 kube-proxy.kubeconfig\n",
    "drwxr-xr-x   2 zdyfjh zdyfjh  4096 Oct 23 20:15 ssl/\n",
    "-rw-r--r--   1 root   root      84 Oct 23 19:33 token.csv\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 创建kubelet的service文件并启动\n",
    "\n",
    "```shell\n",
    "root@Slave:/home/zdyfjh# cd /etc/kubernetes/\n",
    "root@Slave:/etc/kubernetes# ls\n",
    "audit-policy.yaml  bootstrap.kubeconfig  kube-proxy.kubeconfig  ssl  token.csv\n",
    "root@Slave:/etc/kubernetes# mkdir /var/lib/kubelet\n",
    "root@Slave:/etc/kubernetes# vim /etc/systemd/system/kubelet.service\n",
    "root@Slave:/etc/kubernetes# cat /etc/systemd/system/kubelet.service \n",
    "[Unit]\n",
    "Description=Kubernetes Kubelet\n",
    "Documentation=https://github.com/GoogleCloudPlatform/kubernetes\n",
    "After=docker.service\n",
    "Requires=docker.service\n",
    "\n",
    "[Service]\n",
    "WorkingDirectory=/var/lib/kubelet\n",
    "ExecStart=/usr/local/bin/kubelet \\\n",
    "  --cgroup-driver=cgroupfs \\\n",
    "  --hostname-override=Slave1 \\ #这里改为主机名\n",
    "  --pod-infra-container-image=jicki/pause-amd64:3.0 \\ #最好提前pull该镜像到本地\n",
    "  --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \\\n",
    "  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\\n",
    "  --cert-dir=/etc/kubernetes/ssl \\\n",
    "  --cluster_dns=10.254.0.2 \\\n",
    "  --cluster_domain=cluster.local \\\n",
    "  --hairpin-mode promiscuous-bridge \\\n",
    "  --allow-privileged=true \\\n",
    "  --fail-swap-on=false \\\n",
    "  --serialize-image-pulls=false \\\n",
    "  --logtostderr=true \\\n",
    "  --max-pods=512 \\\n",
    "  --v=1\n",
    "\n",
    "[Install]\n",
    "WantedBy=multi-user.target\n",
    "root@Slave1:/etc/kubernetes# docker pull jicki/pause-amd64:3.0\n",
    "3.0: Pulling from jicki/pause-amd64\n",
    "a3ed95caeb02: Pull complete \n",
    "f11233434377: Pull complete \n",
    "Digest: sha256:3b3a29e3c90ae7762bdf587d19302e62485b6bef46e114b741f7d75dba023bd3\n",
    "Status: Downloaded newer image for jicki/pause-amd64:3.0\n",
    "root@Slave1:/etc/kubernetes/ssl# systemctl daemon-reload\n",
    "root@Slave1:/etc/kubernetes/ssl# systemctl enable kubelet\n",
    "root@Slave1:/etc/kubernetes/ssl# systemctl start kubelet\n",
    "root@Slave1:/etc/kubernetes/ssl# systemctl status kubelet\n",
    "● kubelet.service - Kubernetes Kubelet\n",
    "   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)\n",
    "   Active: active (running) since Tue 2018-10-23 20:56:04 CST; 3min 11s ago\n",
    "     Docs: https://github.com/GoogleCloudPlatform/kubernetes\n",
    " Main PID: 12262 (kubelet)\n",
    "   CGroup: /system.slice/kubelet.service\n",
    "           └─12262 /usr/local/bin/kubelet --cgroup-driver=cgroupfs --hostname-override=Slave1 --pod-infra-container-image=jicki/pause-amd64:3\n",
    "\n",
    "Oct 23 20:56:04 Slave1 kubelet[12262]: I1023 20:56:04.878631   12262 feature_gate.go:226] feature gates: &{{} map[]}\n",
    "Oct 23 20:56:04 Slave1 kubelet[12262]: I1023 20:56:04.878805   12262 controller.go:114] kubelet config controller: starting controller\n",
    "Oct 23 20:56:04 Slave1 kubelet[12262]: I1023 20:56:04.878817   12262 controller.go:118] kubelet config controller: validating combination of \n",
    "Oct 23 20:56:04 Slave1 kubelet[12262]: W1023 20:56:04.885009   12262 cni.go:171] Unable to update cni config: No networks found in /etc/cni/n\n",
    "Oct 23 20:56:04 Slave1 kubelet[12262]: I1023 20:56:04.889856   12262 server.go:182] Version: v1.9.6\n",
    "Oct 23 20:56:04 Slave1 kubelet[12262]: I1023 20:56:04.889931   12262 feature_gate.go:226] feature gates: &{{} map[]}\n",
    "Oct 23 20:56:04 Slave1 kubelet[12262]: I1023 20:56:04.890039   12262 plugins.go:101] No cloud provider specified.\n",
    "Oct 23 20:56:04 Slave1 kubelet[12262]: I1023 20:56:04.941488   12262 csr.go:105] csr for this node already exists, reusing\n",
    "Oct 23 20:56:04 Slave1 kubelet[12262]: I1023 20:56:04.948060   12262 csr.go:113] csr for this node is still valid\n",
    "Oct 23 20:59:06 Slave1 systemd[1]: Started Kubernetes Kubelet.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Master节点上的操做（注意：每个从节点添加后都需要在主节点上进行该操作）\n",
    "\n",
    "* ### 信任证书\n",
    "\n",
    "```shell\n",
    "root@Master:/etc/kubernetes# kubectl get csr\n",
    "NAME                                                   AGE       REQUESTOR           CONDITION\n",
    "node-csr-DBQb9r6wLENeG7oTjRMzMhsvf34FBkG92tMg9BhPhpA   17m       kubelet-bootstrap   Pending\n",
    "node-csr-eufLGPzV7r2HVCmqCAanrltdfrcf1c96ixfn0gKACL0   1h        kubelet-bootstrap   Approved,Issued\n",
    "root@Master:/etc/kubernetes# kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve\n",
    "certificatesigningrequest \"node-csr-DBQb9r6wLENeG7oTjRMzMhsvf34FBkG92tMg9BhPhpA\" approved\n",
    "root@Master:/etc/kubernetes# kubectl get csr\n",
    "NAME                                                   AGE       REQUESTOR           CONDITION\n",
    "node-csr-DBQb9r6wLENeG7oTjRMzMhsvf34FBkG92tMg9BhPhpA   20m       kubelet-bootstrap   Approved,Issued\n",
    "node-csr-eufLGPzV7r2HVCmqCAanrltdfrcf1c96ixfn0gKACL0   1h        kubelet-bootstrap   Approved,Issued\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slave节点上的操作\n",
    "\n",
    "* ### 配置kube-proxy的service文件并启动\n",
    "\n",
    "```shell\n",
    "root@Slave1:/etc/kubernetes# mkdir -p /var/lib/kube-proxy\n",
    "root@Slave1:/etc/kubernetes# vim /etc/systemd/system/kube-proxy.service\n",
    "root@Slave1:/etc/kubernetes# cat /etc/systemd/system/kube-proxy.service\n",
    "[Unit]\n",
    "Description=Kubernetes Kube-Proxy Server\n",
    "Documentation=https://github.com/GoogleCloudPlatform/kubernetes\n",
    "After=network.target\n",
    "\n",
    "[Service]\n",
    "WorkingDirectory=/var/lib/kube-proxy\n",
    "ExecStart=/usr/local/bin/kube-proxy \\\n",
    "  --bind-address=192.168.10.11 \\ #需要改成自己的IP\n",
    "  --hostname-override=Slave1 \\ #改成自己的主机名\n",
    "  --cluster-cidr=10.254.64.0/18 \\\n",
    "  --masquerade-all \\\n",
    "  --feature-gates=SupportIPVSProxyMode=true \\\n",
    "  --proxy-mode=ipvs \\\n",
    "  --ipvs-min-sync-period=5s \\\n",
    "  --ipvs-sync-period=5s \\\n",
    "  --ipvs-scheduler=rr \\\n",
    "  --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \\\n",
    "  --logtostderr=true \\\n",
    "  --v=2\n",
    "Restart=on-failure\n",
    "RestartSec=5\n",
    "LimitNOFILE=65536\n",
    "\n",
    "[Install]\n",
    "WantedBy=multi-user.target\n",
    "root@Slave1:/etc/kubernetes# systemctl daemon-reload\n",
    "root@Slave1:/etc/kubernetes# systemctl start kube-proxy\n",
    "root@Slave1:/etc/kubernetes# systemctl status kube-proxy\n",
    "● kube-proxy.service - Kubernetes Kube-Proxy Server\n",
    "   Loaded: loaded (/etc/systemd/system/kube-proxy.service; disabled; vendor preset: enabled)\n",
    "   Active: active (running) since Tue 2018-10-23 21:08:06 CST; 7s ago\n",
    "     Docs: https://github.com/GoogleCloudPlatform/kubernetes\n",
    " Main PID: 13300 (kube-proxy)\n",
    "    Tasks: 0\n",
    "   Memory: 13.9M\n",
    "      CPU: 254ms\n",
    "   CGroup: /system.slice/kube-proxy.service\n",
    "           ‣ 13300 /usr/local/bin/kube-proxy --bind-address=192.168.10.11 --hostname-override=k8s-node-1 --cluster-cidr=10.254.64.0/18 --masq\n",
    "\n",
    "Oct 23 21:08:06 Slave1 kube-proxy[13300]: I1023 21:08:06.345881   13300 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_e\n",
    "Oct 23 21:08:06 Slave1 kube-proxy[13300]: I1023 21:08:06.345995   13300 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_c\n",
    "Oct 23 21:08:06 Slave1 kube-proxy[13300]: I1023 21:08:06.346180   13300 config.go:102] Starting endpoints config controller\n",
    "Oct 23 21:08:06 Slave1 kube-proxy[13300]: I1023 21:08:06.346219   13300 controller_utils.go:1019] Waiting for caches to sync for endpoints co\n",
    "Oct 23 21:08:06 Slave1 kube-proxy[13300]: I1023 21:08:06.346237   13300 config.go:202] Starting service config controller\n",
    "Oct 23 21:08:06 Slave1 kube-proxy[13300]: I1023 21:08:06.346371   13300 controller_utils.go:1019] Waiting for caches to sync for service conf\n",
    "Oct 23 21:08:06 Slave1 kube-proxy[13300]: I1023 21:08:06.446382   13300 controller_utils.go:1026] Caches are synced for endpoints config cont\n",
    "Oct 23 21:08:06 Slave1 kube-proxy[13300]: I1023 21:08:06.446490   13300 proxier.go:984] Not syncing iptables until Services and Endpoints hav\n",
    "Oct 23 21:08:06 Slave1 kube-proxy[13300]: I1023 21:08:06.446596   13300 controller_utils.go:1026] Caches are synced for service config contro\n",
    "Oct 23 21:08:06 Slave1 kube-proxy[13300]: I1023 21:08:06.446689   13300 proxier.go:329] Adding new service port \"default/kubernetes:https\" at\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 安装Flannel网络插件\n",
    "\n",
    "## Master节点上的操作\n",
    "\n",
    "* ### 下载并安装Flannel\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh/download# wget https://github.com/coreos/flannel/releases/download/v0.9.0/flannel-v0.9.0-linux-amd64.tar.gz\n",
    "root@Master:/home/zdyfjh/download# ls\n",
    "etcd-v3.3.10-linux-amd64         flannel-v0.9.0-linux-amd64.tar.gz  kubernetes-server-linux-amd64.tar.gz\n",
    "etcd-v3.3.10-linux-amd64.tar.gz  kubernetes\n",
    "root@Master:/home/zdyfjh/download# tar -zxvf flannel-v0.9.0-linux-amd64.tar.gz \n",
    "flanneld\n",
    "mk-docker-opts.sh\n",
    "README.md\n",
    "root@Master:/home/zdyfjh/download# ls\n",
    "etcd-v3.3.10-linux-amd64         flanneld                           kubernetes                            mk-docker-opts.sh\n",
    "etcd-v3.3.10-linux-amd64.tar.gz  flannel-v0.9.0-linux-amd64.tar.gz  kubernetes-server-linux-amd64.tar.gz  README.md\n",
    "root@Master:/home/zdyfjh/download# mv flanneld /usr/local/bin/\n",
    "root@Master:/home/zdyfjh/download# mv mk-docker-opts.sh /usr/local/bin/\n",
    "root@Master:/home/zdyfjh/download# scp flannel-v0.9.0-linux-amd64.tar.gz zdyfjh@192.168.10.11:/home/zdyfjh/download   \n",
    "root@Master:/home/zdyfjh/download# scp flannel-v0.9.0-linux-amd64.tar.gz zdyfjh@192.168.10.12:/home/zdyfjh/download   \n",
    "root@Master:/home/zdyfjh/download# scp flannel-v0.9.0-linux-amd64.tar.gz zdyfjh@192.168.10.13:/home/zdyfjh/download   \n",
    "root@Master:/home/zdyfjh/download# scp flannel-v0.9.0-linux-amd64.tar.gz zdyfjh@192.168.10.14:/home/zdyfjh/download   \n",
    "root@Master:/home/zdyfjh/download# scp flannel-v0.9.0-linux-amd64.tar.gz zdyfjh@192.168.10.15:/home/zdyfjh/download \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 在Etcd中配置flannel信息\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh/download# etcdctl --endpoints=https://192.168.10.10:2379,https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379,https://192.168.10.15:2379 \\\n",
    ">         --cert-file=/etc/kubernetes/ssl/etcd.pem \\\n",
    ">         --ca-file=/etc/kubernetes/ssl/ca.pem \\\n",
    ">         --key-file=/etc/kubernetes/ssl/etcd-key.pem \\\n",
    ">         set /flannel/network/config \\ '{\"Network\":\"10.254.64.0/18\",\"SubnetLen\":24,\"Backend\":{\"Type\":\"vxlan\"}}'\n",
    " {\"Network\":\"10.254.64.0/18\",\"SubnetLen\":24,\"Backend\":{\"Type\":\"vxlan\"}}\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 编写flannel的service文件并启动\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh/download# vim /etc/systemd/system/flanneld.service\n",
    "root@Master:/home/zdyfjh/download# cat /etc/systemd/system/flanneld.service \n",
    "[Unit]\n",
    "Description=Flanneld\n",
    "Documentation=https://github.com/coreos/flannel\n",
    "After=network.target\n",
    "After=network-online.target\n",
    "Wants=network-online.target\n",
    "After=etcd.service\n",
    "Before=docker.service\n",
    "[Service]\n",
    "User=root\n",
    "ExecStart=/usr/local/bin/flanneld \\\n",
    "        -etcd-endpoints=\"https://192.168.10.10:2379,https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379,https://192.168.10.15:2379\" \\\n",
    "        -ip-masq=true \\\n",
    "        -etcd-cafile=/etc/kubernetes/ssl/ca.pem \\\n",
    "        -etcd-certfile=/etc/kubernetes/ssl/etcd.pem \\\n",
    "        -etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem \\\n",
    "\t    -iface=eno4 \\ #这里需要改成自己的网卡名称或填入本机的IP地址\n",
    "        -etcd-prefix=/flannel/network\n",
    "ExecStartPost=/usr/local/bin/mk-docker-opts.sh -d /etc/default/docker -c #注意最后的/etc/default/docker需要将其中的内容拷贝到docker.service中\n",
    "Restart=on-failure\n",
    "Type=notify\n",
    "LimitNOFILE=65536\n",
    "\n",
    "[Install]\n",
    "WantedBy=multi-user.target\n",
    "root@Master:/etc/default# cat /etc/default/docker \n",
    "DOCKER_OPTS=\" --bip=10.254.69.1/24 --ip-masq=false --mtu=1450\"\n",
    "root@Master:/etc/default# vim /lib/systemd/system/docker.service \n",
    "root@Master:/etc/default# cat /lib/systemd/system/docker.service \n",
    "...\n",
    "ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --bip=10.254.69.1/24 --ip-masq=false --mtu=1450 #在末尾加上/etc/default/docker中的启动内容\n",
    "...\n",
    "root@Master:/home/zdyfjh/download# systemctl daemon-reload\n",
    "root@Master:/home/zdyfjh/download# systemctl enable flanneld\n",
    "root@Master:/home/zdyfjh/download# systemctl start flanneld\n",
    "root@Master:/home/zdyfjh/download# systemctl restart docker\n",
    "root@Master:/home/zdyfjh/download# systemctl restart kubelet\n",
    "root@Master:/home/zdyfjh/download# systemctl restart kube-proxy\n",
    "root@Master:/home/zdyfjh# iptables -P INPUT ACCEPT\n",
    "root@Master:/home/zdyfjh# iptables -P FORWARD ACCEPT\n",
    "root@Master:/home/zdyfjh# iptables -F\n",
    "root@Master:/home/zdyfjh# iptables -L -n\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slave节点上的操作\n",
    "\n",
    "* ### 安装Flannel\n",
    "\n",
    "```shell\n",
    "root@Slave:/home/zdyfjh# cd ./download/\n",
    "root@Slave:/home/zdyfjh/download# ls\n",
    "etcd-v3.3.10-linux-amd64         flannel-v0.9.0-linux-amd64.tar.gz  kubernetes-server-linux-amd64.tar.gz\n",
    "etcd-v3.3.10-linux-amd64.tar.gz  kubernetes\n",
    "root@Slave:/home/zdyfjh/download# tar -zxvf flannel-v0.9.0-linux-amd64.tar.gz \n",
    "flanneld\n",
    "mk-docker-opts.sh\n",
    "README.md\n",
    "root@Slave:/home/zdyfjh/download# mv flanneld /usr/local/bin/\n",
    "root@Slave:/home/zdyfjh/download# mv mk-docker-opts.sh /usr/local/bin/\n",
    "root@Slave:/home/zdyfjh/download# vim /etc/systemd/system/flanneld.service\n",
    "root@Slave:/home/zdyfjh/download# cat /etc/systemd/system/flanneld.service \n",
    "[Unit]\n",
    "Description=Flanneld\n",
    "Documentation=https://github.com/coreos/flannel\n",
    "After=network.target\n",
    "After=network-online.target\n",
    "Wants=network-online.target\n",
    "After=etcd.service\n",
    "Before=docker.service\n",
    "[Service]\n",
    "User=root\n",
    "ExecStart=/usr/local/bin/flanneld \\\n",
    "        -etcd-endpoints=\"https://192.168.10.10:2379,https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379,https://192.168.10.14:2379,https://192.168.10.15:2379\" \\\n",
    "        -ip-masq=true \\\n",
    "        -etcd-cafile=/etc/kubernetes/ssl/ca.pem \\\n",
    "        -etcd-certfile=/etc/kubernetes/ssl/etcd.pem \\\n",
    "        -etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem \\\n",
    "    -iface=eno4 \\\n",
    "        -etcd-prefix=/flannel/network\n",
    "ExecStartPost=/usr/local/bin/mk-docker-opts.sh -d /etc/default/docker -c #注意最后的/etc/default/docker需要将其中的内容拷贝到docker.service中\n",
    "Restart=on-failure\n",
    "Type=notify\n",
    "LimitNOFILE=65536\n",
    "\n",
    "[Install]\n",
    "WantedBy=multi-user.target\n",
    "root@Slave:/etc/default# cat /etc/default/docker \n",
    "DOCKER_OPTS=\" --bip=10.254.69.1/24 --ip-masq=false --mtu=1450\"\n",
    "root@Slave:/etc/default# vim /lib/systemd/system/docker.service \n",
    "root@Slave:/etc/default# cat /lib/systemd/system/docker.service \n",
    "...\n",
    "ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --bip=10.254.69.1/24 --ip-masq=false --mtu=1450 #在末尾加上/etc/default/docker中的启动内容\n",
    "...\n",
    "root@Slave:/home/zdyfjh/download# systemctl daemon-reload\n",
    "root@Slave:/home/zdyfjh/download# systemctl enable flanneld\n",
    "Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /etc/systemd/system/flanneld.service.\n",
    "root@Slave:/home/zdyfjh/download# systemctl start flanneld\n",
    "root@Slave:/home/zdyfjh/download# systemctl restart docker\n",
    "root@Slave:/home/zdyfjh/download# systemctl restart kubelet\n",
    "root@Slave:/home/zdyfjh/download# systemctl restart kube-proxy\n",
    "root@Slave:/home/zdyfjh/download# systemctl enable kubelet\n",
    "root@Slave:/home/zdyfjh/download# systemctl enable kube-proxy\n",
    "root@Slave3:/home/zdyfjh# iptables -P INPUT ACCEPT\n",
    "root@Slave3:/home/zdyfjh# iptables -P FORWARD ACCEPT\n",
    "root@Slave3:/home/zdyfjh# iptables -F\n",
    "root@Slave3:/home/zdyfjh# iptables -L -n\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 验证flannel\n",
    "\n",
    "要验证flannel的跨主机网络通信能力，最开始需要验证flannel是否添加了网络设备并更改了docker0网桥的网段。\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh/download# ifconfig \n",
    "\n",
    "...\n",
    "\n",
    "docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99  \n",
    "          inet addr:10.254.99.1  Bcast:0.0.0.0  Mask:255.255.255.0\n",
    "          UP BROADCAST MULTICAST  MTU:1500  Metric:1\n",
    "          RX packets:0 errors:0 dropped:0 overruns:0 frame:0\n",
    "          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0\n",
    "          collisions:0 txqueuelen:1000 \n",
    "          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)\n",
    "\n",
    "flannel.1 Link encap:Ethernet  HWaddr 46:a2:b1:8c:08:94  \n",
    "          inet addr:10.254.99.0  Bcast:0.0.0.0  Mask:255.255.255.255\n",
    "          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1\n",
    "          RX packets:3 errors:0 dropped:0 overruns:0 frame:0\n",
    "          TX packets:3 errors:0 dropped:9 overruns:0 carrier:0\n",
    "          collisions:0 txqueuelen:0 \n",
    "          RX bytes:252 (252.0 B)  TX bytes:252 (252.0 B)\n",
    "\n",
    "...\n",
    "\n",
    "```\n",
    "\n",
    "可以看到：docker0网桥的网段已经更改，而且已经添加了flannel.1网络设备，说明配置安装工作已经完成，接下来验证flannel的跨主机通信能力，该验证通过建立多个跨主机容器相互ping实验来验证。\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh/download# vim test.yaml \n",
    "root@Master:/home/zdyfjh/download# cat test.yaml \n",
    "apiVersion: extensions/v1beta1 \n",
    "kind: Deployment \n",
    "metadata: \n",
    "  name: tomcat-dm\n",
    "spec: \n",
    "  replicas: 6\n",
    "  template: \n",
    "    metadata: \n",
    "      labels: \n",
    "        name: tomcat\n",
    "    spec: \n",
    "      containers: \n",
    "        - name: tomcat\n",
    "          image: tomcat \n",
    "          imagePullPolicy: IfNotPresent\n",
    "          ports: \n",
    "            - containerPort: 80\n",
    "root@Master:/home/zdyfjh/download# kubectl create -f test.yaml\n",
    "root@Master:/home/zdyfjh/download# kubectl get po -o wide\n",
    "NAME                         READY     STATUS    RESTARTS   AGE       IP             NODE\n",
    "tomcat-dm-6bbbfd6b66-2p2dz   1/1       Running   0          8m        10.254.106.2   slave1\n",
    "tomcat-dm-6bbbfd6b66-bx5x8   1/1       Running   0          8m        10.254.70.2    slave3\n",
    "tomcat-dm-6bbbfd6b66-c7lsx   1/1       Running   0          8m        10.254.88.2    slave4\n",
    "tomcat-dm-6bbbfd6b66-g8jqd   1/1       Running   0          8m        10.254.125.2   slave2\n",
    "tomcat-dm-6bbbfd6b66-kqwgl   1/1       Running   0          8m        10.254.111.2   slave5\n",
    "tomcat-dm-6bbbfd6b66-tpl4t   1/1       Running   0          8m        10.254.70.3    slave3\n",
    "root@Master:/home/zdyfjh/download# ping 10.254.106.2\n",
    "PING 10.254.106.2 (10.254.106.2) 56(84) bytes of data.\n",
    "64 bytes from 10.254.106.2: icmp_seq=1 ttl=63 time=0.343 ms\n",
    "64 bytes from 10.254.106.2: icmp_seq=2 ttl=63 time=0.254 ms\n",
    "64 bytes from 10.254.106.2: icmp_seq=3 ttl=63 time=0.253 ms\n",
    "^C\n",
    "--- 10.254.106.2 ping statistics ---\n",
    "3 packets transmitted, 3 received, 0% packet loss, time 2028ms\n",
    "rtt min/avg/max/mdev = 0.253/0.283/0.343/0.044 ms\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 安装CoreDNS解析插件\n",
    "\n",
    "## Master节点上的操作\n",
    "\n",
    "* ### 下载coredns镜像，配置dns并启动\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh/download# docker pull coredns/coredns:1.0.4\n",
    "1.0.4: Pulling from coredns/coredns\n",
    "3690ec4760f9: Pull complete \n",
    "0e610ed088f8: Pull complete \n",
    "dabd0d361ae6: Pull complete \n",
    "Digest: sha256:d291f8b87eab26845a0c4605df4194924806712c4f624b9a9ddfc9d382b3ddbd\n",
    "Status: Downloaded newer image for coredns/coredns:1.0.4\n",
    "root@Master:/home/zdyfjh/download# vim coredns.yaml \n",
    "root@Master:/home/zdyfjh/download# cat coredns.yaml \n",
    "apiVersion: v1\n",
    "kind: ServiceAccount\n",
    "metadata:\n",
    "  name: coredns\n",
    "  namespace: kube-system\n",
    "---\n",
    "apiVersion: rbac.authorization.k8s.io/v1beta1\n",
    "kind: ClusterRole\n",
    "metadata:\n",
    "  labels:\n",
    "    kubernetes.io/bootstrapping: rbac-defaults\n",
    "  name: system:coredns\n",
    "rules:\n",
    "- apiGroups:\n",
    "  - \"\"\n",
    "  resources:\n",
    "  - endpoints\n",
    "  - services\n",
    "  - pods\n",
    "  - namespaces\n",
    "  verbs:\n",
    "  - list\n",
    "  - watch\n",
    "- apiGroups:\n",
    "  - \"\"\n",
    "  resources:\n",
    "  - nodes\n",
    "  verbs:\n",
    "  - get\n",
    "---\n",
    "apiVersion: rbac.authorization.k8s.io/v1beta1\n",
    "kind: ClusterRoleBinding\n",
    "metadata:\n",
    "  annotations:\n",
    "    rbac.authorization.kubernetes.io/autoupdate: \"true\"\n",
    "  labels:\n",
    "    kubernetes.io/bootstrapping: rbac-defaults\n",
    "  name: system:coredns\n",
    "roleRef:\n",
    "  apiGroup: rbac.authorization.k8s.io\n",
    "  kind: ClusterRole\n",
    "  name: system:coredns\n",
    "subjects:\n",
    "- kind: ServiceAccount\n",
    "  name: coredns\n",
    "  namespace: kube-system\n",
    "---\n",
    "apiVersion: v1\n",
    "kind: ConfigMap\n",
    "metadata:\n",
    "  name: coredns\n",
    "  namespace: kube-system\n",
    "data:\n",
    "  Corefile: |\n",
    "    .:53 {\n",
    "        errors\n",
    "        health\n",
    "        kubernetes cluster.local 10.254.0.0/18 {\n",
    "          pods insecure\n",
    "          upstream /etc/resolv.conf #这里要改成机器上dns解析的conf文件地址\n",
    "          fallthrough in-addr.arpa ip6.arpa\n",
    "        }\n",
    "        prometheus :9153\n",
    "        proxy . /etc/resolv.conf\n",
    "        cache 30\n",
    "        loadbalance\n",
    "    }\n",
    "---\n",
    "apiVersion: extensions/v1beta1\n",
    "kind: Deployment\n",
    "metadata:\n",
    "  name: coredns\n",
    "  namespace: kube-system\n",
    "  labels:\n",
    "    k8s-app: kube-dns\n",
    "    kubernetes.io/name: \"CoreDNS\"\n",
    "spec:\n",
    "  replicas: 2\n",
    "  strategy:\n",
    "    type: RollingUpdate\n",
    "    rollingUpdate:\n",
    "      maxUnavailable: 1\n",
    "  selector:\n",
    "    matchLabels:\n",
    "      k8s-app: kube-dns\n",
    "  template:\n",
    "    metadata:\n",
    "      labels:\n",
    "        k8s-app: kube-dns\n",
    "    spec:\n",
    "      serviceAccountName: coredns\n",
    "      tolerations:\n",
    "        - key: node-role.kubernetes.io/master\n",
    "          effect: NoSchedule\n",
    "        - key: \"CriticalAddonsOnly\"\n",
    "          operator: \"Exists\"\n",
    "      containers:\n",
    "      - name: coredns\n",
    "        image: coredns/coredns:1.0.4\n",
    "        imagePullPolicy: IfNotPresent\n",
    "        resources:\n",
    "          limits:\n",
    "            memory: 170Mi\n",
    "          requests:\n",
    "            cpu: 100m\n",
    "            memory: 70Mi\n",
    "        args: [ \"-conf\", \"/etc/coredns/Corefile\" ]\n",
    "        volumeMounts:\n",
    "        - name: config-volume\n",
    "          mountPath: /etc/coredns\n",
    "          readOnly: true\n",
    "        ports:\n",
    "        - containerPort: 53\n",
    "          name: dns\n",
    "          protocol: UDP\n",
    "        - containerPort: 53\n",
    "          name: dns-tcp\n",
    "          protocol: TCP\n",
    "        - containerPort: 9153\n",
    "          name: metrics\n",
    "          protocol: TCP\n",
    "        securityContext:\n",
    "          allowPrivilegeEscalation: false\n",
    "          capabilities:\n",
    "            add:\n",
    "            - NET_BIND_SERVICE\n",
    "            drop:\n",
    "            - all\n",
    "          readOnlyRootFilesystem: true\n",
    "        livenessProbe:\n",
    "          httpGet:\n",
    "            path: /health\n",
    "            port: 8080\n",
    "            scheme: HTTP\n",
    "          initialDelaySeconds: 60\n",
    "          timeoutSeconds: 5\n",
    "          successThreshold: 1\n",
    "          failureThreshold: 5\n",
    "      dnsPolicy: Default\n",
    "      volumes:\n",
    "        - name: config-volume\n",
    "          configMap:\n",
    "            name: coredns\n",
    "            items:\n",
    "            - key: Corefile\n",
    "              path: Corefile\n",
    "---\n",
    "apiVersion: v1\n",
    "kind: Service\n",
    "metadata:\n",
    "  name: kube-dns\n",
    "  namespace: kube-system\n",
    "  annotations:\n",
    "    prometheus.io/port: \"9153\"\n",
    "    prometheus.io/scrape: \"true\"\n",
    "  labels:\n",
    "    k8s-app: kube-dns\n",
    "    kubernetes.io/cluster-service: \"true\"\n",
    "    kubernetes.io/name: \"CoreDNS\"\n",
    "spec:\n",
    "  selector:\n",
    "    k8s-app: kube-dns\n",
    "  clusterIP: 10.254.0.2\n",
    "  ports:\n",
    "  - name: dns\n",
    "    port: 53\n",
    "    protocol: UDP\n",
    "  - name: dns-tcp\n",
    "    port: 53\n",
    "    protocol: TCP\n",
    "root@Master:/home/zdyfjh/download# kubectl apply -f coredns.yaml\n",
    "serviceaccount \"coredns\" created\n",
    "clusterrole \"system:coredns\" created\n",
    "clusterrolebinding \"system:coredns\" created\n",
    "configmap \"coredns\" created\n",
    "deployment \"coredns\" created\n",
    "service \"kube-dns\" created\n",
    "root@Master:/home/zdyfjh/download# kubectl get po -o wide -n kube-system\n",
    "NAME                       READY     STATUS    RESTARTS   AGE       IP             NODE\n",
    "coredns-5955c6bfd4-6fs8l   1/1       Running   0          32s       10.254.125.2   slave2\n",
    "coredns-5955c6bfd4-g7nlr   1/1       Running   0          32s       10.254.88.2    slave4\n",
    "root@Master:/home/zdyfjh/download# kubectl get pod,svc -n kube-system -o wide\n",
    "NAME                          READY     STATUS    RESTARTS   AGE       IP             NODE\n",
    "po/coredns-5955c6bfd4-6fs8l   1/1       Running   0          50s       10.254.125.2   slave2\n",
    "po/coredns-5955c6bfd4-g7nlr   1/1       Running   0          50s       10.254.88.2    slave4\n",
    "\n",
    "NAME           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE       SELECTOR\n",
    "svc/kube-dns   ClusterIP   10.254.0.2   <none>        53/UDP,53/TCP   50s       k8s-app=kube-dns\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ### 测试coreDNS\n",
    "\n",
    "```shell\n",
    "root@Master:/home/zdyfjh/download# kubectl create -f test.yaml \n",
    "deployment \"tomcat-dm\" created\n",
    "root@Master:/home/zdyfjh/download# kubectl get po -o wide\n",
    "NAME                         READY     STATUS    RESTARTS   AGE       IP             NODE\n",
    "tomcat-dm-6bbbfd6b66-82gvr   1/1       Running   0          5s        10.254.88.3    slave4\n",
    "tomcat-dm-6bbbfd6b66-8jz54   1/1       Running   0          5s        10.254.111.3   slave5\n",
    "tomcat-dm-6bbbfd6b66-bbh4d   1/1       Running   0          5s        10.254.70.2    slave3\n",
    "tomcat-dm-6bbbfd6b66-f7bwn   1/1       Running   0          5s        10.254.111.2   slave5\n",
    "tomcat-dm-6bbbfd6b66-p7lz4   1/1       Running   0          5s        10.254.106.2   slave1\n",
    "tomcat-dm-6bbbfd6b66-xv67k   1/1       Running   0          5s        10.254.125.3   slave2\n",
    "root@Master:/home/zdyfjh/download# kubectl exec -it tomcat-dm-6bbbfd6b66-bbh4d /bin/bash\n",
    "root@tomcat-dm-6bbbfd6b66-bbh4d:/usr/local/tomcat# ping baidu.com\n",
    "PING baidu.com (123.125.115.110) 56(84) bytes of data.\n",
    "64 bytes from 123.125.115.110 (123.125.115.110): icmp_seq=1 ttl=49 time=33.7 ms\n",
    "64 bytes from 123.125.115.110 (123.125.115.110): icmp_seq=2 ttl=49 time=33.7 ms\n",
    "64 bytes from 123.125.115.110 (123.125.115.110): icmp_seq=3 ttl=49 time=33.6 ms\n",
    "^C\n",
    "--- baidu.com ping statistics ---\n",
    "3 packets transmitted, 3 received, 0% packet loss, time 2003ms\n",
    "rtt min/avg/max/mdev = 33.680/33.731/33.788/0.216 ms\n",
    "```\n",
    "\n",
    "如上面代码所示，`baidu.com`已经`ping`通，说明容器内部的域名解析服务是正常工作的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 后记\n",
    "\n",
    "算了，没后记了，希望以后看到这篇文档安装系统的小伙伴们能一次过。\n",
    "\n",
    "2018-10-24\n",
    "\n",
    "# Ref\n",
    "\n",
    "[基于二进制安装 kubernetes 1.9.6](https://blog.csdn.net/qq_21816375/article/details/80288937)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}