doc2vec-text_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1\n",
    "\n",
    "用labeled_train_data演示Doc2vec的训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '..')\n",
    "import os\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "with zipfile.ZipFile('./data/labeledTrainData.tsv.zip', 'r') as z:\n",
    "    z.extractall('./data/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 3)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5814_8</td>\n",
       "      <td>1</td>\n",
       "      <td>With all this stuff going down at the moment w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2381_9</td>\n",
       "      <td>1</td>\n",
       "      <td>\\The Classic War of the Worlds\\\" by Timothy Hi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7759_3</td>\n",
       "      <td>0</td>\n",
       "      <td>The film starts with a manager (Nicholas Bell)...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3630_4</td>\n",
       "      <td>0</td>\n",
       "      <td>It must be assumed that those who praised this...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9495_8</td>\n",
       "      <td>1</td>\n",
       "      <td>Superbly trashy and wondrously unpretentious 8...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id  sentiment                                             review\n",
       "0  5814_8          1  With all this stuff going down at the moment w...\n",
       "1  2381_9          1  \\The Classic War of the Worlds\\\" by Timothy Hi...\n",
       "2  7759_3          0  The film starts with a manager (Nicholas Bell)...\n",
       "3  3630_4          0  It must be assumed that those who praised this...\n",
       "4  9495_8          1  Superbly trashy and wondrously unpretentious 8..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labeled_train = pd.read_csv('./data/labeledTrainData.tsv', sep='\\t')\n",
    "print(labeled_train.shape)\n",
    "labeled_train.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"With all this stuff going down at the moment with MJ i've started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent. Moonwalker is part biography, part feature film which i remember going to see at the cinema when it was originally released. Some of it has subtle messages about MJ's feeling towards the press and also the obvious message of drugs are bad m'kay.<br /><br />Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring. Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him.<br /><br />The actual feature film bit when it finally starts is only on for 20 minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord. Why he wants MJ dead so bad is beyond me. Because MJ overheard his plans? Nah, Joe Pesci's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno, maybe he just hates MJ's music.<br /><br />Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence. Also, the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene.<br /><br />Bottom line, this movie is for people who like MJ on one level or another (which i think is most people). If not, then stay away. It does try and give off a wholesome message and ironically MJ's bestest buddy in this movie is a girl! Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty? Well, with all the attention i've gave this subject....hmmm well i don't know because people can be different behind closed doors, i know this for a fact. He is either an extremely nice but stupid guy or one of the most sickest liars. I hope he is not the latter.\""
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labeled_train['review'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import re\n",
    "\n",
    "def clean_str(string):\n",
    "    \"\"\"\n",
    "    Tokenization/string cleaning for all datasets \n",
    "    Every dataset is lower cased\n",
    "    \"\"\"\n",
    "    string = re.sub(r\"[^A-Za-z0-9(),!?\\'\\`]\", \" \", string)     \n",
    "    string = re.sub(r\"\\'s\", \" \\'s\", string) \n",
    "    string = re.sub(r\"\\'ve\", \" \\'ve\", string) \n",
    "    string = re.sub(r\"n\\'t\", \" n\\'t\", string) \n",
    "    string = re.sub(r\"\\'re\", \" \\'re\", string) \n",
    "    string = re.sub(r\"\\'d\", \" \\'d\", string) \n",
    "    string = re.sub(r\"\\'ll\", \" \\'ll\", string) \n",
    "    string = re.sub(r\",\", \" , \", string) \n",
    "    string = re.sub(r\"!\", \" ! \", string) \n",
    "    string = re.sub(r\"\\(\", \" \\( \", string) \n",
    "    string = re.sub(r\"\\)\", \" \\) \", string) \n",
    "    string = re.sub(r\"\\?\", \" \\? \", string) \n",
    "    string = re.sub(r\"\\s{2,}\", \" \", string)    \n",
    "    return string.strip().lower()\n",
    "\n",
    "labeled_texts = []\n",
    "labels = []\n",
    "for idx in range(labeled_train.review.shape[0]):\n",
    "    labeled_texts.append(clean_str(BeautifulSoup(labeled_train.review[idx], \"lxml\").get_text()))\n",
    "    labels.append(labeled_train.sentiment[idx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"with all this stuff going down at the moment with mj i 've started listening to his music , watching the odd documentary here and there , watched the wiz and watched moonwalker again maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent moonwalker is part biography , part feature film which i remember going to see at the cinema when it was originally released some of it has subtle messages about mj 's feeling towards the press and also the obvious message of drugs are bad m'kay visually impressive but of course this is all about michael jackson so unless you remotely like mj in anyway then you are going to hate this and find it boring some may call mj an egotist for consenting to the making of this movie but mj and most of his fans would say that he made it for the fans which if true is really nice of him the actual feature film bit when it finally starts is only on for 20 minutes or so excluding the smooth criminal sequence and joe pesci is convincing as a psychopathic all powerful drug lord why he wants mj dead so bad is beyond me because mj overheard his plans \\\\? nah , joe pesci 's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno , maybe he just hates mj 's music lots of cool things in this like mj turning into a car and a robot and the whole speed demon sequence also , the director must have had the patience of a saint when it came to filming the kiddy bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene bottom line , this movie is for people who like mj on one level or another \\\\( which i think is most people \\\\) if not , then stay away it does try and give off a wholesome message and ironically mj 's bestest buddy in this movie is a girl ! michael jackson is truly one of the most talented people ever to grace this planet but is he guilty \\\\? well , with all the attention i 've gave this subject hmmm well i do n't know because people can be different behind closed doors , i know this for a fact he is either an extremely nice but stupid guy or one of the most sickest liars i hope he is not the latter\""
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labeled_texts[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Doc2vec的训练             "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01 15:42:00.135134 0\n",
      "2018-10-01 15:42:00.177021 10000\n",
      "2018-10-01 15:42:00.217945 20000\n"
     ]
    }
   ],
   "source": [
    "import codecs\n",
    "from datetime import datetime\n",
    "# -------------------add row number to query----------------------\n",
    "doc_f = codecs.open('./data/' + 'labeled_train_id.txt','w',encoding='utf8')\n",
    "for i, reviews in enumerate(labeled_texts):\n",
    "    tags = [i]\n",
    "    if i % 10000 == 0:\n",
    "        print(datetime.now(),i)\n",
    "    doc_f.write('_*{} {}\\n'.format(i,reviews))\n",
    "doc_f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_f = codecs.open('./data/' + 'test.txt','w',encoding='utf8')\n",
    "for i, reviews in enumerate(labeled_texts[:5]):\n",
    "    tags = [i]\n",
    "    doc_f.write('_*{} {}\\n'.format(i,reviews))\n",
    "doc_f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "\n",
    "SentimentDocument = namedtuple('SentimentDocument', 'words tags')\n",
    "class Doc_list(object):\n",
    "    def __init__(self,f):\n",
    "        self.f = f\n",
    "    def __iter__(self):\n",
    "        for i,line in enumerate(codecs.open(self.f,encoding='utf8')):\n",
    "            words = line.split()\n",
    "            tags = [int(words[0][2:])]\n",
    "            words = words[1:]\n",
    "            yield SentimentDocument(words,tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\gensim\\utils.py:1197: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n",
      "  warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\gensim\\models\\doc2vec.py:366: UserWarning: The parameter `size` is deprecated, will be removed in 4.0.0, use `vector_size` instead.\n",
      "  warnings.warn(\"The parameter `size` is deprecated, will be removed in 4.0.0, use `vector_size` instead.\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01 14:02:33.488091 pass: 1\n",
      "dbow [0.5026 0.5096 0.5252 0.546  0.53  ] 0.52268\n",
      "2018-10-01 14:02:39.072662 pass: 2\n",
      "dbow [0.5188 0.5254 0.5256 0.532  0.5332] 0.5269999999999999\n",
      "2018-10-01 14:02:44.568629 pass: 3\n",
      "dbow [0.548  0.5422 0.5562 0.566  0.5614] 0.55476\n",
      "2018-10-01 14:02:50.270080 pass: 4\n",
      "dbow [0.602  0.6094 0.6334 0.6324 0.6294] 0.62132\n",
      "2018-10-01 14:02:56.033885 pass: 5\n",
      "dbow [0.6718 0.6656 0.6752 0.6702 0.667 ] 0.66996\n",
      "2018-10-01 14:03:02.088532 pass: 6\n",
      "dbow [0.7084 0.707  0.714  0.719  0.6892] 0.7075199999999999\n",
      "2018-10-01 14:03:08.419933 pass: 7\n",
      "dbow [0.7406 0.7508 0.7472 0.75   0.7376] 0.74524\n",
      "2018-10-01 14:03:15.594719 pass: 8\n",
      "dbow [0.7734 0.777  0.7772 0.7868 0.774 ] 0.7776799999999999\n",
      "2018-10-01 14:03:23.463741 pass: 9\n",
      "dbow [0.7892 0.7936 0.8004 0.8154 0.7916] 0.79804\n",
      "2018-10-01 14:03:32.212353 pass: 10\n",
      "dbow [0.8016 0.8144 0.811  0.8312 0.809 ] 0.8134399999999999\n",
      "2018-10-01 14:03:41.502588 pass: 11\n",
      "dbow [0.8182 0.8252 0.825  0.8384 0.8244] 0.82624\n",
      "2018-10-01 14:03:51.182546 pass: 12\n",
      "dbow [0.8262 0.8354 0.835  0.8468 0.8364] 0.83596\n",
      "2018-10-01 14:04:01.277308 pass: 13\n",
      "dbow [0.8336 0.8472 0.8438 0.8502 0.8412] 0.8432000000000001\n",
      "2018-10-01 14:04:11.454408 pass: 14\n",
      "dbow [0.8458 0.857  0.8524 0.8624 0.85  ] 0.85352\n",
      "2018-10-01 14:04:22.016519 pass: 15\n",
      "dbow [0.856  0.86   0.852  0.866  0.8568] 0.85816\n",
      "2018-10-01 14:04:32.149237 pass: 16\n",
      "dbow [0.8622 0.8684 0.856  0.866  0.8644] 0.8634000000000001\n",
      "2018-10-01 14:04:42.405027 pass: 17\n",
      "dbow [0.866  0.8652 0.8598 0.8704 0.8658] 0.8654399999999999\n",
      "2018-10-01 14:04:51.685947 pass: 18\n",
      "dbow [0.8682 0.866  0.8654 0.8732 0.8676] 0.86808\n",
      "2018-10-01 14:05:00.915759 pass: 19\n",
      "dbow [0.8696 0.8678 0.8664 0.8748 0.8672] 0.8691600000000002\n",
      "2018-10-01 14:05:09.687394 pass: 20\n",
      "dbow [0.8754 0.8688 0.8658 0.877  0.8716] 0.87172\n",
      "2018-10-01 14:05:18.167647 pass: 21\n",
      "dbow [0.8772 0.872  0.866  0.8802 0.8724] 0.87356\n",
      "2018-10-01 14:05:26.286384 pass: 22\n",
      "dbow [0.8738 0.8698 0.8682 0.8834 0.8748] 0.874\n",
      "2018-10-01 14:05:34.388460 pass: 23\n",
      "dbow [0.8772 0.8742 0.866  0.8848 0.879 ] 0.8762399999999999\n",
      "2018-10-01 14:05:42.138748 pass: 24\n",
      "dbow [0.878  0.873  0.869  0.8832 0.8764] 0.87592\n",
      "2018-10-01 14:05:49.733589 pass: 25\n",
      "dbow [0.8782 0.8732 0.8722 0.8858 0.8778] 0.87744\n",
      "2018-10-01 14:05:57.207850 pass: 26\n",
      "dbow [0.8784 0.8754 0.8722 0.8824 0.88  ] 0.87768\n",
      "2018-10-01 14:06:04.418730 pass: 27\n",
      "dbow [0.8784 0.8722 0.8718 0.8864 0.878 ] 0.87736\n",
      "2018-10-01 14:06:11.707598 pass: 28\n",
      "dbow [0.8784 0.875  0.8718 0.8872 0.8784] 0.8781599999999999\n",
      "2018-10-01 14:06:18.969801 pass: 29\n",
      "dbow [0.8784 0.874  0.8722 0.8856 0.8788] 0.8778\n",
      "2018-10-01 14:06:26.107246 pass: 30\n",
      "dbow [0.8812 0.8772 0.8768 0.8822 0.8756] 0.8786000000000002\n",
      "2018-10-01 14:06:33.128408 pass: 31\n",
      "dbow [0.8806 0.8788 0.8734 0.8824 0.8778] 0.8785999999999999\n",
      "2018-10-01 14:06:40.312393 pass: 32\n",
      "dbow [0.8784 0.8782 0.8752 0.8824 0.8784] 0.87852\n",
      "2018-10-01 14:06:47.468225 pass: 33\n",
      "dbow [0.88   0.8798 0.8728 0.8824 0.878 ] 0.8785999999999999\n",
      "2018-10-01 14:06:54.489654 pass: 34\n",
      "dbow [0.8828 0.8806 0.8722 0.8836 0.875 ] 0.87884\n",
      "2018-10-01 14:07:01.778649 pass: 35\n",
      "dbow [0.8834 0.8778 0.8728 0.882  0.8772] 0.8786400000000001\n",
      "2018-10-01 14:07:08.850623 pass: 36\n",
      "dbow [0.8848 0.8816 0.8734 0.8852 0.8804] 0.8810800000000001\n",
      "2018-10-01 14:07:15.927297 pass: 37\n",
      "dbow [0.8862 0.8806 0.8718 0.8838 0.878 ] 0.8800799999999999\n",
      "2018-10-01 14:07:22.885749 pass: 38\n",
      "dbow [0.8856 0.8794 0.8724 0.883  0.878 ] 0.8796799999999999\n",
      "2018-10-01 14:07:29.928271 pass: 39\n",
      "dbow [0.8862 0.8828 0.8732 0.8836 0.8774] 0.88064\n",
      "2018-10-01 14:07:36.891508 pass: 40\n",
      "dbow [0.8856 0.883  0.875  0.8858 0.8764] 0.88116\n",
      "2018-10-01 14:07:43.841659 pass: 41\n",
      "dbow [0.8848 0.8822 0.872  0.883  0.878 ] 0.8799999999999999\n",
      "2018-10-01 14:07:50.747739 pass: 42\n",
      "dbow [0.8834 0.8836 0.8738 0.8826 0.8806] 0.8808\n",
      "2018-10-01 14:07:57.513940 pass: 43\n",
      "dbow [0.8846 0.8854 0.8754 0.8824 0.8776] 0.8810800000000001\n",
      "2018-10-01 14:08:04.290009 pass: 44\n",
      "dbow [0.8838 0.8846 0.8744 0.8846 0.8756] 0.8806\n",
      "2018-10-01 14:08:11.064008 pass: 45\n",
      "dbow [0.8816 0.8838 0.8754 0.8854 0.8768] 0.8806\n",
      "2018-10-01 14:08:17.823334 pass: 46\n",
      "dbow [0.8808 0.8834 0.8762 0.8818 0.8794] 0.88032\n",
      "2018-10-01 14:08:24.559222 pass: 47\n",
      "dbow [0.8838 0.8816 0.8768 0.8858 0.8788] 0.8813600000000001\n",
      "2018-10-01 14:08:31.289100 pass: 48\n",
      "dbow [0.8836 0.8856 0.8772 0.884  0.878 ] 0.8816799999999999\n",
      "2018-10-01 14:08:38.035051 pass: 49\n",
      "dbow [0.884  0.886  0.8772 0.8856 0.8786] 0.8822800000000001\n",
      "2018-10-01 14:08:44.683341 pass: 50\n",
      "dbow [0.8828 0.8852 0.8758 0.886  0.8798] 0.88192\n",
      "2018-10-01 14:08:52.278105 save done\n"
     ]
    }
   ],
   "source": [
    "from gensim.models.doc2vec import Doc2Vec\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "d2v = Doc2Vec(dm=0, size=300, negative=5, hs=0, min_count=3, window=30,sample=1e-5,\n",
    "              workers=6,alpha=0.025,min_alpha=0.025,epochs=1)\n",
    "doc_list = Doc_list('./data/' + 'labeled_train_id.txt')\n",
    "d2v.build_vocab(doc_list)\n",
    "\n",
    "# -------------------train dbow doc2vec---------------------------------------------\n",
    "for i in range(50):\n",
    "    print(datetime.now(),'pass:',i + 1)\n",
    "    doc_list = Doc_list('./data/' + 'labeled_train_id.txt')\n",
    "    d2v.train(doc_list,total_examples=d2v.corpus_count, epochs=d2v.epochs)\n",
    "    X_d2v = np.array([d2v.docvecs[i] for i in range(25000)])\n",
    "    scores = cross_val_score(LogisticRegression(C=3),X_d2v,labels,cv=5,n_jobs=4)\n",
    "    print('dbow',scores,np.mean(scores))\n",
    "d2v.save('./data/' + 'dbow_d2v.model')\n",
    "print(datetime.now(),'save done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\gensim\\models\\doc2vec.py:366: UserWarning: The parameter `size` is deprecated, will be removed in 4.0.0, use `vector_size` instead.\n",
      "  warnings.warn(\"The parameter `size` is deprecated, will be removed in 4.0.0, use `vector_size` instead.\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01 14:14:06.046697 pass: 1\n",
      "dbow [0.5036 0.5104 0.5452 0.5664 0.5684] 0.5388000000000001\n",
      "2018-10-01 14:14:12.480981 pass: 2\n",
      "dbow [0.5512 0.6006 0.6178 0.6248 0.6232] 0.60352\n",
      "2018-10-01 14:14:18.932892 pass: 3\n",
      "dbow [0.6474 0.6664 0.6898 0.696  0.6874] 0.6774000000000001\n",
      "2018-10-01 14:14:26.339173 pass: 4\n",
      "dbow [0.7266 0.7394 0.7562 0.7614 0.7478] 0.7462799999999999\n",
      "2018-10-01 14:14:34.815545 pass: 5\n",
      "dbow [0.7714 0.786  0.7802 0.7866 0.7716] 0.77916\n",
      "2018-10-01 14:14:43.397165 pass: 6\n",
      "dbow [0.7938 0.809  0.809  0.8114 0.8046] 0.80556\n",
      "2018-10-01 14:14:53.505202 pass: 7\n",
      "dbow [0.8226 0.817  0.8256 0.8238 0.8282] 0.82344\n",
      "2018-10-01 14:15:04.129360 pass: 8\n",
      "dbow [0.834  0.8326 0.8264 0.8298 0.8344] 0.83144\n",
      "2018-10-01 14:15:15.649132 pass: 9\n",
      "dbow [0.8454 0.8374 0.834  0.8416 0.839 ] 0.83948\n",
      "2018-10-01 14:15:27.403376 pass: 10\n",
      "dbow [0.8502 0.843  0.8398 0.842  0.8406] 0.8431200000000001\n",
      "2018-10-01 14:15:40.155957 pass: 11\n",
      "dbow [0.8534 0.851  0.8412 0.8496 0.8456] 0.84816\n",
      "2018-10-01 14:15:53.372527 pass: 12\n",
      "dbow [0.8564 0.8458 0.844  0.852  0.8518] 0.85\n",
      "2018-10-01 14:16:07.015432 pass: 13\n",
      "dbow [0.8582 0.8466 0.8444 0.8556 0.8524] 0.85144\n",
      "2018-10-01 14:16:21.308747 pass: 14\n",
      "dbow [0.8584 0.8504 0.8492 0.8592 0.8508] 0.8535999999999999\n",
      "2018-10-01 14:16:35.308897 pass: 15\n",
      "dbow [0.8588 0.8526 0.8458 0.859  0.8496] 0.8531599999999999\n",
      "2018-10-01 14:16:48.705623 pass: 16\n",
      "dbow [0.8616 0.8542 0.8482 0.8644 0.8492] 0.85552\n",
      "2018-10-01 14:17:01.998081 pass: 17\n",
      "dbow [0.8628 0.856  0.8466 0.8656 0.8538] 0.8569599999999999\n",
      "2018-10-01 14:17:14.577311 pass: 18\n",
      "dbow [0.8622 0.8526 0.8488 0.8652 0.8534] 0.8564399999999999\n",
      "2018-10-01 14:17:26.494210 pass: 19\n",
      "dbow [0.86   0.8522 0.8464 0.8614 0.8558] 0.85516\n",
      "2018-10-01 14:17:37.332190 pass: 20\n",
      "dbow [0.8626 0.8524 0.8492 0.8598 0.857 ] 0.8562\n",
      "2018-10-01 14:17:47.859206 pass: 21\n",
      "dbow [0.863  0.8534 0.8476 0.8598 0.8544] 0.85564\n",
      "2018-10-01 14:17:57.271722 pass: 22\n",
      "dbow [0.8618 0.8526 0.848  0.8602 0.8558] 0.8556799999999999\n",
      "2018-10-01 14:18:06.096350 pass: 23\n",
      "dbow [0.8656 0.8544 0.8516 0.8616 0.8548] 0.8576\n",
      "2018-10-01 14:18:14.496707 pass: 24\n",
      "dbow [0.8666 0.856  0.8506 0.861  0.8536] 0.85756\n",
      "2018-10-01 14:18:22.822376 pass: 25\n",
      "dbow [0.8674 0.8584 0.8538 0.8626 0.8538] 0.8592000000000001\n",
      "2018-10-01 14:18:30.832746 pass: 26\n",
      "dbow [0.8692 0.8598 0.8534 0.8606 0.8526] 0.8591200000000001\n",
      "2018-10-01 14:18:38.874076 pass: 27\n",
      "dbow [0.8696 0.8576 0.8528 0.863  0.8536] 0.85932\n",
      "2018-10-01 14:18:46.957361 pass: 28\n",
      "dbow [0.8704 0.8592 0.8538 0.8612 0.8544] 0.8598000000000001\n",
      "2018-10-01 14:18:54.803750 pass: 29\n",
      "dbow [0.87   0.8578 0.8568 0.861  0.8542] 0.8599599999999998\n",
      "2018-10-01 14:19:02.561957 pass: 30\n",
      "dbow [0.8692 0.8568 0.8558 0.8646 0.8576] 0.8607999999999999\n",
      "2018-10-01 14:19:10.207308 pass: 31\n",
      "dbow [0.8682 0.8586 0.857  0.8664 0.8568] 0.8614\n",
      "2018-10-01 14:19:17.608726 pass: 32\n",
      "dbow [0.8704 0.8596 0.8578 0.8656 0.8578] 0.8622400000000001\n",
      "2018-10-01 14:19:25.074340 pass: 33\n",
      "dbow [0.8668 0.8604 0.8574 0.8658 0.8584] 0.86176\n",
      "2018-10-01 14:19:32.549179 pass: 34\n",
      "dbow [0.8658 0.8602 0.8572 0.8678 0.8588] 0.8619599999999998\n",
      "2018-10-01 14:19:39.943151 pass: 35\n",
      "dbow [0.8682 0.8628 0.8566 0.8664 0.8554] 0.86188\n",
      "2018-10-01 14:19:47.150062 pass: 36\n",
      "dbow [0.8654 0.8612 0.8578 0.8686 0.8564] 0.86188\n",
      "2018-10-01 14:19:54.377786 pass: 37\n",
      "dbow [0.8686 0.8626 0.857  0.8688 0.8558] 0.86256\n",
      "2018-10-01 14:20:01.579110 pass: 38\n",
      "dbow [0.8662 0.8634 0.86   0.867  0.8588] 0.8630800000000001\n",
      "2018-10-01 14:20:08.770369 pass: 39\n",
      "dbow [0.8654 0.8646 0.86   0.8662 0.8614] 0.86352\n",
      "2018-10-01 14:20:16.133369 pass: 40\n",
      "dbow [0.8668 0.8646 0.8604 0.8672 0.861 ] 0.8640000000000001\n",
      "2018-10-01 14:20:23.309626 pass: 41\n",
      "dbow [0.8678 0.8632 0.8596 0.8664 0.8642] 0.86424\n",
      "2018-10-01 14:20:30.743312 pass: 42\n",
      "dbow [0.868  0.861  0.86   0.8682 0.8652] 0.86448\n",
      "2018-10-01 14:20:38.171830 pass: 43\n",
      "dbow [0.8672 0.8636 0.86   0.8666 0.8644] 0.8643599999999999\n",
      "2018-10-01 14:20:45.397878 pass: 44\n",
      "dbow [0.8668 0.8648 0.8592 0.8672 0.864 ] 0.8644000000000001\n",
      "2018-10-01 14:20:52.508220 pass: 45\n",
      "dbow [0.8666 0.8644 0.864  0.8678 0.8654] 0.86564\n",
      "2018-10-01 14:20:59.522539 pass: 46\n",
      "dbow [0.865  0.8652 0.8644 0.8672 0.8646] 0.8652799999999999\n",
      "2018-10-01 14:21:06.757564 pass: 47\n",
      "dbow [0.8656 0.863  0.8658 0.8682 0.8652] 0.86556\n",
      "2018-10-01 14:21:13.765224 pass: 48\n",
      "dbow [0.8672 0.8648 0.8624 0.866  0.8642] 0.86492\n",
      "2018-10-01 14:21:21.253975 pass: 49\n",
      "dbow [0.8686 0.8632 0.8636 0.8662 0.8628] 0.8648800000000001\n",
      "2018-10-01 14:21:29.059441 pass: 50\n",
      "dbow [0.8682 0.8648 0.8656 0.867  0.8662] 0.86636\n",
      "2018-10-01 14:21:37.799807 save done\n"
     ]
    }
   ],
   "source": [
    "d2v = Doc2Vec(dm=1, size=300, negative=5, hs=0, min_count=3, window=10,sample=1e-5,\n",
    "              workers=6,alpha=0.05,min_alpha=0.025,epochs=1)\n",
    "doc_list = Doc_list('./data/' + 'labeled_train_id.txt')\n",
    "d2v.build_vocab(doc_list)\n",
    "\n",
    "# -------------------train dm doc2vec---------------------------------------------\n",
    "for i in range(50):\n",
    "    print(datetime.now(),'pass:',i + 1)\n",
    "    doc_list = Doc_list('./data/' + 'labeled_train_id.txt')\n",
    "    d2v.train(doc_list,total_examples=d2v.corpus_count, epochs=d2v.epochs)\n",
    "    X_d2v = np.array([d2v.docvecs[i] for i in range(25000)])\n",
    "    scores = cross_val_score(LogisticRegression(C=3),X_d2v,labels,cv=5,n_jobs=4)\n",
    "    print('dbow',scores,np.mean(scores))\n",
    "d2v.save('./data/' + 'dm_d2v.model')\n",
    "print(datetime.now(),'save done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2\n",
    "\n",
    "用labeld_train_data和unlabeled_train_data进行Doc2vec的训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_csv( './data/labeledTrainData.tsv', header=0, delimiter='\\t', quoting=3)\n",
    "test = pd.read_csv( './data/testData.tsv', header=0, delimiter='\\t', quoting=3)\n",
    "unlabeled_train = pd.read_csv( './data/unlabeledTrainData.tsv', header=0, delimiter='\\t', quoting=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 3)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"5814_8\"</td>\n",
       "      <td>1</td>\n",
       "      <td>\"With all this stuff going down at the moment ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"2381_9\"</td>\n",
       "      <td>1</td>\n",
       "      <td>\"\\\"The Classic War of the Worlds\\\" by Timothy ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"7759_3\"</td>\n",
       "      <td>0</td>\n",
       "      <td>\"The film starts with a manager (Nicholas Bell...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"3630_4\"</td>\n",
       "      <td>0</td>\n",
       "      <td>\"It must be assumed that those who praised thi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"9495_8\"</td>\n",
       "      <td>1</td>\n",
       "      <td>\"Superbly trashy and wondrously unpretentious ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id  sentiment                                             review\n",
       "0  \"5814_8\"          1  \"With all this stuff going down at the moment ...\n",
       "1  \"2381_9\"          1  \"\\\"The Classic War of the Worlds\\\" by Timothy ...\n",
       "2  \"7759_3\"          0  \"The film starts with a manager (Nicholas Bell...\n",
       "3  \"3630_4\"          0  \"It must be assumed that those who praised thi...\n",
       "4  \"9495_8\"          1  \"Superbly trashy and wondrously unpretentious ..."
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(train.shape)\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"12311_10\"</td>\n",
       "      <td>\"Naturally in a film who's main themes are of ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"8348_2\"</td>\n",
       "      <td>\"This movie is a disaster within a disaster fi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"5828_4\"</td>\n",
       "      <td>\"All in all, this is a movie for kids. We saw ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"7186_2\"</td>\n",
       "      <td>\"Afraid of the Dark left me with the impressio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"12128_7\"</td>\n",
       "      <td>\"A very accurate depiction of small time mob l...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           id                                             review\n",
       "0  \"12311_10\"  \"Naturally in a film who's main themes are of ...\n",
       "1    \"8348_2\"  \"This movie is a disaster within a disaster fi...\n",
       "2    \"5828_4\"  \"All in all, this is a movie for kids. We saw ...\n",
       "3    \"7186_2\"  \"Afraid of the Dark left me with the impressio...\n",
       "4   \"12128_7\"  \"A very accurate depiction of small time mob l..."
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(test.shape)\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50000, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"9999_0\"</td>\n",
       "      <td>\"Watching Time Chasers, it obvious that it was...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"45057_0\"</td>\n",
       "      <td>\"I saw this film about 20 years ago and rememb...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"15561_0\"</td>\n",
       "      <td>\"Minor Spoilers&lt;br /&gt;&lt;br /&gt;In New York, Joan B...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"7161_0\"</td>\n",
       "      <td>\"I went to see this film with a great deal of ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"43971_0\"</td>\n",
       "      <td>\"Yes, I agree with everyone on this site this ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id                                             review\n",
       "0   \"9999_0\"  \"Watching Time Chasers, it obvious that it was...\n",
       "1  \"45057_0\"  \"I saw this film about 20 years ago and rememb...\n",
       "2  \"15561_0\"  \"Minor Spoilers<br /><br />In New York, Joan B...\n",
       "3   \"7161_0\"  \"I went to see this film with a great deal of ...\n",
       "4  \"43971_0\"  \"Yes, I agree with everyone on this site this ..."
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(unlabeled_train.shape)\n",
    "unlabeled_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train = pd.concat([train,unlabeled_train]).fillna(0)\n",
    "data_train.to_csv('./data/data_train.csv',index=None,encoding='utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train = pd.read_csv('./data/data_train.csv', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_texts = []\n",
    "for idx in range(data_train.review.shape[0]):\n",
    "    train_texts.append(clean_str(BeautifulSoup(data_train.review[idx], \"lxml\").get_text()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"with all this stuff going down at the moment with mj i 've started listening to his music , watching the odd documentary here and there , watched the wiz and watched moonwalker again maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent moonwalker is part biography , part feature film which i remember going to see at the cinema when it was originally released some of it has subtle messages about mj 's feeling towards the press and also the obvious message of drugs are bad m'kay visually impressive but of course this is all about michael jackson so unless you remotely like mj in anyway then you are going to hate this and find it boring some may call mj an egotist for consenting to the making of this movie but mj and most of his fans would say that he made it for the fans which if true is really nice of him the actual feature film bit when it finally starts is only on for 20 minutes or so excluding the smooth criminal sequence and joe pesci is convincing as a psychopathic all powerful drug lord why he wants mj dead so bad is beyond me because mj overheard his plans \\\\? nah , joe pesci 's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno , maybe he just hates mj 's music lots of cool things in this like mj turning into a car and a robot and the whole speed demon sequence also , the director must have had the patience of a saint when it came to filming the kiddy bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene bottom line , this movie is for people who like mj on one level or another \\\\( which i think is most people \\\\) if not , then stay away it does try and give off a wholesome message and ironically mj 's bestest buddy in this movie is a girl ! michael jackson is truly one of the most talented people ever to grace this planet but is he guilty \\\\? well , with all the attention i 've gave this subject hmmm well i do n't know because people can be different behind closed doors , i know this for a fact he is either an extremely nice but stupid guy or one of the most sickest liars i hope he is not the latter\""
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_texts[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01 16:08:07.578801 0\n",
      "2018-10-01 16:08:07.620714 10000\n",
      "2018-10-01 16:08:07.659620 20000\n",
      "2018-10-01 16:08:07.699507 30000\n",
      "2018-10-01 16:08:07.739400 40000\n",
      "2018-10-01 16:08:07.778268 50000\n",
      "2018-10-01 16:08:07.821178 60000\n",
      "2018-10-01 16:08:07.860050 70000\n"
     ]
    }
   ],
   "source": [
    "import codecs\n",
    "from datetime import datetime\n",
    "# -------------------add row number to query----------------------\n",
    "doc_f = codecs.open('./data/' + 'data_train_id.txt','w',encoding='utf8')\n",
    "for i, reviews in enumerate(train_texts):\n",
    "    tags = [i]\n",
    "    if i % 10000 == 0:\n",
    "        print(datetime.now(),i)\n",
    "    doc_f.write('_*{} {}\\n'.format(i,reviews))\n",
    "doc_f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "\n",
    "SentimentDocument = namedtuple('SentimentDocument', 'words tags')\n",
    "class Doc_list(object):\n",
    "    def __init__(self,f):\n",
    "        self.f = f\n",
    "    def __iter__(self):\n",
    "        for i,line in enumerate(codecs.open(self.f,encoding='utf8')):\n",
    "            words = line.split()\n",
    "            tags = [int(words[0][2:])]\n",
    "            words = words[1:]\n",
    "            yield SentimentDocument(words,tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\gensim\\utils.py:1197: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n",
      "  warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\gensim\\models\\doc2vec.py:366: UserWarning: The parameter `size` is deprecated, will be removed in 4.0.0, use `vector_size` instead.\n",
      "  warnings.warn(\"The parameter `size` is deprecated, will be removed in 4.0.0, use `vector_size` instead.\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01 16:10:20.332392 pass: 1\n",
      "dbow [0.5072 0.5086 0.536  0.5364 0.54  ] 0.52564\n",
      "2018-10-01 16:10:34.063216 pass: 2\n",
      "dbow [0.5254 0.5406 0.5534 0.5476 0.5322] 0.53984\n",
      "2018-10-01 16:10:47.913431 pass: 3\n",
      "dbow [0.6102 0.6186 0.6352 0.6428 0.6246] 0.6262800000000001\n",
      "2018-10-01 16:11:01.440077 pass: 4\n",
      "dbow [0.6766 0.6762 0.6744 0.6836 0.663 ] 0.67476\n",
      "2018-10-01 16:11:15.808891 pass: 5\n",
      "dbow [0.7278 0.7362 0.7368 0.7376 0.7182] 0.73132\n",
      "2018-10-01 16:11:30.976330 pass: 6\n",
      "dbow [0.7726 0.782  0.7904 0.7858 0.7794] 0.78204\n",
      "2018-10-01 16:11:47.484707 pass: 7\n",
      "dbow [0.8024 0.8106 0.8138 0.825  0.8062] 0.8116\n",
      "2018-10-01 16:12:04.575526 pass: 8\n",
      "dbow [0.8204 0.8252 0.826  0.8326 0.8204] 0.82492\n",
      "2018-10-01 16:12:22.721560 pass: 9\n",
      "dbow [0.8388 0.8386 0.8374 0.8494 0.8364] 0.8401200000000001\n",
      "2018-10-01 16:12:41.326036 pass: 10\n",
      "dbow [0.8522 0.8472 0.8436 0.8648 0.8486] 0.85128\n",
      "2018-10-01 16:12:59.494171 pass: 11\n",
      "dbow [0.8554 0.859  0.8512 0.8672 0.856 ] 0.8577600000000001\n",
      "2018-10-01 16:13:17.179199 pass: 12\n",
      "dbow [0.8586 0.8594 0.86   0.877  0.8556] 0.86212\n",
      "2018-10-01 16:13:35.053340 pass: 13\n",
      "dbow [0.8632 0.865  0.8658 0.8788 0.861 ] 0.86676\n",
      "2018-10-01 16:13:51.882203 pass: 14\n",
      "dbow [0.871  0.8726 0.8674 0.8796 0.8672] 0.8715599999999999\n",
      "2018-10-01 16:14:08.603712 pass: 15\n",
      "dbow [0.872  0.8722 0.8656 0.876  0.8722] 0.8715999999999999\n",
      "2018-10-01 16:14:24.589445 pass: 16\n",
      "dbow [0.8752 0.8742 0.8712 0.8798 0.8708] 0.87424\n",
      "2018-10-01 16:14:40.657631 pass: 17\n",
      "dbow [0.8742 0.8726 0.8718 0.8766 0.871 ] 0.8732399999999998\n",
      "2018-10-01 16:14:56.152767 pass: 18\n",
      "dbow [0.8778 0.8756 0.869  0.877  0.8692] 0.8737199999999999\n",
      "2018-10-01 16:15:11.313955 pass: 19\n",
      "dbow [0.88   0.8746 0.8692 0.8776 0.8732] 0.87492\n",
      "2018-10-01 16:15:26.509472 pass: 20\n",
      "dbow [0.8788 0.875  0.8712 0.8834 0.8714] 0.8759599999999998\n",
      "2018-10-01 16:15:41.745310 pass: 21\n",
      "dbow [0.8834 0.8778 0.869  0.878  0.8764] 0.8769200000000001\n",
      "2018-10-01 16:15:57.180603 pass: 22\n",
      "dbow [0.8862 0.8748 0.8702 0.8762 0.874 ] 0.8762800000000001\n",
      "2018-10-01 16:16:12.406086 pass: 23\n",
      "dbow [0.8792 0.8776 0.8706 0.8812 0.8746] 0.8766400000000001\n",
      "2018-10-01 16:16:27.753959 pass: 24\n",
      "dbow [0.8826 0.8808 0.8714 0.8824 0.8782] 0.8790800000000001\n",
      "2018-10-01 16:16:42.953012 pass: 25\n",
      "dbow [0.885  0.8808 0.8726 0.8798 0.8778] 0.8792\n",
      "2018-10-01 16:16:57.940818 pass: 26\n",
      "dbow [0.8814 0.88   0.8714 0.8816 0.8766] 0.8782\n",
      "2018-10-01 16:17:12.777648 pass: 27\n",
      "dbow [0.8802 0.8834 0.875  0.8808 0.8776] 0.8794000000000001\n",
      "2018-10-01 16:17:27.722891 pass: 28\n",
      "dbow [0.88   0.8806 0.8762 0.8818 0.878 ] 0.8793200000000001\n",
      "2018-10-01 16:17:42.315989 pass: 29\n",
      "dbow [0.8846 0.8796 0.8748 0.8826 0.877 ] 0.8797200000000001\n",
      "2018-10-01 16:17:57.231429 pass: 30\n",
      "dbow [0.884  0.8794 0.8762 0.8824 0.8764] 0.8796799999999999\n",
      "2018-10-01 16:18:11.788143 pass: 31\n",
      "dbow [0.8822 0.8774 0.8734 0.8814 0.8776] 0.8784000000000001\n",
      "2018-10-01 16:18:26.836438 pass: 32\n",
      "dbow [0.8822 0.8794 0.8738 0.8806 0.8736] 0.8779199999999999\n",
      "2018-10-01 16:18:41.826415 pass: 33\n",
      "dbow [0.8832 0.8796 0.8736 0.8804 0.8748] 0.8783200000000001\n",
      "2018-10-01 16:18:56.236506 pass: 34\n",
      "dbow [0.882  0.8808 0.875  0.8822 0.875 ] 0.8789999999999999\n",
      "2018-10-01 16:19:11.232483 pass: 35\n",
      "dbow [0.8848 0.879  0.8726 0.8836 0.877 ] 0.8794000000000001\n",
      "2018-10-01 16:19:26.198603 pass: 36\n",
      "dbow [0.8838 0.8804 0.8734 0.8842 0.8766] 0.8796799999999999\n",
      "2018-10-01 16:19:41.245997 pass: 37\n",
      "dbow [0.8814 0.8814 0.876  0.8856 0.8768] 0.88024\n",
      "2018-10-01 16:19:55.648317 pass: 38\n",
      "dbow [0.8824 0.8822 0.8816 0.8852 0.8782] 0.8819199999999998\n",
      "2018-10-01 16:20:10.514640 pass: 39\n",
      "dbow [0.8824 0.8812 0.877  0.886  0.8782] 0.88096\n",
      "2018-10-01 16:20:25.277696 pass: 40\n",
      "dbow [0.8842 0.8828 0.8796 0.8868 0.8824] 0.88316\n",
      "2018-10-01 16:20:39.861485 pass: 41\n",
      "dbow [0.88   0.8822 0.8782 0.8868 0.879 ] 0.88124\n",
      "2018-10-01 16:20:54.769249 pass: 42\n",
      "dbow [0.8804 0.8862 0.8794 0.885  0.8804] 0.88228\n",
      "2018-10-01 16:21:09.168938 pass: 43\n",
      "dbow [0.8804 0.884  0.8808 0.8828 0.8808] 0.8817600000000001\n",
      "2018-10-01 16:21:23.976182 pass: 44\n",
      "dbow [0.886  0.8848 0.8786 0.886  0.8778] 0.88264\n",
      "2018-10-01 16:21:38.670641 pass: 45\n",
      "dbow [0.8832 0.8816 0.8784 0.8876 0.8786] 0.88188\n",
      "2018-10-01 16:21:53.004431 pass: 46\n",
      "dbow [0.8844 0.883  0.8796 0.8862 0.879 ] 0.8824400000000001\n",
      "2018-10-01 16:22:07.826851 pass: 47\n",
      "dbow [0.8826 0.8776 0.8776 0.885  0.8782] 0.8802\n",
      "2018-10-01 16:22:22.571724 pass: 48\n",
      "dbow [0.8822 0.8804 0.878  0.8884 0.8804] 0.88188\n",
      "2018-10-01 16:22:36.879573 pass: 49\n",
      "dbow [0.8828 0.8802 0.8792 0.8876 0.8806] 0.88208\n",
      "2018-10-01 16:22:51.544674 pass: 50\n",
      "dbow [0.8836 0.8798 0.8778 0.89   0.8812] 0.8824799999999999\n",
      "2018-10-01 16:23:07.627497 save done\n"
     ]
    }
   ],
   "source": [
    "from gensim.models.doc2vec import Doc2Vec\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "d2v = Doc2Vec(dm=0, size=300, negative=5, hs=0, min_count=3, window=30,sample=1e-5,\n",
    "              workers=6,alpha=0.025,min_alpha=0.025,epochs=1)\n",
    "doc_list = Doc_list('./data/' + 'data_train_id.txt')\n",
    "d2v.build_vocab(doc_list)\n",
    "\n",
    "# -------------------train dbow doc2vec---------------------------------------------\n",
    "for i in range(50):\n",
    "    print(datetime.now(),'pass:',i + 1)\n",
    "    doc_list = Doc_list('./data/' + 'data_train_id.txt')\n",
    "    d2v.train(doc_list,total_examples=d2v.corpus_count, epochs=d2v.epochs)\n",
    "    X_d2v = np.array([d2v.docvecs[i] for i in range(25000)])\n",
    "    scores = cross_val_score(LogisticRegression(C=3),X_d2v,labels,cv=5,n_jobs=4)\n",
    "    print('dbow',scores,np.mean(scores))\n",
    "d2v.save('./data/' + 'dbow_d2v_2.model')\n",
    "print(datetime.now(),'save done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\gensim\\models\\doc2vec.py:366: UserWarning: The parameter `size` is deprecated, will be removed in 4.0.0, use `vector_size` instead.\n",
      "  warnings.warn(\"The parameter `size` is deprecated, will be removed in 4.0.0, use `vector_size` instead.\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01 16:26:06.380045 pass: 1\n",
      "dbow [0.5012 0.5108 0.5348 0.5692 0.5716] 0.53752\n",
      "2018-10-01 16:26:21.271950 pass: 2\n",
      "dbow [0.687  0.7096 0.7366 0.742  0.7192] 0.71888\n",
      "2018-10-01 16:26:37.673938 pass: 3\n",
      "dbow [0.7616 0.7714 0.7792 0.7804 0.7806] 0.77464\n",
      "2018-10-01 16:26:55.519131 pass: 4\n",
      "dbow [0.7952 0.8038 0.802  0.8002 0.8014] 0.80052\n",
      "2018-10-01 16:27:14.858856 pass: 5\n",
      "dbow [0.8188 0.8248 0.8182 0.8238 0.8166] 0.82044\n",
      "2018-10-01 16:27:34.357457 pass: 6\n",
      "dbow [0.8326 0.8354 0.827  0.8336 0.8272] 0.83116\n",
      "2018-10-01 16:27:55.431112 pass: 7\n",
      "dbow [0.8388 0.8426 0.8374 0.8358 0.8348] 0.83788\n",
      "2018-10-01 16:28:15.609199 pass: 8\n",
      "dbow [0.8456 0.8452 0.8426 0.8426 0.8306] 0.84132\n",
      "2018-10-01 16:28:35.812930 pass: 9\n",
      "dbow [0.8496 0.8466 0.842  0.8438 0.8334] 0.8430799999999999\n",
      "2018-10-01 16:28:55.118947 pass: 10\n",
      "dbow [0.8486 0.8478 0.8422 0.8468 0.8396] 0.8450000000000001\n",
      "2018-10-01 16:29:12.356837 pass: 11\n",
      "dbow [0.849  0.8504 0.8436 0.8484 0.8458] 0.84744\n",
      "2018-10-01 16:29:29.181745 pass: 12\n",
      "dbow [0.8534 0.851  0.8416 0.848  0.84  ] 0.8468\n",
      "2018-10-01 16:29:45.705072 pass: 13\n",
      "dbow [0.8546 0.8524 0.8444 0.8462 0.8428] 0.8480800000000001\n",
      "2018-10-01 16:30:01.596471 pass: 14\n",
      "dbow [0.8562 0.8544 0.8436 0.8504 0.843 ] 0.84952\n",
      "2018-10-01 16:30:17.191229 pass: 15\n",
      "dbow [0.857  0.8564 0.8472 0.8512 0.8448] 0.85132\n",
      "2018-10-01 16:30:32.644377 pass: 16\n",
      "dbow [0.8604 0.8554 0.8492 0.8528 0.8486] 0.8532800000000001\n",
      "2018-10-01 16:30:47.866278 pass: 17\n",
      "dbow [0.8614 0.857  0.8498 0.8552 0.8476] 0.8542\n",
      "2018-10-01 16:31:03.145044 pass: 18\n",
      "dbow [0.8598 0.8596 0.85   0.8562 0.8446] 0.85404\n",
      "2018-10-01 16:31:18.421512 pass: 19\n",
      "dbow [0.8578 0.862  0.8498 0.8576 0.8468] 0.8548\n",
      "2018-10-01 16:31:33.624979 pass: 20\n",
      "dbow [0.8618 0.8622 0.8518 0.8604 0.851 ] 0.8574400000000001\n",
      "2018-10-01 16:31:48.972269 pass: 21\n",
      "dbow [0.8618 0.8602 0.8486 0.863  0.8534] 0.8573999999999999\n",
      "2018-10-01 16:32:04.867827 pass: 22\n",
      "dbow [0.8618 0.8612 0.8504 0.8584 0.853 ] 0.8569599999999999\n",
      "2018-10-01 16:32:20.067544 pass: 23\n",
      "dbow [0.86   0.8618 0.8536 0.859  0.8552] 0.85792\n",
      "2018-10-01 16:32:35.375695 pass: 24\n",
      "dbow [0.8602 0.8628 0.856  0.8582 0.857 ] 0.85884\n",
      "2018-10-01 16:32:50.816806 pass: 25\n",
      "dbow [0.8578 0.863  0.8524 0.8612 0.854 ] 0.85768\n",
      "2018-10-01 16:33:06.413151 pass: 26\n",
      "dbow [0.8594 0.8618 0.8522 0.858  0.8562] 0.8575200000000001\n",
      "2018-10-01 16:33:21.915759 pass: 27\n",
      "dbow [0.8618 0.8616 0.8532 0.863  0.8546] 0.85884\n",
      "2018-10-01 16:33:37.375946 pass: 28\n",
      "dbow [0.863  0.864  0.8522 0.8632 0.856 ] 0.85968\n",
      "2018-10-01 16:33:52.716324 pass: 29\n",
      "dbow [0.861  0.8654 0.8522 0.8646 0.8544] 0.8595200000000001\n",
      "2018-10-01 16:34:07.891634 pass: 30\n",
      "dbow [0.8652 0.8648 0.8536 0.8662 0.856 ] 0.8611600000000001\n",
      "2018-10-01 16:34:23.211452 pass: 31\n",
      "dbow [0.8642 0.8648 0.8502 0.8666 0.8556] 0.86028\n",
      "2018-10-01 16:34:38.346771 pass: 32\n",
      "dbow [0.8642 0.864  0.8526 0.8628 0.8614] 0.861\n",
      "2018-10-01 16:34:53.419808 pass: 33\n",
      "dbow [0.8624 0.8646 0.8558 0.8642 0.8598] 0.86136\n",
      "2018-10-01 16:35:08.745051 pass: 34\n",
      "dbow [0.8638 0.862  0.8558 0.8638 0.8604] 0.8611599999999999\n",
      "2018-10-01 16:35:23.872724 pass: 35\n",
      "dbow [0.8672 0.8676 0.8564 0.8644 0.8612] 0.8633599999999999\n",
      "2018-10-01 16:35:38.951442 pass: 36\n",
      "dbow [0.869  0.8656 0.8592 0.8626 0.863 ] 0.86388\n",
      "2018-10-01 16:35:54.153412 pass: 37\n",
      "dbow [0.8666 0.8648 0.8576 0.866  0.8612] 0.86324\n",
      "2018-10-01 16:36:09.293386 pass: 38\n",
      "dbow [0.8662 0.8676 0.8594 0.8648 0.8618] 0.86396\n",
      "2018-10-01 16:36:24.294264 pass: 39\n",
      "dbow [0.867  0.8654 0.8594 0.865  0.8612] 0.8636000000000001\n",
      "2018-10-01 16:36:39.430998 pass: 40\n",
      "dbow [0.8664 0.865  0.858  0.8648 0.8614] 0.86312\n",
      "2018-10-01 16:36:54.652840 pass: 41\n",
      "dbow [0.866  0.8668 0.8594 0.8646 0.8612] 0.8636000000000001\n",
      "2018-10-01 16:37:09.724203 pass: 42\n",
      "dbow [0.8664 0.8682 0.8584 0.865  0.864 ] 0.8644000000000001\n",
      "2018-10-01 16:37:24.832137 pass: 43\n",
      "dbow [0.8662 0.8684 0.8586 0.8668 0.863 ] 0.8646\n",
      "2018-10-01 16:37:39.857404 pass: 44\n",
      "dbow [0.866  0.8682 0.8608 0.8678 0.862 ] 0.86496\n",
      "2018-10-01 16:37:55.008954 pass: 45\n",
      "dbow [0.8676 0.8696 0.8602 0.8694 0.8628] 0.86592\n",
      "2018-10-01 16:38:10.339951 pass: 46\n",
      "dbow [0.8666 0.8698 0.8584 0.8686 0.863 ] 0.8652799999999999\n",
      "2018-10-01 16:38:25.733656 pass: 47\n",
      "dbow [0.8668 0.8704 0.8594 0.8694 0.8634] 0.86588\n",
      "2018-10-01 16:38:40.947996 pass: 48\n",
      "dbow [0.8674 0.8714 0.86   0.8698 0.8632] 0.8663599999999999\n",
      "2018-10-01 16:38:56.117170 pass: 49\n",
      "dbow [0.8666 0.8716 0.8586 0.8698 0.8632] 0.8659600000000001\n",
      "2018-10-01 16:39:11.308000 pass: 50\n",
      "dbow [0.8672 0.8716 0.8564 0.8684 0.8646] 0.86564\n",
      "2018-10-01 16:39:27.557738 save done\n"
     ]
    }
   ],
   "source": [
    "d2v = Doc2Vec(dm=1, size=300, negative=5, hs=0, min_count=3, window=10,sample=1e-5,\n",
    "              workers=6,alpha=0.05,min_alpha=0.025,epochs=1)\n",
    "doc_list = Doc_list('./data/' + 'data_train_id.txt')\n",
    "d2v.build_vocab(doc_list)\n",
    "\n",
    "# -------------------train dm doc2vec---------------------------------------------\n",
    "for i in range(50):\n",
    "    print(datetime.now(),'pass:',i + 1)\n",
    "    doc_list = Doc_list('./data/' + 'data_train_id.txt')\n",
    "    d2v.train(doc_list,total_examples=d2v.corpus_count, epochs=d2v.epochs)\n",
    "    X_d2v = np.array([d2v.docvecs[i] for i in range(25000)])\n",
    "    scores = cross_val_score(LogisticRegression(C=3),X_d2v,labels,cv=5,n_jobs=4)\n",
    "    print('dbow',scores,np.mean(scores))\n",
    "d2v.save('./data/' + 'dm_d2v_2.model')\n",
    "print(datetime.now(),'save done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\gensim\\models\\doc2vec.py:366: UserWarning: The parameter `size` is deprecated, will be removed in 4.0.0, use `vector_size` instead.\n",
      "  warnings.warn(\"The parameter `size` is deprecated, will be removed in 4.0.0, use `vector_size` instead.\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01 16:43:16.771053 pass: 1\n",
      "dbow [0.515  0.5092 0.5426 0.564  0.5674] 0.53964\n",
      "2018-10-01 16:43:32.768131 pass: 2\n",
      "dbow [0.6874 0.7098 0.7244 0.7334 0.719 ] 0.7148\n",
      "2018-10-01 16:43:50.502366 pass: 3\n",
      "dbow [0.757  0.7838 0.7706 0.78   0.7772] 0.7737200000000001\n",
      "2018-10-01 16:44:09.880086 pass: 4\n",
      "dbow [0.801  0.804  0.8006 0.812  0.7996] 0.8034399999999999\n",
      "2018-10-01 16:44:30.825801 pass: 5\n",
      "dbow [0.8134 0.8302 0.8098 0.8206 0.8172] 0.81824\n",
      "2018-10-01 16:44:53.859074 pass: 6\n",
      "dbow [0.8296 0.8424 0.8222 0.8298 0.8262] 0.83004\n",
      "2018-10-01 16:45:16.009211 pass: 7\n",
      "dbow [0.8372 0.8458 0.8304 0.8348 0.829 ] 0.83544\n",
      "2018-10-01 16:45:40.372005 pass: 8\n",
      "dbow [0.8376 0.8476 0.8346 0.8404 0.8344] 0.8389199999999999\n",
      "2018-10-01 16:46:05.600841 pass: 9\n",
      "dbow [0.8446 0.8444 0.8326 0.8416 0.8382] 0.8402800000000001\n",
      "2018-10-01 16:46:29.385896 pass: 10\n",
      "dbow [0.8452 0.8474 0.8396 0.8418 0.8404] 0.8428800000000001\n",
      "2018-10-01 16:46:51.753766 pass: 11\n",
      "dbow [0.8482 0.8498 0.8382 0.8412 0.8442] 0.84432\n",
      "2018-10-01 16:47:12.663795 pass: 12\n",
      "dbow [0.8484 0.8538 0.843  0.8472 0.843 ] 0.84708\n",
      "2018-10-01 16:47:32.273471 pass: 13\n",
      "dbow [0.8534 0.8542 0.8428 0.849  0.8414] 0.84816\n",
      "2018-10-01 16:47:50.863345 pass: 14\n",
      "dbow [0.8552 0.8568 0.843  0.8494 0.8488] 0.85064\n",
      "2018-10-01 16:48:08.813606 pass: 15\n",
      "dbow [0.8518 0.8574 0.8482 0.852  0.8544] 0.85276\n",
      "2018-10-01 16:48:26.205959 pass: 16\n",
      "dbow [0.8528 0.8536 0.8446 0.8516 0.8524] 0.851\n",
      "2018-10-01 16:48:43.467230 pass: 17\n",
      "dbow [0.853  0.8526 0.8448 0.8542 0.8538] 0.85168\n",
      "2018-10-01 16:49:00.370630 pass: 18\n",
      "dbow [0.8556 0.8528 0.8468 0.8558 0.8564] 0.85348\n",
      "2018-10-01 16:49:17.036565 pass: 19\n",
      "dbow [0.8562 0.8582 0.8464 0.858  0.8578] 0.8553200000000001\n",
      "2018-10-01 16:49:33.619065 pass: 20\n",
      "dbow [0.859  0.8564 0.8452 0.859  0.8572] 0.8553599999999999\n",
      "2018-10-01 16:49:49.950220 pass: 21\n",
      "dbow [0.861  0.8586 0.8508 0.8634 0.8574] 0.85824\n",
      "2018-10-01 16:50:06.393654 pass: 22\n",
      "dbow [0.8602 0.856  0.851  0.8632 0.8568] 0.8574399999999999\n",
      "2018-10-01 16:50:22.715855 pass: 23\n",
      "dbow [0.8626 0.8564 0.8514 0.8592 0.8596] 0.85784\n",
      "2018-10-01 16:50:38.949572 pass: 24\n",
      "dbow [0.862  0.8596 0.8508 0.8626 0.8588] 0.85876\n",
      "2018-10-01 16:50:55.447728 pass: 25\n",
      "dbow [0.8624 0.8582 0.8512 0.8598 0.8578] 0.85788\n",
      "2018-10-01 16:51:11.915352 pass: 26\n",
      "dbow [0.8618 0.858  0.8518 0.8646 0.859 ] 0.85904\n",
      "2018-10-01 16:51:28.276395 pass: 27\n",
      "dbow [0.8608 0.861  0.8538 0.8636 0.8602] 0.8598800000000001\n",
      "2018-10-01 16:51:44.725595 pass: 28\n",
      "dbow [0.8638 0.8602 0.854  0.8622 0.8574] 0.8595200000000001\n",
      "2018-10-01 16:52:01.190344 pass: 29\n",
      "dbow [0.8664 0.8564 0.8522 0.8624 0.8572] 0.85892\n",
      "2018-10-01 16:52:17.467488 pass: 30\n",
      "dbow [0.8646 0.8594 0.8514 0.8648 0.86  ] 0.86004\n",
      "2018-10-01 16:52:33.932802 pass: 31\n",
      "dbow [0.8668 0.8622 0.8568 0.8658 0.8608] 0.86248\n",
      "2018-10-01 16:52:50.387550 pass: 32\n",
      "dbow [0.869  0.8608 0.858  0.8644 0.8618] 0.8628\n",
      "2018-10-01 16:53:06.629782 pass: 33\n",
      "dbow [0.8674 0.8612 0.8582 0.8638 0.8594] 0.8619999999999999\n",
      "2018-10-01 16:53:23.104712 pass: 34\n",
      "dbow [0.8674 0.8632 0.859  0.8674 0.8584] 0.8630800000000001\n",
      "2018-10-01 16:53:39.454363 pass: 35\n",
      "dbow [0.8676 0.8626 0.8576 0.866  0.8604] 0.86284\n",
      "2018-10-01 16:53:55.559803 pass: 36\n",
      "dbow [0.8688 0.8626 0.858  0.8672 0.8602] 0.8633599999999999\n",
      "2018-10-01 16:54:11.797827 pass: 37\n",
      "dbow [0.868  0.861  0.859  0.8686 0.8586] 0.86304\n",
      "2018-10-01 16:54:28.114544 pass: 38\n",
      "dbow [0.8678 0.865  0.8556 0.8676 0.8604] 0.8632799999999999\n",
      "2018-10-01 16:54:44.232051 pass: 39\n",
      "dbow [0.867  0.8674 0.8574 0.8702 0.8608] 0.86456\n",
      "2018-10-01 16:55:00.301122 pass: 40\n",
      "dbow [0.8678 0.864  0.857  0.8704 0.8616] 0.86416\n",
      "2018-10-01 16:55:16.333156 pass: 41\n",
      "dbow [0.869  0.866  0.8554 0.8714 0.862 ] 0.8647599999999999\n",
      "2018-10-01 16:55:32.266636 pass: 42\n",
      "dbow [0.868  0.8654 0.8556 0.8698 0.8642] 0.8646\n",
      "2018-10-01 16:55:48.143190 pass: 43\n",
      "dbow [0.8686 0.8678 0.858  0.8718 0.8634] 0.86592\n",
      "2018-10-01 16:56:04.108157 pass: 44\n",
      "dbow [0.8688 0.8662 0.857  0.87   0.864 ] 0.8652\n",
      "2018-10-01 16:56:20.045888 pass: 45\n",
      "dbow [0.867  0.8678 0.8602 0.8708 0.864 ] 0.86596\n",
      "2018-10-01 16:56:35.986840 pass: 46\n",
      "dbow [0.8678 0.8676 0.8602 0.8718 0.8652] 0.8665200000000001\n",
      "2018-10-01 16:56:51.903730 pass: 47\n",
      "dbow [0.8678 0.8672 0.8586 0.8714 0.8664] 0.8662799999999999\n",
      "2018-10-01 16:57:08.175221 pass: 48\n",
      "dbow [0.8712 0.866  0.8574 0.8712 0.8676] 0.86668\n",
      "2018-10-01 16:57:24.641224 pass: 49\n",
      "dbow [0.8698 0.8696 0.8578 0.8734 0.863 ] 0.8667200000000002\n",
      "2018-10-01 16:57:41.011798 pass: 50\n",
      "dbow [0.8712 0.8682 0.862  0.8716 0.8624] 0.86708\n",
      "2018-10-01 16:57:58.986838 save done\n"
     ]
    }
   ],
   "source": [
    "d2v = Doc2Vec(dm=1, size=400, negative=5, hs=0, min_count=3, window=10,sample=1e-5,\n",
    "              workers=6,alpha=0.05,min_alpha=0.025,epochs=1)\n",
    "doc_list = Doc_list('./data/' + 'data_train_id.txt')\n",
    "d2v.build_vocab(doc_list)\n",
    "\n",
    "# -------------------train dm doc2vec---------------------------------------------\n",
    "for i in range(50):\n",
    "    print(datetime.now(),'pass:',i + 1)\n",
    "    doc_list = Doc_list('./data/' + 'data_train_id.txt')\n",
    "    d2v.train(doc_list,total_examples=d2v.corpus_count, epochs=d2v.epochs)\n",
    "    X_d2v = np.array([d2v.docvecs[i] for i in range(25000)])\n",
    "    scores = cross_val_score(LogisticRegression(C=3),X_d2v,labels,cv=5,n_jobs=4)\n",
    "    print('dbow',scores,np.mean(scores))\n",
    "d2v.save('./data/' + 'dm_d2v_3.model')\n",
    "print(datetime.now(),'save done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\gensim\\models\\doc2vec.py:366: UserWarning: The parameter `size` is deprecated, will be removed in 4.0.0, use `vector_size` instead.\n",
      "  warnings.warn(\"The parameter `size` is deprecated, will be removed in 4.0.0, use `vector_size` instead.\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01 17:50:51.895676 pass: 1\n",
      "dbow [0.5164 0.5164 0.5306 0.5366 0.5196] 0.5239199999999999\n",
      "2018-10-01 18:00:03.867105 pass: 2\n",
      "dbow [0.5296 0.542  0.5456 0.5528 0.5382] 0.5416399999999999\n",
      "2018-10-01 18:10:13.599449 save done\n"
     ]
    }
   ],
   "source": [
    "import codecs\n",
    "from datetime import datetime\n",
    "from gensim.models.doc2vec import Doc2Vec\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "d2v = Doc2Vec(dm=1, dm_concat=1, size=400, negative=5, hs=0, min_count=3, window=10, sample=1e-5,\n",
    "              workers=6, alpha=0.05, min_alpha=0.025, epochs=1)\n",
    "doc_list = Doc_list('./data/' + 'data_train_id.txt')\n",
    "d2v.build_vocab(doc_list)\n",
    "\n",
    "# -------------------train dm doc2vec---------------------------------------------\n",
    "for i in range(2):\n",
    "    print(datetime.now(),'pass:',i + 1)\n",
    "    doc_list = Doc_list('./data/' + 'data_train_id.txt')\n",
    "    d2v.train(doc_list,total_examples=d2v.corpus_count, epochs=d2v.epochs)\n",
    "    X_d2v = np.array([d2v.docvecs[i] for i in range(25000)])\n",
    "    scores = cross_val_score(LogisticRegression(C=3),X_d2v,labels,cv=5,n_jobs=4)\n",
    "    print('dbow',scores,np.mean(scores))\n",
    "d2v.save('./data/' + 'dm_d2v_4.model')\n",
    "print(datetime.now(),'save done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dbow-nn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(pd.value_counts(labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\sklearn\\cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n",
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\h5py\\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01 23:03:11.215836 stack:1/5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:66: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 3s - loss: 0.3370 - acc: 0.8564 - val_loss: 0.3113 - val_acc: 0.8660\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.2902 - acc: 0.8782 - val_loss: 0.3125 - val_acc: 0.8692\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2865 - acc: 0.8824 - val_loss: 0.3000 - val_acc: 0.8754\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2836 - acc: 0.8811 - val_loss: 0.2958 - val_acc: 0.8772\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2812 - acc: 0.8817 - val_loss: 0.2953 - val_acc: 0.8766\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2775 - acc: 0.8834 - val_loss: 0.3022 - val_acc: 0.8726\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2775 - acc: 0.8851 - val_loss: 0.2967 - val_acc: 0.8764\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2755 - acc: 0.8859 - val_loss: 0.2962 - val_acc: 0.8762\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2736 - acc: 0.8859 - val_loss: 0.2979 - val_acc: 0.8760\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.2724 - acc: 0.8855 - val_loss: 0.2954 - val_acc: 0.8754\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.2698 - acc: 0.8862 - val_loss: 0.2953 - val_acc: 0.8754\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.2671 - acc: 0.8890 - val_loss: 0.2944 - val_acc: 0.8766\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.2659 - acc: 0.8892 - val_loss: 0.2944 - val_acc: 0.8774\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.2645 - acc: 0.8907 - val_loss: 0.2949 - val_acc: 0.8760\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.2613 - acc: 0.8916 - val_loss: 0.3000 - val_acc: 0.8738\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.2582 - acc: 0.8935 - val_loss: 0.2963 - val_acc: 0.8764\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.2563 - acc: 0.8944 - val_loss: 0.2957 - val_acc: 0.8758\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.2548 - acc: 0.8939 - val_loss: 0.2989 - val_acc: 0.8734\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.2498 - acc: 0.8968 - val_loss: 0.2973 - val_acc: 0.8758\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.2470 - acc: 0.8982 - val_loss: 0.2978 - val_acc: 0.8762\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.2416 - acc: 0.9009 - val_loss: 0.2928 - val_acc: 0.8770\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.2384 - acc: 0.9016 - val_loss: 0.2963 - val_acc: 0.8774\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.2352 - acc: 0.9044 - val_loss: 0.2931 - val_acc: 0.8760\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.2308 - acc: 0.9060 - val_loss: 0.2988 - val_acc: 0.8766\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.2249 - acc: 0.9092 - val_loss: 0.2950 - val_acc: 0.8806\n",
      "Epoch 26/35\n",
      " - 0s - loss: 0.2209 - acc: 0.9093 - val_loss: 0.2922 - val_acc: 0.8796\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.2163 - acc: 0.9119 - val_loss: 0.3256 - val_acc: 0.8656\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.2089 - acc: 0.9173 - val_loss: 0.2972 - val_acc: 0.8776\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.2059 - acc: 0.9174 - val_loss: 0.2975 - val_acc: 0.8784\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.1991 - acc: 0.9197 - val_loss: 0.2979 - val_acc: 0.8798\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.1926 - acc: 0.9241 - val_loss: 0.3037 - val_acc: 0.8756\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.1838 - acc: 0.9283 - val_loss: 0.3301 - val_acc: 0.8678\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.1786 - acc: 0.9334 - val_loss: 0.3069 - val_acc: 0.8800\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.1723 - acc: 0.9342 - val_loss: 0.3360 - val_acc: 0.8678\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.1705 - acc: 0.9353 - val_loss: 0.3080 - val_acc: 0.8750\n",
      "va acc: 0.881\n",
      "te acc: 0.875\n",
      "2018-10-01 23:03:26.858157 stack:2/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3373 - acc: 0.8518 - val_loss: 0.3150 - val_acc: 0.8674\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.2916 - acc: 0.8781 - val_loss: 0.3031 - val_acc: 0.8710\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2871 - acc: 0.8794 - val_loss: 0.3130 - val_acc: 0.8668\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2842 - acc: 0.8806 - val_loss: 0.3130 - val_acc: 0.8696\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2786 - acc: 0.8837 - val_loss: 0.2977 - val_acc: 0.8782\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2782 - acc: 0.8848 - val_loss: 0.3007 - val_acc: 0.8728\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2761 - acc: 0.8846 - val_loss: 0.3029 - val_acc: 0.8746\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2753 - acc: 0.8858 - val_loss: 0.3007 - val_acc: 0.8728\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2735 - acc: 0.8861 - val_loss: 0.3010 - val_acc: 0.8746\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.2707 - acc: 0.8876 - val_loss: 0.2979 - val_acc: 0.8744\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.2706 - acc: 0.8864 - val_loss: 0.2989 - val_acc: 0.8770\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.2676 - acc: 0.8918 - val_loss: 0.2979 - val_acc: 0.8732\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.2659 - acc: 0.8908 - val_loss: 0.2996 - val_acc: 0.8724\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.2641 - acc: 0.8896 - val_loss: 0.3016 - val_acc: 0.8744\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.2614 - acc: 0.8909 - val_loss: 0.2986 - val_acc: 0.8742\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.2586 - acc: 0.8936 - val_loss: 0.2978 - val_acc: 0.8736\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.2566 - acc: 0.8950 - val_loss: 0.2994 - val_acc: 0.8752\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.2546 - acc: 0.8951 - val_loss: 0.2979 - val_acc: 0.8732\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.2510 - acc: 0.8979 - val_loss: 0.2965 - val_acc: 0.8752\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.2448 - acc: 0.9008 - val_loss: 0.3014 - val_acc: 0.8740\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.2415 - acc: 0.9038 - val_loss: 0.3027 - val_acc: 0.8736\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.2374 - acc: 0.9033 - val_loss: 0.3003 - val_acc: 0.8746\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.2338 - acc: 0.9040 - val_loss: 0.3003 - val_acc: 0.8754\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.2278 - acc: 0.9098 - val_loss: 0.3025 - val_acc: 0.8750\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.2228 - acc: 0.9106 - val_loss: 0.3005 - val_acc: 0.8762\n",
      "Epoch 26/35\n",
      " - 0s - loss: 0.2183 - acc: 0.9134 - val_loss: 0.3033 - val_acc: 0.8766\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.2124 - acc: 0.9174 - val_loss: 0.3056 - val_acc: 0.8736\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.2083 - acc: 0.9194 - val_loss: 0.3133 - val_acc: 0.8726\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.2039 - acc: 0.9202 - val_loss: 0.3063 - val_acc: 0.8740\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.1957 - acc: 0.9235 - val_loss: 0.3113 - val_acc: 0.8726\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.1889 - acc: 0.9281 - val_loss: 0.3071 - val_acc: 0.8768\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.1854 - acc: 0.9297 - val_loss: 0.3100 - val_acc: 0.8720\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.1772 - acc: 0.9346 - val_loss: 0.3190 - val_acc: 0.8724\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.1712 - acc: 0.9338 - val_loss: 0.3141 - val_acc: 0.8760\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.1668 - acc: 0.9393 - val_loss: 0.3152 - val_acc: 0.8748\n",
      "va acc: 0.8815\n",
      "te acc: 0.8748\n",
      "2018-10-01 23:03:40.326058 stack:3/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3308 - acc: 0.8589 - val_loss: 0.3009 - val_acc: 0.8726\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.2854 - acc: 0.8814 - val_loss: 0.2982 - val_acc: 0.8744\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2802 - acc: 0.8858 - val_loss: 0.2960 - val_acc: 0.8746\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2781 - acc: 0.8849 - val_loss: 0.2939 - val_acc: 0.8736\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2752 - acc: 0.8874 - val_loss: 0.2931 - val_acc: 0.8756\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2737 - acc: 0.8876 - val_loss: 0.2950 - val_acc: 0.8770\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2717 - acc: 0.8874 - val_loss: 0.2948 - val_acc: 0.8726\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2687 - acc: 0.8914 - val_loss: 0.2980 - val_acc: 0.8718\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2683 - acc: 0.8912 - val_loss: 0.2949 - val_acc: 0.8734\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.2651 - acc: 0.8928 - val_loss: 0.2932 - val_acc: 0.8756\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.2634 - acc: 0.8922 - val_loss: 0.2983 - val_acc: 0.8738\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.2608 - acc: 0.8925 - val_loss: 0.2957 - val_acc: 0.8724\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.2595 - acc: 0.8955 - val_loss: 0.3060 - val_acc: 0.8696\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.2580 - acc: 0.8944 - val_loss: 0.2928 - val_acc: 0.8776\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.2545 - acc: 0.8969 - val_loss: 0.3046 - val_acc: 0.8702\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.2530 - acc: 0.8984 - val_loss: 0.2928 - val_acc: 0.8754\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.2513 - acc: 0.8978 - val_loss: 0.2965 - val_acc: 0.8746\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.2466 - acc: 0.8999 - val_loss: 0.2920 - val_acc: 0.8766\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.2425 - acc: 0.9012 - val_loss: 0.2928 - val_acc: 0.8768\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.2403 - acc: 0.9032 - val_loss: 0.2953 - val_acc: 0.8732\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.2341 - acc: 0.9056 - val_loss: 0.2909 - val_acc: 0.8774\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.2315 - acc: 0.9074 - val_loss: 0.2972 - val_acc: 0.8762\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.2265 - acc: 0.9084 - val_loss: 0.2947 - val_acc: 0.8766\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.2226 - acc: 0.9113 - val_loss: 0.3196 - val_acc: 0.8698\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.2188 - acc: 0.9148 - val_loss: 0.2955 - val_acc: 0.8782\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 26/35\n",
      " - 0s - loss: 0.2093 - acc: 0.9176 - val_loss: 0.2961 - val_acc: 0.8760\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.2059 - acc: 0.9207 - val_loss: 0.2942 - val_acc: 0.8794\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.1988 - acc: 0.9223 - val_loss: 0.2999 - val_acc: 0.8766\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.1962 - acc: 0.9232 - val_loss: 0.2986 - val_acc: 0.8804\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.1911 - acc: 0.9266 - val_loss: 0.3110 - val_acc: 0.8732\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.1838 - acc: 0.9287 - val_loss: 0.3126 - val_acc: 0.8750\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.1763 - acc: 0.9342 - val_loss: 0.3043 - val_acc: 0.8798\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.1730 - acc: 0.9329 - val_loss: 0.3291 - val_acc: 0.8708\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.1660 - acc: 0.9380 - val_loss: 0.3058 - val_acc: 0.8798\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.1602 - acc: 0.9407 - val_loss: 0.3190 - val_acc: 0.8734\n",
      "va acc: 0.873\n",
      "te acc: 0.8734\n",
      "2018-10-01 23:03:53.904864 stack:4/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3381 - acc: 0.8539 - val_loss: 0.2960 - val_acc: 0.8750\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.2895 - acc: 0.8788 - val_loss: 0.2948 - val_acc: 0.8760\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2849 - acc: 0.8813 - val_loss: 0.3019 - val_acc: 0.8764\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2837 - acc: 0.8813 - val_loss: 0.2956 - val_acc: 0.8766\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2793 - acc: 0.8828 - val_loss: 0.2931 - val_acc: 0.8780\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2772 - acc: 0.8856 - val_loss: 0.2941 - val_acc: 0.8798\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2755 - acc: 0.8846 - val_loss: 0.2936 - val_acc: 0.8802\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2743 - acc: 0.8844 - val_loss: 0.2927 - val_acc: 0.8786\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2719 - acc: 0.8862 - val_loss: 0.2926 - val_acc: 0.8804\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.2687 - acc: 0.8882 - val_loss: 0.2956 - val_acc: 0.8766\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.2675 - acc: 0.8877 - val_loss: 0.2952 - val_acc: 0.8762\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.2651 - acc: 0.8893 - val_loss: 0.2950 - val_acc: 0.8786\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.2634 - acc: 0.8908 - val_loss: 0.2924 - val_acc: 0.8790\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.2623 - acc: 0.8911 - val_loss: 0.2933 - val_acc: 0.8778\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.2592 - acc: 0.8940 - val_loss: 0.2965 - val_acc: 0.8760\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.2555 - acc: 0.8949 - val_loss: 0.2920 - val_acc: 0.8798\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.2542 - acc: 0.8944 - val_loss: 0.2948 - val_acc: 0.8764\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.2515 - acc: 0.8966 - val_loss: 0.2929 - val_acc: 0.8796\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.2470 - acc: 0.8974 - val_loss: 0.2906 - val_acc: 0.8798\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.2420 - acc: 0.9026 - val_loss: 0.2898 - val_acc: 0.8798\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.2392 - acc: 0.9009 - val_loss: 0.2930 - val_acc: 0.8782\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.2353 - acc: 0.9040 - val_loss: 0.2910 - val_acc: 0.8792\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.2274 - acc: 0.9079 - val_loss: 0.2946 - val_acc: 0.8810\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.2251 - acc: 0.9075 - val_loss: 0.2983 - val_acc: 0.8772\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.2175 - acc: 0.9137 - val_loss: 0.3060 - val_acc: 0.8768\n",
      "Epoch 26/35\n",
      " - 0s - loss: 0.2149 - acc: 0.9140 - val_loss: 0.2976 - val_acc: 0.8810\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.2084 - acc: 0.9169 - val_loss: 0.2969 - val_acc: 0.8782\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.2024 - acc: 0.9206 - val_loss: 0.2962 - val_acc: 0.8778\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.1967 - acc: 0.9233 - val_loss: 0.2955 - val_acc: 0.8812\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.1901 - acc: 0.9259 - val_loss: 0.2974 - val_acc: 0.8824\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.1852 - acc: 0.9279 - val_loss: 0.3030 - val_acc: 0.8772\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.1771 - acc: 0.9318 - val_loss: 0.3028 - val_acc: 0.8814\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.1721 - acc: 0.9333 - val_loss: 0.3025 - val_acc: 0.8768\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.1672 - acc: 0.9368 - val_loss: 0.3066 - val_acc: 0.8834\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.1596 - acc: 0.9412 - val_loss: 0.3244 - val_acc: 0.8792\n",
      "va acc: 0.87925\n",
      "te acc: 0.8792\n",
      "2018-10-01 23:04:07.590329 stack:5/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3353 - acc: 0.8557 - val_loss: 0.3045 - val_acc: 0.8714\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.2937 - acc: 0.8761 - val_loss: 0.2985 - val_acc: 0.8726\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2862 - acc: 0.8792 - val_loss: 0.2978 - val_acc: 0.8754\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2841 - acc: 0.8818 - val_loss: 0.3012 - val_acc: 0.8730\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2814 - acc: 0.8808 - val_loss: 0.2963 - val_acc: 0.8746\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2796 - acc: 0.8844 - val_loss: 0.2972 - val_acc: 0.8696\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2774 - acc: 0.8849 - val_loss: 0.3016 - val_acc: 0.8720\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2758 - acc: 0.8858 - val_loss: 0.2990 - val_acc: 0.8734\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2726 - acc: 0.8868 - val_loss: 0.2950 - val_acc: 0.8738\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.2705 - acc: 0.8881 - val_loss: 0.2932 - val_acc: 0.8752\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.2701 - acc: 0.8868 - val_loss: 0.2946 - val_acc: 0.8770\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.2684 - acc: 0.8881 - val_loss: 0.2939 - val_acc: 0.8772\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.2660 - acc: 0.8889 - val_loss: 0.2962 - val_acc: 0.8762\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.2621 - acc: 0.8917 - val_loss: 0.2936 - val_acc: 0.8764\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.2604 - acc: 0.8920 - val_loss: 0.2954 - val_acc: 0.8746\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.2575 - acc: 0.8943 - val_loss: 0.2942 - val_acc: 0.8756\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.2547 - acc: 0.8966 - val_loss: 0.3016 - val_acc: 0.8750\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.2517 - acc: 0.8962 - val_loss: 0.2917 - val_acc: 0.8798\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.2487 - acc: 0.8988 - val_loss: 0.2941 - val_acc: 0.8772\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.2440 - acc: 0.8994 - val_loss: 0.2933 - val_acc: 0.8782\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.2402 - acc: 0.9033 - val_loss: 0.2928 - val_acc: 0.8758\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.2352 - acc: 0.9051 - val_loss: 0.2927 - val_acc: 0.8776\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.2318 - acc: 0.9058 - val_loss: 0.2940 - val_acc: 0.8762\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.2240 - acc: 0.9086 - val_loss: 0.2995 - val_acc: 0.8746\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.2214 - acc: 0.9126 - val_loss: 0.2955 - val_acc: 0.8754\n",
      "Epoch 26/35\n",
      " - 0s - loss: 0.2140 - acc: 0.9148 - val_loss: 0.2966 - val_acc: 0.8774\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.2081 - acc: 0.9197 - val_loss: 0.3052 - val_acc: 0.8746\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.2050 - acc: 0.9181 - val_loss: 0.3063 - val_acc: 0.8776\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.1988 - acc: 0.9210 - val_loss: 0.3012 - val_acc: 0.8786\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.1909 - acc: 0.9244 - val_loss: 0.2997 - val_acc: 0.8770\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.1863 - acc: 0.9270 - val_loss: 0.3012 - val_acc: 0.8796\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.1792 - acc: 0.9325 - val_loss: 0.3041 - val_acc: 0.8762\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.1723 - acc: 0.9339 - val_loss: 0.3334 - val_acc: 0.8688\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.1685 - acc: 0.9364 - val_loss: 0.3075 - val_acc: 0.8778\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.1622 - acc: 0.9399 - val_loss: 0.3129 - val_acc: 0.8780\n",
      "va acc: 0.884\n",
      "te acc: 0.878\n",
      "2018-10-01 23:04:21.481002 save dbowd2v stack done!\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from datetime import datetime\n",
    "from sklearn.cross_validation import KFold\n",
    "from gensim.models import Doc2Vec\n",
    "from collections import OrderedDict\n",
    "\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Dropout, Activation\n",
    "from keras.optimizers import SGD, Adam, RMSprop\n",
    "from keras.utils import np_utils\n",
    "import re\n",
    "\n",
    "# ----------------------- myfunc -----------------------\n",
    "def myAcc(y_true,y_pred):\n",
    "    y_pred = np.argmax(y_pred,axis=1)\n",
    "    return np.mean(y_true == y_pred)\n",
    "\n",
    "# ----------------------- load dataset -----------------\n",
    "labeled_train = pd.read_csv('./data/labeledTrainData.tsv', sep='\\t', encoding='utf8')\n",
    "model = Doc2Vec.load('./data/dbow_d2v.model')\n",
    "X_sp = np.array([model.docvecs[i] for i in range(25000)])\n",
    "labels = labeled_train['sentiment']\n",
    "\n",
    "# ----------------------dbowd2v stack -------------------\n",
    "df_stack = pd.DataFrame(index=range(len(labeled_train)))\n",
    "TR = 20000\n",
    "n = 5\n",
    "\n",
    "X = X_sp[:TR]\n",
    "y = labels[:TR]\n",
    "X_te = X_sp[TR:]\n",
    "y_te = labels[TR:]\n",
    "\n",
    "num_class = len(pd.value_counts(labels))\n",
    "stack = np.zeros((X.shape[0],num_class))\n",
    "stack_te = np.zeros((X_te.shape[0],num_class))\n",
    "\n",
    "for k,(tr,va) in enumerate(KFold(len(y),n_folds=n)):\n",
    "    print('{} stack:{}/{}'.format(datetime.now(), k+1, n))\n",
    "    nb_classes = num_class\n",
    "    X_train = X[tr]\n",
    "    y_train = y[tr]\n",
    "    X_test = X_te\n",
    "    y_test = y_te\n",
    "    \n",
    "    X_train = X_train.astype('float32')\n",
    "    X_test = X_test.astype('float32')\n",
    "    Y_train = np_utils.to_categorical(y_train, nb_classes)\n",
    "    Y_test = np_utils.to_categorical(y_test, nb_classes)\n",
    "\n",
    "    model = Sequential()\n",
    "    model.add(Dense(300,input_shape=(X_train.shape[1],)))\n",
    "    model.add(Dropout(0.1))\n",
    "    model.add(Activation('tanh'))\n",
    "    model.add(Dense(nb_classes))\n",
    "    model.add(Activation('softmax'))\n",
    "\n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "                      optimizer='adadelta',\n",
    "                      metrics=['accuracy'])\n",
    "\n",
    "    history = model.fit(X_train, Y_train,shuffle=True,\n",
    "                            batch_size=128, nb_epoch=35,\n",
    "                            verbose=2, validation_data=(X_test, Y_test))\n",
    "    y_pred_va = model.predict_proba(X[va])\n",
    "    y_pred_te = model.predict_proba(X_te)\n",
    "    print('va acc:',myAcc(y[va],y_pred_va))\n",
    "    print('te acc:',myAcc(y_te,y_pred_te))\n",
    "    stack[va] += y_pred_va\n",
    "    stack_te += y_pred_te\n",
    "stack_te /= n\n",
    "stack_all = np.vstack([stack,stack_te])\n",
    "for l in range(stack_all.shape[1]):\n",
    "    df_stack['dbowd2v_{}'.format(l)] = stack_all[:,l]\n",
    "        \n",
    "df_stack.to_csv('./data/dbowd2v_stack.csv',encoding='utf8',index=None)\n",
    "print(datetime.now(),'save dbowd2v stack done!')        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01 23:08:32.398921 stack:1/5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:66: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3356 - acc: 0.8546 - val_loss: 0.3361 - val_acc: 0.8592\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.2896 - acc: 0.8794 - val_loss: 0.3058 - val_acc: 0.8736\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2856 - acc: 0.8789 - val_loss: 0.2992 - val_acc: 0.8778\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2819 - acc: 0.8830 - val_loss: 0.2971 - val_acc: 0.8794\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2805 - acc: 0.8826 - val_loss: 0.3046 - val_acc: 0.8736\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2773 - acc: 0.8834 - val_loss: 0.2942 - val_acc: 0.8804\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2753 - acc: 0.8844 - val_loss: 0.2960 - val_acc: 0.8810\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2730 - acc: 0.8862 - val_loss: 0.2950 - val_acc: 0.8818\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2712 - acc: 0.8881 - val_loss: 0.2957 - val_acc: 0.8816\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.2684 - acc: 0.8876 - val_loss: 0.3019 - val_acc: 0.8766\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.2678 - acc: 0.8886 - val_loss: 0.3053 - val_acc: 0.8728\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.2666 - acc: 0.8883 - val_loss: 0.2952 - val_acc: 0.8820\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.2629 - acc: 0.8902 - val_loss: 0.2993 - val_acc: 0.8792\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.2606 - acc: 0.8939 - val_loss: 0.2955 - val_acc: 0.8818\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.2588 - acc: 0.8907 - val_loss: 0.2972 - val_acc: 0.8816\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.2567 - acc: 0.8934 - val_loss: 0.2988 - val_acc: 0.8802\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.2538 - acc: 0.8929 - val_loss: 0.2968 - val_acc: 0.8796\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.2505 - acc: 0.8944 - val_loss: 0.2968 - val_acc: 0.8790\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.2483 - acc: 0.8974 - val_loss: 0.2968 - val_acc: 0.8804\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.2446 - acc: 0.8986 - val_loss: 0.3140 - val_acc: 0.8690\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.2405 - acc: 0.9013 - val_loss: 0.2949 - val_acc: 0.8818\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.2360 - acc: 0.9032 - val_loss: 0.2991 - val_acc: 0.8816\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.2310 - acc: 0.9053 - val_loss: 0.3002 - val_acc: 0.8818\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.2258 - acc: 0.9094 - val_loss: 0.2952 - val_acc: 0.8812\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.2213 - acc: 0.9097 - val_loss: 0.3001 - val_acc: 0.8800\n",
      "Epoch 26/35\n",
      " - 0s - loss: 0.2139 - acc: 0.9131 - val_loss: 0.3004 - val_acc: 0.8800\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.2090 - acc: 0.9160 - val_loss: 0.3008 - val_acc: 0.8796\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.2004 - acc: 0.9188 - val_loss: 0.3125 - val_acc: 0.8770\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.1974 - acc: 0.9224 - val_loss: 0.3053 - val_acc: 0.8802\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.1915 - acc: 0.9251 - val_loss: 0.3079 - val_acc: 0.8790\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.1845 - acc: 0.9276 - val_loss: 0.3102 - val_acc: 0.8754\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.1747 - acc: 0.9336 - val_loss: 0.3111 - val_acc: 0.8810\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.1704 - acc: 0.9326 - val_loss: 0.3347 - val_acc: 0.8726\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.1647 - acc: 0.9375 - val_loss: 0.3145 - val_acc: 0.8800\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.1554 - acc: 0.9400 - val_loss: 0.3201 - val_acc: 0.8822\n",
      "va acc: 0.8855\n",
      "te acc: 0.8822\n",
      "2018-10-01 23:08:46.041414 stack:2/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3360 - acc: 0.8559 - val_loss: 0.3009 - val_acc: 0.8732\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.2914 - acc: 0.8797 - val_loss: 0.3010 - val_acc: 0.8784\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2851 - acc: 0.8825 - val_loss: 0.3035 - val_acc: 0.8742\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2822 - acc: 0.8841 - val_loss: 0.2995 - val_acc: 0.8784\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2784 - acc: 0.8853 - val_loss: 0.2997 - val_acc: 0.8768\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2777 - acc: 0.8866 - val_loss: 0.3002 - val_acc: 0.8746\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2761 - acc: 0.8861 - val_loss: 0.2995 - val_acc: 0.8766\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2746 - acc: 0.8862 - val_loss: 0.3049 - val_acc: 0.8740\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2723 - acc: 0.8897 - val_loss: 0.3019 - val_acc: 0.8744\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.2703 - acc: 0.8881 - val_loss: 0.2991 - val_acc: 0.8764\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.2698 - acc: 0.8881 - val_loss: 0.2982 - val_acc: 0.8764\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.2681 - acc: 0.8898 - val_loss: 0.2992 - val_acc: 0.8770\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.2655 - acc: 0.8906 - val_loss: 0.3050 - val_acc: 0.8752\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.2633 - acc: 0.8922 - val_loss: 0.3024 - val_acc: 0.8748\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.2614 - acc: 0.8920 - val_loss: 0.3009 - val_acc: 0.8762\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.2599 - acc: 0.8939 - val_loss: 0.2993 - val_acc: 0.8786\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.2576 - acc: 0.8927 - val_loss: 0.3059 - val_acc: 0.8738\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.2538 - acc: 0.8962 - val_loss: 0.2982 - val_acc: 0.8784\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.2514 - acc: 0.8965 - val_loss: 0.3025 - val_acc: 0.8778\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.2483 - acc: 0.8988 - val_loss: 0.2997 - val_acc: 0.8784\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.2446 - acc: 0.8996 - val_loss: 0.3011 - val_acc: 0.8764\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.2411 - acc: 0.8994 - val_loss: 0.3031 - val_acc: 0.8764\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.2338 - acc: 0.9062 - val_loss: 0.3016 - val_acc: 0.8764\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.2322 - acc: 0.9048 - val_loss: 0.3064 - val_acc: 0.8766\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.2260 - acc: 0.9086 - val_loss: 0.3050 - val_acc: 0.8770\n",
      "Epoch 26/35\n",
      " - 0s - loss: 0.2226 - acc: 0.9112 - val_loss: 0.3089 - val_acc: 0.8748\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.2170 - acc: 0.9122 - val_loss: 0.3184 - val_acc: 0.8710\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.2115 - acc: 0.9163 - val_loss: 0.3072 - val_acc: 0.8754\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.2051 - acc: 0.9215 - val_loss: 0.3100 - val_acc: 0.8764\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.1974 - acc: 0.9248 - val_loss: 0.3071 - val_acc: 0.8770\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.1923 - acc: 0.9246 - val_loss: 0.3101 - val_acc: 0.8790\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.1855 - acc: 0.9298 - val_loss: 0.3186 - val_acc: 0.8762\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.1821 - acc: 0.9337 - val_loss: 0.3158 - val_acc: 0.8782\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.1754 - acc: 0.9346 - val_loss: 0.3219 - val_acc: 0.8774\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.1681 - acc: 0.9369 - val_loss: 0.3209 - val_acc: 0.8762\n",
      "va acc: 0.87675\n",
      "te acc: 0.8762\n",
      "2018-10-01 23:08:59.948423 stack:3/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3392 - acc: 0.8529 - val_loss: 0.2943 - val_acc: 0.8742\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.2848 - acc: 0.8803 - val_loss: 0.2954 - val_acc: 0.8776\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2820 - acc: 0.8834 - val_loss: 0.2993 - val_acc: 0.8742\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2793 - acc: 0.8813 - val_loss: 0.2988 - val_acc: 0.8746\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2754 - acc: 0.8848 - val_loss: 0.2964 - val_acc: 0.8776\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2737 - acc: 0.8835 - val_loss: 0.2944 - val_acc: 0.8782\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2715 - acc: 0.8864 - val_loss: 0.3032 - val_acc: 0.8746\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2704 - acc: 0.8874 - val_loss: 0.2941 - val_acc: 0.8762\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2672 - acc: 0.8875 - val_loss: 0.3011 - val_acc: 0.8762\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.2662 - acc: 0.8888 - val_loss: 0.2940 - val_acc: 0.8750\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.2642 - acc: 0.8904 - val_loss: 0.2941 - val_acc: 0.8782\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.2643 - acc: 0.8894 - val_loss: 0.2942 - val_acc: 0.8774\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.2609 - acc: 0.8907 - val_loss: 0.2993 - val_acc: 0.8740\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.2584 - acc: 0.8935 - val_loss: 0.2986 - val_acc: 0.8760\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.2580 - acc: 0.8934 - val_loss: 0.2995 - val_acc: 0.8776\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.2544 - acc: 0.8942 - val_loss: 0.2940 - val_acc: 0.8774\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.2521 - acc: 0.8942 - val_loss: 0.3003 - val_acc: 0.8786\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.2480 - acc: 0.8972 - val_loss: 0.2994 - val_acc: 0.8764\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.2459 - acc: 0.8987 - val_loss: 0.2993 - val_acc: 0.8782\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.2440 - acc: 0.8984 - val_loss: 0.2962 - val_acc: 0.8788\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.2393 - acc: 0.9020 - val_loss: 0.3060 - val_acc: 0.8760\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.2367 - acc: 0.9026 - val_loss: 0.2986 - val_acc: 0.8780\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.2303 - acc: 0.9070 - val_loss: 0.2990 - val_acc: 0.8794\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.2271 - acc: 0.9090 - val_loss: 0.2994 - val_acc: 0.8780\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.2189 - acc: 0.9105 - val_loss: 0.3039 - val_acc: 0.8762\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 26/35\n",
      " - 0s - loss: 0.2161 - acc: 0.9134 - val_loss: 0.3043 - val_acc: 0.8752\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.2107 - acc: 0.9167 - val_loss: 0.3144 - val_acc: 0.8720\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.2037 - acc: 0.9194 - val_loss: 0.3052 - val_acc: 0.8764\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.1972 - acc: 0.9233 - val_loss: 0.3050 - val_acc: 0.8776\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.1886 - acc: 0.9257 - val_loss: 0.3106 - val_acc: 0.8784\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.1869 - acc: 0.9286 - val_loss: 0.3108 - val_acc: 0.8788\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.1790 - acc: 0.9324 - val_loss: 0.3178 - val_acc: 0.8742\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.1705 - acc: 0.9369 - val_loss: 0.3206 - val_acc: 0.8774\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.1676 - acc: 0.9364 - val_loss: 0.3439 - val_acc: 0.8686\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.1597 - acc: 0.9401 - val_loss: 0.3276 - val_acc: 0.8752\n",
      "va acc: 0.8745\n",
      "te acc: 0.8752\n",
      "2018-10-01 23:09:14.120132 stack:4/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3280 - acc: 0.8631 - val_loss: 0.3070 - val_acc: 0.8738\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.2865 - acc: 0.8806 - val_loss: 0.3141 - val_acc: 0.8686\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2834 - acc: 0.8819 - val_loss: 0.2969 - val_acc: 0.8752\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2792 - acc: 0.8839 - val_loss: 0.2989 - val_acc: 0.8770\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2774 - acc: 0.8834 - val_loss: 0.2978 - val_acc: 0.8770\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2762 - acc: 0.8852 - val_loss: 0.3020 - val_acc: 0.8736\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2723 - acc: 0.8845 - val_loss: 0.2965 - val_acc: 0.8782\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2700 - acc: 0.8866 - val_loss: 0.2954 - val_acc: 0.8780\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2700 - acc: 0.8882 - val_loss: 0.2957 - val_acc: 0.8794\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.2680 - acc: 0.8891 - val_loss: 0.2999 - val_acc: 0.8758\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.2675 - acc: 0.8881 - val_loss: 0.2963 - val_acc: 0.8800\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.2644 - acc: 0.8889 - val_loss: 0.2943 - val_acc: 0.8802\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.2634 - acc: 0.8907 - val_loss: 0.2952 - val_acc: 0.8766\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.2591 - acc: 0.8918 - val_loss: 0.2954 - val_acc: 0.8784\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.2579 - acc: 0.8913 - val_loss: 0.2948 - val_acc: 0.8794\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.2560 - acc: 0.8937 - val_loss: 0.2950 - val_acc: 0.8794\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.2533 - acc: 0.8934 - val_loss: 0.2940 - val_acc: 0.8794\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.2506 - acc: 0.8957 - val_loss: 0.2979 - val_acc: 0.8776\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.2458 - acc: 0.8979 - val_loss: 0.2920 - val_acc: 0.8800\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.2433 - acc: 0.8992 - val_loss: 0.2960 - val_acc: 0.8792\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.2389 - acc: 0.9006 - val_loss: 0.2981 - val_acc: 0.8778\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.2337 - acc: 0.9022 - val_loss: 0.2943 - val_acc: 0.8808\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.2306 - acc: 0.9052 - val_loss: 0.2956 - val_acc: 0.8804\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.2253 - acc: 0.9087 - val_loss: 0.2956 - val_acc: 0.8834\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.2191 - acc: 0.9109 - val_loss: 0.3057 - val_acc: 0.8778\n",
      "Epoch 26/35\n",
      " - 0s - loss: 0.2152 - acc: 0.9134 - val_loss: 0.2942 - val_acc: 0.8826\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.2097 - acc: 0.9163 - val_loss: 0.2997 - val_acc: 0.8786\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.2034 - acc: 0.9200 - val_loss: 0.2992 - val_acc: 0.8806\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.1964 - acc: 0.9221 - val_loss: 0.3010 - val_acc: 0.8786\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.1882 - acc: 0.9276 - val_loss: 0.3065 - val_acc: 0.8780\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.1839 - acc: 0.9296 - val_loss: 0.3038 - val_acc: 0.8778\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.1784 - acc: 0.9309 - val_loss: 0.3064 - val_acc: 0.8810\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.1703 - acc: 0.9368 - val_loss: 0.3090 - val_acc: 0.8828\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.1655 - acc: 0.9370 - val_loss: 0.3306 - val_acc: 0.8716\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.1563 - acc: 0.9429 - val_loss: 0.3141 - val_acc: 0.8766\n",
      "va acc: 0.8815\n",
      "te acc: 0.8766\n",
      "2018-10-01 23:09:28.136608 stack:5/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3377 - acc: 0.8576 - val_loss: 0.2970 - val_acc: 0.8764\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.2895 - acc: 0.8801 - val_loss: 0.2989 - val_acc: 0.8770\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2859 - acc: 0.8809 - val_loss: 0.2954 - val_acc: 0.8786\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2815 - acc: 0.8818 - val_loss: 0.3028 - val_acc: 0.8742\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2793 - acc: 0.8838 - val_loss: 0.2976 - val_acc: 0.8790\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2778 - acc: 0.8838 - val_loss: 0.2964 - val_acc: 0.8794\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2765 - acc: 0.8852 - val_loss: 0.2945 - val_acc: 0.8806\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2744 - acc: 0.8861 - val_loss: 0.2953 - val_acc: 0.8778\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2732 - acc: 0.8841 - val_loss: 0.2966 - val_acc: 0.8776\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.2712 - acc: 0.8859 - val_loss: 0.2942 - val_acc: 0.8804\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.2698 - acc: 0.8873 - val_loss: 0.2959 - val_acc: 0.8798\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.2677 - acc: 0.8870 - val_loss: 0.2957 - val_acc: 0.8802\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.2659 - acc: 0.8889 - val_loss: 0.2928 - val_acc: 0.8804\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.2649 - acc: 0.8889 - val_loss: 0.2921 - val_acc: 0.8824\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.2621 - acc: 0.8892 - val_loss: 0.2948 - val_acc: 0.8796\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.2597 - acc: 0.8928 - val_loss: 0.2989 - val_acc: 0.8784\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.2573 - acc: 0.8917 - val_loss: 0.2935 - val_acc: 0.8794\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.2539 - acc: 0.8965 - val_loss: 0.2957 - val_acc: 0.8796\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.2510 - acc: 0.8948 - val_loss: 0.2954 - val_acc: 0.8808\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.2493 - acc: 0.8956 - val_loss: 0.2970 - val_acc: 0.8824\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.2446 - acc: 0.8997 - val_loss: 0.2978 - val_acc: 0.8780\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.2415 - acc: 0.9015 - val_loss: 0.2967 - val_acc: 0.8802\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.2369 - acc: 0.9022 - val_loss: 0.2987 - val_acc: 0.8820\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.2333 - acc: 0.9029 - val_loss: 0.2947 - val_acc: 0.8832\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.2267 - acc: 0.9087 - val_loss: 0.2953 - val_acc: 0.8816\n",
      "Epoch 26/35\n",
      " - 0s - loss: 0.2243 - acc: 0.9093 - val_loss: 0.2949 - val_acc: 0.8812\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.2175 - acc: 0.9135 - val_loss: 0.2972 - val_acc: 0.8812\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.2118 - acc: 0.9157 - val_loss: 0.2985 - val_acc: 0.8810\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.2053 - acc: 0.9190 - val_loss: 0.3161 - val_acc: 0.8738\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.2033 - acc: 0.9207 - val_loss: 0.3038 - val_acc: 0.8804\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.1955 - acc: 0.9195 - val_loss: 0.3075 - val_acc: 0.8790\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.1853 - acc: 0.9278 - val_loss: 0.3138 - val_acc: 0.8764\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.1803 - acc: 0.9308 - val_loss: 0.3046 - val_acc: 0.8808\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.1743 - acc: 0.9324 - val_loss: 0.3272 - val_acc: 0.8740\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.1688 - acc: 0.9349 - val_loss: 0.3166 - val_acc: 0.8822\n",
      "va acc: 0.88925\n",
      "te acc: 0.8822\n",
      "2018-10-01 23:09:41.748909 save dbowd2v stack done!\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from datetime import datetime\n",
    "from sklearn.cross_validation import KFold\n",
    "from gensim.models import Doc2Vec\n",
    "from collections import OrderedDict\n",
    "\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Dropout, Activation\n",
    "from keras.optimizers import SGD, Adam, RMSprop\n",
    "from keras.utils import np_utils\n",
    "import re\n",
    "\n",
    "# ----------------------- myfunc -----------------------\n",
    "def myAcc(y_true,y_pred):\n",
    "    y_pred = np.argmax(y_pred,axis=1)\n",
    "    return np.mean(y_true == y_pred)\n",
    "\n",
    "# ----------------------- load dataset -----------------\n",
    "labeled_train = pd.read_csv('./data/labeledTrainData.tsv', sep='\\t', encoding='utf8')\n",
    "model = Doc2Vec.load('./data/dbow_d2v_2.model')\n",
    "X_sp = np.array([model.docvecs[i] for i in range(25000)])\n",
    "labels = labeled_train['sentiment']\n",
    "\n",
    "# ----------------------dbowd2v stack -------------------\n",
    "df_stack = pd.DataFrame(index=range(len(labeled_train)))\n",
    "TR = 20000\n",
    "n = 5\n",
    "\n",
    "X = X_sp[:TR]\n",
    "y = labels[:TR]\n",
    "X_te = X_sp[TR:]\n",
    "y_te = labels[TR:]\n",
    "\n",
    "num_class = len(pd.value_counts(labels))\n",
    "stack = np.zeros((X.shape[0],num_class))\n",
    "stack_te = np.zeros((X_te.shape[0],num_class))\n",
    "\n",
    "for k,(tr,va) in enumerate(KFold(len(y),n_folds=n)):\n",
    "    print('{} stack:{}/{}'.format(datetime.now(), k+1, n))\n",
    "    nb_classes = num_class\n",
    "    X_train = X[tr]\n",
    "    y_train = y[tr]\n",
    "    X_test = X_te\n",
    "    y_test = y_te\n",
    "    \n",
    "    X_train = X_train.astype('float32')\n",
    "    X_test = X_test.astype('float32')\n",
    "    Y_train = np_utils.to_categorical(y_train, nb_classes)\n",
    "    Y_test = np_utils.to_categorical(y_test, nb_classes)\n",
    "\n",
    "    model = Sequential()\n",
    "    model.add(Dense(300,input_shape=(X_train.shape[1],)))\n",
    "    model.add(Dropout(0.1))\n",
    "    model.add(Activation('tanh'))\n",
    "    model.add(Dense(nb_classes))\n",
    "    model.add(Activation('softmax'))\n",
    "\n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "                      optimizer='adadelta',\n",
    "                      metrics=['accuracy'])\n",
    "\n",
    "    history = model.fit(X_train, Y_train,shuffle=True,\n",
    "                            batch_size=128, nb_epoch=35,\n",
    "                            verbose=2, validation_data=(X_test, Y_test))\n",
    "    y_pred_va = model.predict_proba(X[va])\n",
    "    y_pred_te = model.predict_proba(X_te)\n",
    "    print('va acc:',myAcc(y[va],y_pred_va))\n",
    "    print('te acc:',myAcc(y_te,y_pred_te))\n",
    "    stack[va] += y_pred_va\n",
    "    stack_te += y_pred_te\n",
    "stack_te /= n\n",
    "stack_all = np.vstack([stack,stack_te])\n",
    "for l in range(stack_all.shape[1]):\n",
    "    df_stack['dbowd2v_2_{}'.format(l)] = stack_all[:,l]\n",
    "        \n",
    "df_stack.to_csv('./data/dbowd2v_stack_2.csv',encoding='utf8',index=None)\n",
    "print(datetime.now(),'save dbowd2v stack done!')        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01 23:13:51.531822 stack:1/5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:66: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3700 - acc: 0.8348 - val_loss: 0.3387 - val_acc: 0.8568\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.3183 - acc: 0.8671 - val_loss: 0.3347 - val_acc: 0.8594\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2994 - acc: 0.8737 - val_loss: 0.3407 - val_acc: 0.8592\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2872 - acc: 0.8799 - val_loss: 0.3659 - val_acc: 0.8434\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2730 - acc: 0.8855 - val_loss: 0.3500 - val_acc: 0.8558\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2599 - acc: 0.8918 - val_loss: 0.3506 - val_acc: 0.8574\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2457 - acc: 0.8992 - val_loss: 0.3408 - val_acc: 0.8632\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2280 - acc: 0.9077 - val_loss: 0.3479 - val_acc: 0.8596\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2125 - acc: 0.9161 - val_loss: 0.3466 - val_acc: 0.8590\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.1951 - acc: 0.9240 - val_loss: 0.3651 - val_acc: 0.8574\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.1798 - acc: 0.9291 - val_loss: 0.3685 - val_acc: 0.8540\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.1565 - acc: 0.9430 - val_loss: 0.4110 - val_acc: 0.8438\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.1517 - acc: 0.9426 - val_loss: 0.3627 - val_acc: 0.8592\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.1319 - acc: 0.9533 - val_loss: 0.3880 - val_acc: 0.8558\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.1179 - acc: 0.9585 - val_loss: 0.3853 - val_acc: 0.8604\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.1086 - acc: 0.9619 - val_loss: 0.3925 - val_acc: 0.8596\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.1003 - acc: 0.9637 - val_loss: 0.4755 - val_acc: 0.8396\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.0913 - acc: 0.9675 - val_loss: 0.4118 - val_acc: 0.8602\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.0801 - acc: 0.9739 - val_loss: 0.4185 - val_acc: 0.8602\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.0713 - acc: 0.9769 - val_loss: 0.4156 - val_acc: 0.8606\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.0665 - acc: 0.9789 - val_loss: 0.5131 - val_acc: 0.8438\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.0572 - acc: 0.9822 - val_loss: 0.4571 - val_acc: 0.8612\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.0528 - acc: 0.9823 - val_loss: 0.4722 - val_acc: 0.8542\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.0481 - acc: 0.9851 - val_loss: 0.4918 - val_acc: 0.8522\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.0425 - acc: 0.9874 - val_loss: 0.4929 - val_acc: 0.8598\n",
      "Epoch 26/35\n",
      " - 0s - loss: 0.0403 - acc: 0.9879 - val_loss: 0.5657 - val_acc: 0.8402\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.0357 - acc: 0.9891 - val_loss: 0.4960 - val_acc: 0.8592\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.0338 - acc: 0.9892 - val_loss: 0.5161 - val_acc: 0.8594\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.0280 - acc: 0.9926 - val_loss: 0.5249 - val_acc: 0.8588\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.0288 - acc: 0.9914 - val_loss: 0.5372 - val_acc: 0.8588\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.0267 - acc: 0.9929 - val_loss: 0.5713 - val_acc: 0.8530\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.0227 - acc: 0.9938 - val_loss: 0.5736 - val_acc: 0.8546\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.0236 - acc: 0.9938 - val_loss: 0.5886 - val_acc: 0.8554\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.0199 - acc: 0.9945 - val_loss: 0.5724 - val_acc: 0.8544\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.0184 - acc: 0.9955 - val_loss: 0.6384 - val_acc: 0.8494\n",
      "va acc: 0.8395\n",
      "te acc: 0.8494\n",
      "2018-10-01 23:14:05.562954 stack:2/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3653 - acc: 0.8353 - val_loss: 0.3495 - val_acc: 0.8558\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.3154 - acc: 0.8666 - val_loss: 0.3407 - val_acc: 0.8562\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.3007 - acc: 0.8745 - val_loss: 0.3380 - val_acc: 0.8588\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2828 - acc: 0.8801 - val_loss: 0.3339 - val_acc: 0.8626\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2683 - acc: 0.8875 - val_loss: 0.3493 - val_acc: 0.8542\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2554 - acc: 0.8939 - val_loss: 0.3447 - val_acc: 0.8580\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2461 - acc: 0.8978 - val_loss: 0.3407 - val_acc: 0.8594\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2279 - acc: 0.9091 - val_loss: 0.3454 - val_acc: 0.8592\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2090 - acc: 0.9153 - val_loss: 0.3729 - val_acc: 0.8496\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.1907 - acc: 0.9239 - val_loss: 0.3641 - val_acc: 0.8560\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.1765 - acc: 0.9321 - val_loss: 0.3637 - val_acc: 0.8592\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.1541 - acc: 0.9447 - val_loss: 0.3648 - val_acc: 0.8590\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.1483 - acc: 0.9425 - val_loss: 0.3754 - val_acc: 0.8586\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.1317 - acc: 0.9529 - val_loss: 0.3861 - val_acc: 0.8588\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.1233 - acc: 0.9539 - val_loss: 0.3897 - val_acc: 0.8568\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.1082 - acc: 0.9629 - val_loss: 0.4018 - val_acc: 0.8612\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.0918 - acc: 0.9691 - val_loss: 0.4077 - val_acc: 0.8564\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.0904 - acc: 0.9693 - val_loss: 0.4063 - val_acc: 0.8604\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.0792 - acc: 0.9728 - val_loss: 0.4815 - val_acc: 0.8440\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.0732 - acc: 0.9753 - val_loss: 0.4378 - val_acc: 0.8586\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.0669 - acc: 0.9780 - val_loss: 0.4458 - val_acc: 0.8574\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.0567 - acc: 0.9822 - val_loss: 0.4945 - val_acc: 0.8562\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.0579 - acc: 0.9802 - val_loss: 0.4779 - val_acc: 0.8554\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.0475 - acc: 0.9854 - val_loss: 0.4708 - val_acc: 0.8632\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.0434 - acc: 0.9870 - val_loss: 0.4825 - val_acc: 0.8570\n",
      "Epoch 26/35\n",
      " - 0s - loss: 0.0407 - acc: 0.9876 - val_loss: 0.5105 - val_acc: 0.8592\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.0399 - acc: 0.9876 - val_loss: 0.5147 - val_acc: 0.8598\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.0359 - acc: 0.9899 - val_loss: 0.5273 - val_acc: 0.8614\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.0287 - acc: 0.9920 - val_loss: 0.5179 - val_acc: 0.8606\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.0311 - acc: 0.9903 - val_loss: 0.5524 - val_acc: 0.8556\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.0268 - acc: 0.9929 - val_loss: 0.6247 - val_acc: 0.8414\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.0225 - acc: 0.9948 - val_loss: 0.5700 - val_acc: 0.8588\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.0256 - acc: 0.9928 - val_loss: 0.5560 - val_acc: 0.8588\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.0193 - acc: 0.9952 - val_loss: 0.5890 - val_acc: 0.8540\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.0193 - acc: 0.9946 - val_loss: 0.6003 - val_acc: 0.8556\n",
      "va acc: 0.85975\n",
      "te acc: 0.8556\n",
      "2018-10-01 23:14:19.798728 stack:3/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3699 - acc: 0.8364 - val_loss: 0.3558 - val_acc: 0.8520\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.3140 - acc: 0.8676 - val_loss: 0.3364 - val_acc: 0.8610\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2949 - acc: 0.8747 - val_loss: 0.3563 - val_acc: 0.8504\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2804 - acc: 0.8814 - val_loss: 0.3394 - val_acc: 0.8604\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2670 - acc: 0.8906 - val_loss: 0.3530 - val_acc: 0.8548\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2537 - acc: 0.8942 - val_loss: 0.3462 - val_acc: 0.8574\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2387 - acc: 0.9016 - val_loss: 0.3850 - val_acc: 0.8430\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2207 - acc: 0.9128 - val_loss: 0.3529 - val_acc: 0.8584\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2041 - acc: 0.9189 - val_loss: 0.3628 - val_acc: 0.8556\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.1839 - acc: 0.9285 - val_loss: 0.3638 - val_acc: 0.8582\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.1650 - acc: 0.9383 - val_loss: 0.3813 - val_acc: 0.8538\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.1534 - acc: 0.9420 - val_loss: 0.3900 - val_acc: 0.8514\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.1366 - acc: 0.9521 - val_loss: 0.3878 - val_acc: 0.8546\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.1289 - acc: 0.9553 - val_loss: 0.4159 - val_acc: 0.8486\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.1085 - acc: 0.9628 - val_loss: 0.4171 - val_acc: 0.8462\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.1025 - acc: 0.9646 - val_loss: 0.4093 - val_acc: 0.8570\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.0869 - acc: 0.9712 - val_loss: 0.4309 - val_acc: 0.8578\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.0811 - acc: 0.9730 - val_loss: 0.4779 - val_acc: 0.8446\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.0790 - acc: 0.9723 - val_loss: 0.4350 - val_acc: 0.8590\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.0632 - acc: 0.9801 - val_loss: 0.4429 - val_acc: 0.8578\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.0617 - acc: 0.9799 - val_loss: 0.5078 - val_acc: 0.8458\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.0516 - acc: 0.9844 - val_loss: 0.4740 - val_acc: 0.8568\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.0440 - acc: 0.9876 - val_loss: 0.4823 - val_acc: 0.8600\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.0399 - acc: 0.9888 - val_loss: 0.5053 - val_acc: 0.8588\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.0441 - acc: 0.9858 - val_loss: 0.4996 - val_acc: 0.8622\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 26/35\n",
      " - 0s - loss: 0.0353 - acc: 0.9890 - val_loss: 0.5152 - val_acc: 0.8618\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.0306 - acc: 0.9924 - val_loss: 0.5460 - val_acc: 0.8528\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.0310 - acc: 0.9909 - val_loss: 0.5490 - val_acc: 0.8542\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.0261 - acc: 0.9929 - val_loss: 0.5365 - val_acc: 0.8602\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.0223 - acc: 0.9946 - val_loss: 0.5821 - val_acc: 0.8566\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.0235 - acc: 0.9930 - val_loss: 0.6046 - val_acc: 0.8556\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.0195 - acc: 0.9949 - val_loss: 0.5684 - val_acc: 0.8598\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.0175 - acc: 0.9954 - val_loss: 0.5913 - val_acc: 0.8606\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.0178 - acc: 0.9951 - val_loss: 0.6117 - val_acc: 0.8528\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.0147 - acc: 0.9958 - val_loss: 0.6080 - val_acc: 0.8594\n",
      "va acc: 0.84925\n",
      "te acc: 0.8594\n",
      "2018-10-01 23:14:33.989746 stack:4/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3729 - acc: 0.8369 - val_loss: 0.3393 - val_acc: 0.8594\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.3160 - acc: 0.8640 - val_loss: 0.3397 - val_acc: 0.8574\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2964 - acc: 0.8754 - val_loss: 0.3476 - val_acc: 0.8500\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2800 - acc: 0.8806 - val_loss: 0.3455 - val_acc: 0.8596\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2668 - acc: 0.8876 - val_loss: 0.3547 - val_acc: 0.8570\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2500 - acc: 0.8976 - val_loss: 0.3516 - val_acc: 0.8542\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2366 - acc: 0.9038 - val_loss: 0.3475 - val_acc: 0.8550\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2212 - acc: 0.9099 - val_loss: 0.3625 - val_acc: 0.8566\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2005 - acc: 0.9193 - val_loss: 0.3644 - val_acc: 0.8560\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.1842 - acc: 0.9281 - val_loss: 0.3704 - val_acc: 0.8550\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.1715 - acc: 0.9335 - val_loss: 0.3795 - val_acc: 0.8550\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.1564 - acc: 0.9418 - val_loss: 0.3864 - val_acc: 0.8530\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.1381 - acc: 0.9490 - val_loss: 0.3744 - val_acc: 0.8538\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.1255 - acc: 0.9544 - val_loss: 0.3923 - val_acc: 0.8572\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.1152 - acc: 0.9596 - val_loss: 0.4070 - val_acc: 0.8542\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.0975 - acc: 0.9663 - val_loss: 0.4009 - val_acc: 0.8584\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.0914 - acc: 0.9685 - val_loss: 0.4373 - val_acc: 0.8546\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.0761 - acc: 0.9760 - val_loss: 0.4624 - val_acc: 0.8444\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.0708 - acc: 0.9771 - val_loss: 0.4407 - val_acc: 0.8526\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.0669 - acc: 0.9788 - val_loss: 0.4556 - val_acc: 0.8558\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.0575 - acc: 0.9819 - val_loss: 0.4585 - val_acc: 0.8576\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.0547 - acc: 0.9838 - val_loss: 0.4609 - val_acc: 0.8594\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.0467 - acc: 0.9859 - val_loss: 0.5573 - val_acc: 0.8412\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.0450 - acc: 0.9871 - val_loss: 0.4919 - val_acc: 0.8558\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.0394 - acc: 0.9882 - val_loss: 0.6002 - val_acc: 0.8362\n",
      "Epoch 26/35\n",
      " - 0s - loss: 0.0347 - acc: 0.9911 - val_loss: 0.5172 - val_acc: 0.8530\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.0344 - acc: 0.9889 - val_loss: 0.5632 - val_acc: 0.8488\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.0298 - acc: 0.9921 - val_loss: 0.5498 - val_acc: 0.8550\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.0272 - acc: 0.9934 - val_loss: 0.5497 - val_acc: 0.8526\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.0252 - acc: 0.9931 - val_loss: 0.5746 - val_acc: 0.8532\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.0240 - acc: 0.9933 - val_loss: 0.5591 - val_acc: 0.8560\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.0234 - acc: 0.9933 - val_loss: 0.5679 - val_acc: 0.8558\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.0195 - acc: 0.9941 - val_loss: 0.5811 - val_acc: 0.8560\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.0188 - acc: 0.9953 - val_loss: 0.6198 - val_acc: 0.8538\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.0160 - acc: 0.9961 - val_loss: 0.6105 - val_acc: 0.8548\n",
      "va acc: 0.8625\n",
      "te acc: 0.8548\n",
      "2018-10-01 23:14:47.976345 stack:5/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/35\n",
      " - 1s - loss: 0.3688 - acc: 0.8326 - val_loss: 0.3364 - val_acc: 0.8596\n",
      "Epoch 2/35\n",
      " - 0s - loss: 0.3157 - acc: 0.8669 - val_loss: 0.3358 - val_acc: 0.8608\n",
      "Epoch 3/35\n",
      " - 0s - loss: 0.2982 - acc: 0.8752 - val_loss: 0.3359 - val_acc: 0.8614\n",
      "Epoch 4/35\n",
      " - 0s - loss: 0.2834 - acc: 0.8821 - val_loss: 0.3373 - val_acc: 0.8608\n",
      "Epoch 5/35\n",
      " - 0s - loss: 0.2718 - acc: 0.8858 - val_loss: 0.3439 - val_acc: 0.8570\n",
      "Epoch 6/35\n",
      " - 0s - loss: 0.2565 - acc: 0.8949 - val_loss: 0.3383 - val_acc: 0.8622\n",
      "Epoch 7/35\n",
      " - 0s - loss: 0.2391 - acc: 0.9022 - val_loss: 0.3577 - val_acc: 0.8562\n",
      "Epoch 8/35\n",
      " - 0s - loss: 0.2246 - acc: 0.9091 - val_loss: 0.3484 - val_acc: 0.8594\n",
      "Epoch 9/35\n",
      " - 0s - loss: 0.2068 - acc: 0.9173 - val_loss: 0.3850 - val_acc: 0.8442\n",
      "Epoch 10/35\n",
      " - 0s - loss: 0.1885 - acc: 0.9266 - val_loss: 0.3543 - val_acc: 0.8598\n",
      "Epoch 11/35\n",
      " - 0s - loss: 0.1716 - acc: 0.9356 - val_loss: 0.3611 - val_acc: 0.8598\n",
      "Epoch 12/35\n",
      " - 0s - loss: 0.1547 - acc: 0.9427 - val_loss: 0.3734 - val_acc: 0.8590\n",
      "Epoch 13/35\n",
      " - 0s - loss: 0.1404 - acc: 0.9481 - val_loss: 0.3730 - val_acc: 0.8606\n",
      "Epoch 14/35\n",
      " - 0s - loss: 0.1209 - acc: 0.9575 - val_loss: 0.3883 - val_acc: 0.8570\n",
      "Epoch 15/35\n",
      " - 0s - loss: 0.1107 - acc: 0.9611 - val_loss: 0.4045 - val_acc: 0.8566\n",
      "Epoch 16/35\n",
      " - 0s - loss: 0.1003 - acc: 0.9647 - val_loss: 0.4028 - val_acc: 0.8574\n",
      "Epoch 17/35\n",
      " - 0s - loss: 0.0921 - acc: 0.9686 - val_loss: 0.4136 - val_acc: 0.8610\n",
      "Epoch 18/35\n",
      " - 0s - loss: 0.0840 - acc: 0.9715 - val_loss: 0.4408 - val_acc: 0.8584\n",
      "Epoch 19/35\n",
      " - 0s - loss: 0.0717 - acc: 0.9772 - val_loss: 0.4891 - val_acc: 0.8416\n",
      "Epoch 20/35\n",
      " - 0s - loss: 0.0683 - acc: 0.9773 - val_loss: 0.4341 - val_acc: 0.8618\n",
      "Epoch 21/35\n",
      " - 0s - loss: 0.0591 - acc: 0.9814 - val_loss: 0.4632 - val_acc: 0.8588\n",
      "Epoch 22/35\n",
      " - 0s - loss: 0.0533 - acc: 0.9836 - val_loss: 0.5423 - val_acc: 0.8450\n",
      "Epoch 23/35\n",
      " - 0s - loss: 0.0550 - acc: 0.9814 - val_loss: 0.4688 - val_acc: 0.8590\n",
      "Epoch 24/35\n",
      " - 0s - loss: 0.0415 - acc: 0.9886 - val_loss: 0.4689 - val_acc: 0.8596\n",
      "Epoch 25/35\n",
      " - 0s - loss: 0.0371 - acc: 0.9896 - val_loss: 0.5140 - val_acc: 0.8508\n",
      "Epoch 26/35\n",
      " - 0s - loss: 0.0385 - acc: 0.9874 - val_loss: 0.5309 - val_acc: 0.8564\n",
      "Epoch 27/35\n",
      " - 0s - loss: 0.0322 - acc: 0.9913 - val_loss: 0.5035 - val_acc: 0.8606\n",
      "Epoch 28/35\n",
      " - 0s - loss: 0.0272 - acc: 0.9921 - val_loss: 0.5244 - val_acc: 0.8628\n",
      "Epoch 29/35\n",
      " - 0s - loss: 0.0293 - acc: 0.9904 - val_loss: 0.5580 - val_acc: 0.8548\n",
      "Epoch 30/35\n",
      " - 0s - loss: 0.0252 - acc: 0.9928 - val_loss: 0.5372 - val_acc: 0.8652\n",
      "Epoch 31/35\n",
      " - 0s - loss: 0.0206 - acc: 0.9944 - val_loss: 0.5586 - val_acc: 0.8620\n",
      "Epoch 32/35\n",
      " - 0s - loss: 0.0212 - acc: 0.9941 - val_loss: 0.6154 - val_acc: 0.8500\n",
      "Epoch 33/35\n",
      " - 0s - loss: 0.0174 - acc: 0.9966 - val_loss: 0.5978 - val_acc: 0.8578\n",
      "Epoch 34/35\n",
      " - 0s - loss: 0.0190 - acc: 0.9946 - val_loss: 0.5764 - val_acc: 0.8586\n",
      "Epoch 35/35\n",
      " - 0s - loss: 0.0152 - acc: 0.9964 - val_loss: 0.5955 - val_acc: 0.8636\n",
      "va acc: 0.86175\n",
      "te acc: 0.8636\n",
      "2018-10-01 23:15:02.175941 save dmd2v stack done!\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from datetime import datetime\n",
    "from sklearn.cross_validation import KFold\n",
    "from gensim.models import Doc2Vec\n",
    "from collections import OrderedDict\n",
    "\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Dropout, Activation\n",
    "from keras.optimizers import SGD, Adam, RMSprop\n",
    "from keras.utils import np_utils\n",
    "import re\n",
    "\n",
    "# ----------------------- myfunc -----------------------\n",
    "def myAcc(y_true,y_pred):\n",
    "    y_pred = np.argmax(y_pred,axis=1)\n",
    "    return np.mean(y_true == y_pred)\n",
    "\n",
    "# ----------------------- load dataset -----------------\n",
    "labeled_train = pd.read_csv('./data/labeledTrainData.tsv', sep='\\t', encoding='utf8')\n",
    "model = Doc2Vec.load('./data/dm_d2v.model')\n",
    "X_sp = np.array([model.docvecs[i] for i in range(25000)])\n",
    "labels = labeled_train['sentiment']\n",
    "\n",
    "# ----------------------dmd2v stack -------------------\n",
    "df_stack = pd.DataFrame(index=range(len(labeled_train)))\n",
    "TR = 20000\n",
    "n = 5\n",
    "\n",
    "X = X_sp[:TR]\n",
    "y = labels[:TR]\n",
    "X_te = X_sp[TR:]\n",
    "y_te = labels[TR:]\n",
    "\n",
    "num_class = len(pd.value_counts(labels))\n",
    "stack = np.zeros((X.shape[0],num_class))\n",
    "stack_te = np.zeros((X_te.shape[0],num_class))\n",
    "\n",
    "for k,(tr,va) in enumerate(KFold(len(y),n_folds=n)):\n",
    "    print('{} stack:{}/{}'.format(datetime.now(), k+1, n))\n",
    "    nb_classes = num_class\n",
    "    X_train = X[tr]\n",
    "    y_train = y[tr]\n",
    "    X_test = X_te\n",
    "    y_test = y_te\n",
    "    \n",
    "    X_train = X_train.astype('float32')\n",
    "    X_test = X_test.astype('float32')\n",
    "    Y_train = np_utils.to_categorical(y_train, nb_classes)\n",
    "    Y_test = np_utils.to_categorical(y_test, nb_classes)\n",
    "\n",
    "    model = Sequential()\n",
    "    model.add(Dense(300,input_shape=(X_train.shape[1],)))\n",
    "    model.add(Dropout(0.1))\n",
    "    model.add(Activation('tanh'))\n",
    "    model.add(Dense(nb_classes))\n",
    "    model.add(Activation('softmax'))\n",
    "\n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "                      optimizer='adadelta',\n",
    "                      metrics=['accuracy'])\n",
    "\n",
    "    history = model.fit(X_train, Y_train,shuffle=True,\n",
    "                            batch_size=128, nb_epoch=35,\n",
    "                            verbose=2, validation_data=(X_test, Y_test))\n",
    "    y_pred_va = model.predict_proba(X[va])\n",
    "    y_pred_te = model.predict_proba(X_te)\n",
    "    print('va acc:',myAcc(y[va],y_pred_va))\n",
    "    print('te acc:',myAcc(y_te,y_pred_te))\n",
    "    stack[va] += y_pred_va\n",
    "    stack_te += y_pred_te\n",
    "stack_te /= n\n",
    "stack_all = np.vstack([stack,stack_te])\n",
    "for l in range(stack_all.shape[1]):\n",
    "    df_stack['dmd2v_{}'.format(l)] = stack_all[:,l]\n",
    "        \n",
    "df_stack.to_csv('./data/dmd2v_stack.csv',encoding='utf8',index=None)\n",
    "print(datetime.now(),'save dmd2v stack done!')        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-10-01 23:21:47.005353 stack:1/5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\sun\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:66: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      " - 1s - loss: 0.3693 - acc: 0.8368 - val_loss: 0.3633 - val_acc: 0.8480\n",
      "Epoch 2/25\n",
      " - 0s - loss: 0.3057 - acc: 0.8709 - val_loss: 0.3474 - val_acc: 0.8526\n",
      "Epoch 3/25\n",
      " - 0s - loss: 0.2790 - acc: 0.8826 - val_loss: 0.3526 - val_acc: 0.8570\n",
      "Epoch 4/25\n",
      " - 0s - loss: 0.2554 - acc: 0.8939 - val_loss: 0.3453 - val_acc: 0.8602\n",
      "Epoch 5/25\n",
      " - 0s - loss: 0.2317 - acc: 0.9067 - val_loss: 0.3548 - val_acc: 0.8520\n",
      "Epoch 6/25\n",
      " - 0s - loss: 0.2057 - acc: 0.9157 - val_loss: 0.3680 - val_acc: 0.8544\n",
      "Epoch 7/25\n",
      " - 0s - loss: 0.1808 - acc: 0.9308 - val_loss: 0.3748 - val_acc: 0.8514\n",
      "Epoch 8/25\n",
      " - 0s - loss: 0.1545 - acc: 0.9442 - val_loss: 0.3790 - val_acc: 0.8594\n",
      "Epoch 9/25\n",
      " - 0s - loss: 0.1347 - acc: 0.9520 - val_loss: 0.3904 - val_acc: 0.8560\n",
      "Epoch 10/25\n",
      " - 0s - loss: 0.1121 - acc: 0.9626 - val_loss: 0.4123 - val_acc: 0.8520\n",
      "Epoch 11/25\n",
      " - 0s - loss: 0.0938 - acc: 0.9687 - val_loss: 0.4273 - val_acc: 0.8524\n",
      "Epoch 12/25\n",
      " - 0s - loss: 0.0808 - acc: 0.9747 - val_loss: 0.4353 - val_acc: 0.8598\n",
      "Epoch 13/25\n",
      " - 0s - loss: 0.0690 - acc: 0.9789 - val_loss: 0.4571 - val_acc: 0.8528\n",
      "Epoch 14/25\n",
      " - 0s - loss: 0.0607 - acc: 0.9811 - val_loss: 0.4730 - val_acc: 0.8536\n",
      "Epoch 15/25\n",
      " - 0s - loss: 0.0506 - acc: 0.9854 - val_loss: 0.4993 - val_acc: 0.8532\n",
      "Epoch 16/25\n",
      " - 0s - loss: 0.0415 - acc: 0.9879 - val_loss: 0.4910 - val_acc: 0.8596\n",
      "Epoch 17/25\n",
      " - 0s - loss: 0.0388 - acc: 0.9889 - val_loss: 0.5271 - val_acc: 0.8488\n",
      "Epoch 18/25\n",
      " - 0s - loss: 0.0316 - acc: 0.9919 - val_loss: 0.5899 - val_acc: 0.8452\n",
      "Epoch 19/25\n",
      " - 0s - loss: 0.0261 - acc: 0.9939 - val_loss: 0.5365 - val_acc: 0.8550\n",
      "Epoch 20/25\n",
      " - 0s - loss: 0.0250 - acc: 0.9936 - val_loss: 0.5654 - val_acc: 0.8528\n",
      "Epoch 21/25\n",
      " - 0s - loss: 0.0217 - acc: 0.9948 - val_loss: 0.6064 - val_acc: 0.8488\n",
      "Epoch 22/25\n",
      " - 0s - loss: 0.0182 - acc: 0.9959 - val_loss: 0.5783 - val_acc: 0.8546\n",
      "Epoch 23/25\n",
      " - 0s - loss: 0.0173 - acc: 0.9960 - val_loss: 0.6129 - val_acc: 0.8534\n",
      "Epoch 24/25\n",
      " - 0s - loss: 0.0148 - acc: 0.9966 - val_loss: 0.6105 - val_acc: 0.8480\n",
      "Epoch 25/25\n",
      " - 0s - loss: 0.0137 - acc: 0.9972 - val_loss: 0.6296 - val_acc: 0.8556\n",
      "va acc: 0.85\n",
      "te acc: 0.8556\n",
      "2018-10-01 23:21:57.649158 stack:2/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      " - 1s - loss: 0.3760 - acc: 0.8307 - val_loss: 0.3397 - val_acc: 0.8552\n",
      "Epoch 2/25\n",
      " - 0s - loss: 0.3121 - acc: 0.8670 - val_loss: 0.3387 - val_acc: 0.8570\n",
      "Epoch 3/25\n",
      " - 0s - loss: 0.2851 - acc: 0.8796 - val_loss: 0.3364 - val_acc: 0.8578\n",
      "Epoch 4/25\n",
      " - 0s - loss: 0.2615 - acc: 0.8918 - val_loss: 0.3560 - val_acc: 0.8502\n",
      "Epoch 5/25\n",
      " - 0s - loss: 0.2358 - acc: 0.9046 - val_loss: 0.3474 - val_acc: 0.8576\n",
      "Epoch 6/25\n",
      " - 0s - loss: 0.2133 - acc: 0.9159 - val_loss: 0.3573 - val_acc: 0.8570\n",
      "Epoch 7/25\n",
      " - 0s - loss: 0.1871 - acc: 0.9304 - val_loss: 0.4188 - val_acc: 0.8452\n",
      "Epoch 8/25\n",
      " - 0s - loss: 0.1629 - acc: 0.9386 - val_loss: 0.4180 - val_acc: 0.8412\n",
      "Epoch 9/25\n",
      " - 0s - loss: 0.1384 - acc: 0.9504 - val_loss: 0.3879 - val_acc: 0.8518\n",
      "Epoch 10/25\n",
      " - 0s - loss: 0.1192 - acc: 0.9586 - val_loss: 0.4790 - val_acc: 0.8394\n",
      "Epoch 11/25\n",
      " - 0s - loss: 0.1004 - acc: 0.9681 - val_loss: 0.4277 - val_acc: 0.8488\n",
      "Epoch 12/25\n",
      " - 0s - loss: 0.0828 - acc: 0.9726 - val_loss: 0.4640 - val_acc: 0.8474\n",
      "Epoch 13/25\n",
      " - 0s - loss: 0.0729 - acc: 0.9763 - val_loss: 0.4481 - val_acc: 0.8504\n",
      "Epoch 14/25\n",
      " - 0s - loss: 0.0617 - acc: 0.9818 - val_loss: 0.5047 - val_acc: 0.8494\n",
      "Epoch 15/25\n",
      " - 0s - loss: 0.0538 - acc: 0.9839 - val_loss: 0.4878 - val_acc: 0.8502\n",
      "Epoch 16/25\n",
      " - 0s - loss: 0.0462 - acc: 0.9867 - val_loss: 0.4853 - val_acc: 0.8540\n",
      "Epoch 17/25\n",
      " - 0s - loss: 0.0390 - acc: 0.9899 - val_loss: 0.5275 - val_acc: 0.8484\n",
      "Epoch 18/25\n",
      " - 0s - loss: 0.0343 - acc: 0.9909 - val_loss: 0.5171 - val_acc: 0.8504\n",
      "Epoch 19/25\n",
      " - 0s - loss: 0.0281 - acc: 0.9939 - val_loss: 0.5472 - val_acc: 0.8506\n",
      "Epoch 20/25\n",
      " - 0s - loss: 0.0259 - acc: 0.9931 - val_loss: 0.5517 - val_acc: 0.8498\n",
      "Epoch 21/25\n",
      " - 0s - loss: 0.0214 - acc: 0.9952 - val_loss: 0.5664 - val_acc: 0.8508\n",
      "Epoch 22/25\n",
      " - 0s - loss: 0.0183 - acc: 0.9957 - val_loss: 0.5887 - val_acc: 0.8488\n",
      "Epoch 23/25\n",
      " - 0s - loss: 0.0167 - acc: 0.9963 - val_loss: 0.5817 - val_acc: 0.8554\n",
      "Epoch 24/25\n",
      " - 0s - loss: 0.0171 - acc: 0.9957 - val_loss: 0.6191 - val_acc: 0.8458\n",
      "Epoch 25/25\n",
      " - 0s - loss: 0.0140 - acc: 0.9973 - val_loss: 0.6391 - val_acc: 0.8482\n",
      "va acc: 0.85675\n",
      "te acc: 0.8482\n",
      "2018-10-01 23:22:08.309621 stack:3/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      " - 1s - loss: 0.3695 - acc: 0.8341 - val_loss: 0.3396 - val_acc: 0.8578\n",
      "Epoch 2/25\n",
      " - 0s - loss: 0.2999 - acc: 0.8737 - val_loss: 0.3449 - val_acc: 0.8552\n",
      "Epoch 3/25\n",
      " - 0s - loss: 0.2757 - acc: 0.8870 - val_loss: 0.3486 - val_acc: 0.8592\n",
      "Epoch 4/25\n",
      " - 0s - loss: 0.2532 - acc: 0.8958 - val_loss: 0.3434 - val_acc: 0.8566\n",
      "Epoch 5/25\n",
      " - 0s - loss: 0.2303 - acc: 0.9091 - val_loss: 0.3526 - val_acc: 0.8578\n",
      "Epoch 6/25\n",
      " - 0s - loss: 0.2034 - acc: 0.9206 - val_loss: 0.3600 - val_acc: 0.8572\n",
      "Epoch 7/25\n",
      " - 0s - loss: 0.1792 - acc: 0.9327 - val_loss: 0.3795 - val_acc: 0.8534\n",
      "Epoch 8/25\n",
      " - 0s - loss: 0.1529 - acc: 0.9454 - val_loss: 0.3750 - val_acc: 0.8578\n",
      "Epoch 9/25\n",
      " - 0s - loss: 0.1340 - acc: 0.9514 - val_loss: 0.4101 - val_acc: 0.8498\n",
      "Epoch 10/25\n",
      " - 0s - loss: 0.1111 - acc: 0.9637 - val_loss: 0.4151 - val_acc: 0.8520\n",
      "Epoch 11/25\n",
      " - 0s - loss: 0.0990 - acc: 0.9666 - val_loss: 0.4223 - val_acc: 0.8522\n",
      "Epoch 12/25\n",
      " - 0s - loss: 0.0865 - acc: 0.9716 - val_loss: 0.4494 - val_acc: 0.8524\n",
      "Epoch 13/25\n",
      " - 0s - loss: 0.0670 - acc: 0.9801 - val_loss: 0.4571 - val_acc: 0.8482\n",
      "Epoch 14/25\n",
      " - 0s - loss: 0.0587 - acc: 0.9823 - val_loss: 0.5249 - val_acc: 0.8414\n",
      "Epoch 15/25\n",
      " - 0s - loss: 0.0527 - acc: 0.9849 - val_loss: 0.4871 - val_acc: 0.8546\n",
      "Epoch 16/25\n",
      " - 0s - loss: 0.0410 - acc: 0.9897 - val_loss: 0.4914 - val_acc: 0.8546\n",
      "Epoch 17/25\n",
      " - 0s - loss: 0.0368 - acc: 0.9900 - val_loss: 0.5357 - val_acc: 0.8458\n",
      "Epoch 18/25\n",
      " - 0s - loss: 0.0339 - acc: 0.9912 - val_loss: 0.5409 - val_acc: 0.8548\n",
      "Epoch 19/25\n",
      " - 0s - loss: 0.0273 - acc: 0.9934 - val_loss: 0.5567 - val_acc: 0.8514\n",
      "Epoch 20/25\n",
      " - 0s - loss: 0.0222 - acc: 0.9946 - val_loss: 0.5670 - val_acc: 0.8558\n",
      "Epoch 21/25\n",
      " - 0s - loss: 0.0228 - acc: 0.9939 - val_loss: 0.5758 - val_acc: 0.8518\n",
      "Epoch 22/25\n",
      " - 0s - loss: 0.0189 - acc: 0.9952 - val_loss: 0.6452 - val_acc: 0.8452\n",
      "Epoch 23/25\n",
      " - 0s - loss: 0.0171 - acc: 0.9954 - val_loss: 0.5938 - val_acc: 0.8510\n",
      "Epoch 24/25\n",
      " - 0s - loss: 0.0151 - acc: 0.9964 - val_loss: 0.6398 - val_acc: 0.8508\n",
      "Epoch 25/25\n",
      " - 0s - loss: 0.0138 - acc: 0.9966 - val_loss: 0.6139 - val_acc: 0.8550\n",
      "va acc: 0.855\n",
      "te acc: 0.855\n",
      "2018-10-01 23:22:19.224733 stack:4/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      " - 1s - loss: 0.3721 - acc: 0.8335 - val_loss: 0.3368 - val_acc: 0.8588\n",
      "Epoch 2/25\n",
      " - 0s - loss: 0.3035 - acc: 0.8721 - val_loss: 0.3339 - val_acc: 0.8582\n",
      "Epoch 3/25\n",
      " - 0s - loss: 0.2771 - acc: 0.8838 - val_loss: 0.3398 - val_acc: 0.8576\n",
      "Epoch 4/25\n",
      " - 0s - loss: 0.2599 - acc: 0.8907 - val_loss: 0.3414 - val_acc: 0.8596\n",
      "Epoch 5/25\n",
      " - 0s - loss: 0.2314 - acc: 0.9081 - val_loss: 0.3437 - val_acc: 0.8610\n",
      "Epoch 6/25\n",
      " - 0s - loss: 0.2089 - acc: 0.9193 - val_loss: 0.3559 - val_acc: 0.8590\n",
      "Epoch 7/25\n",
      " - 0s - loss: 0.1837 - acc: 0.9292 - val_loss: 0.3821 - val_acc: 0.8526\n",
      "Epoch 8/25\n",
      " - 0s - loss: 0.1596 - acc: 0.9417 - val_loss: 0.3664 - val_acc: 0.8588\n",
      "Epoch 9/25\n",
      " - 0s - loss: 0.1394 - acc: 0.9478 - val_loss: 0.4071 - val_acc: 0.8496\n",
      "Epoch 10/25\n",
      " - 0s - loss: 0.1132 - acc: 0.9602 - val_loss: 0.3889 - val_acc: 0.8566\n",
      "Epoch 11/25\n",
      " - 0s - loss: 0.0985 - acc: 0.9673 - val_loss: 0.4103 - val_acc: 0.8520\n",
      "Epoch 12/25\n",
      " - 0s - loss: 0.0812 - acc: 0.9748 - val_loss: 0.4196 - val_acc: 0.8566\n",
      "Epoch 13/25\n",
      " - 0s - loss: 0.0757 - acc: 0.9760 - val_loss: 0.4295 - val_acc: 0.8574\n",
      "Epoch 14/25\n",
      " - 0s - loss: 0.0587 - acc: 0.9828 - val_loss: 0.4439 - val_acc: 0.8584\n",
      "Epoch 15/25\n",
      " - 0s - loss: 0.0530 - acc: 0.9845 - val_loss: 0.4949 - val_acc: 0.8504\n",
      "Epoch 16/25\n",
      " - 0s - loss: 0.0447 - acc: 0.9877 - val_loss: 0.4863 - val_acc: 0.8536\n",
      "Epoch 17/25\n",
      " - 0s - loss: 0.0394 - acc: 0.9889 - val_loss: 0.4853 - val_acc: 0.8610\n",
      "Epoch 18/25\n",
      " - 0s - loss: 0.0313 - acc: 0.9932 - val_loss: 0.5130 - val_acc: 0.8570\n",
      "Epoch 19/25\n",
      " - 0s - loss: 0.0296 - acc: 0.9922 - val_loss: 0.5465 - val_acc: 0.8540\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20/25\n",
      " - 0s - loss: 0.0283 - acc: 0.9919 - val_loss: 0.5242 - val_acc: 0.8576\n",
      "Epoch 21/25\n",
      " - 0s - loss: 0.0246 - acc: 0.9928 - val_loss: 0.5559 - val_acc: 0.8570\n",
      "Epoch 22/25\n",
      " - 0s - loss: 0.0207 - acc: 0.9949 - val_loss: 0.5651 - val_acc: 0.8576\n",
      "Epoch 23/25\n",
      " - 0s - loss: 0.0176 - acc: 0.9958 - val_loss: 0.5769 - val_acc: 0.8534\n",
      "Epoch 24/25\n",
      " - 0s - loss: 0.0168 - acc: 0.9963 - val_loss: 0.5784 - val_acc: 0.8578\n",
      "Epoch 25/25\n",
      " - 0s - loss: 0.0143 - acc: 0.9970 - val_loss: 0.6166 - val_acc: 0.8526\n",
      "va acc: 0.84325\n",
      "te acc: 0.8526\n",
      "2018-10-01 23:22:30.078991 stack:5/5\n",
      "Train on 16000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      " - 1s - loss: 0.3765 - acc: 0.8347 - val_loss: 0.3454 - val_acc: 0.8536\n",
      "Epoch 2/25\n",
      " - 0s - loss: 0.3047 - acc: 0.8709 - val_loss: 0.3452 - val_acc: 0.8530\n",
      "Epoch 3/25\n",
      " - 0s - loss: 0.2795 - acc: 0.8829 - val_loss: 0.3395 - val_acc: 0.8556\n",
      "Epoch 4/25\n",
      " - 0s - loss: 0.2570 - acc: 0.8937 - val_loss: 0.3784 - val_acc: 0.8476\n",
      "Epoch 5/25\n",
      " - 0s - loss: 0.2326 - acc: 0.9068 - val_loss: 0.3579 - val_acc: 0.8558\n",
      "Epoch 6/25\n",
      " - 0s - loss: 0.2089 - acc: 0.9196 - val_loss: 0.3620 - val_acc: 0.8562\n",
      "Epoch 7/25\n",
      " - 0s - loss: 0.1838 - acc: 0.9309 - val_loss: 0.3748 - val_acc: 0.8536\n",
      "Epoch 8/25\n",
      " - 0s - loss: 0.1560 - acc: 0.9417 - val_loss: 0.3821 - val_acc: 0.8554\n",
      "Epoch 9/25\n",
      " - 0s - loss: 0.1350 - acc: 0.9529 - val_loss: 0.3978 - val_acc: 0.8560\n",
      "Epoch 10/25\n",
      " - 0s - loss: 0.1148 - acc: 0.9604 - val_loss: 0.4740 - val_acc: 0.8406\n",
      "Epoch 11/25\n",
      " - 0s - loss: 0.1018 - acc: 0.9636 - val_loss: 0.4232 - val_acc: 0.8504\n",
      "Epoch 12/25\n",
      " - 0s - loss: 0.0806 - acc: 0.9739 - val_loss: 0.4360 - val_acc: 0.8538\n",
      "Epoch 13/25\n",
      " - 0s - loss: 0.0705 - acc: 0.9783 - val_loss: 0.4581 - val_acc: 0.8536\n",
      "Epoch 14/25\n",
      " - 0s - loss: 0.0565 - acc: 0.9844 - val_loss: 0.4688 - val_acc: 0.8500\n",
      "Epoch 15/25\n",
      " - 0s - loss: 0.0523 - acc: 0.9858 - val_loss: 0.4857 - val_acc: 0.8556\n",
      "Epoch 16/25\n",
      " - 0s - loss: 0.0427 - acc: 0.9879 - val_loss: 0.4888 - val_acc: 0.8562\n",
      "Epoch 17/25\n",
      " - 0s - loss: 0.0380 - acc: 0.9903 - val_loss: 0.5174 - val_acc: 0.8584\n",
      "Epoch 18/25\n",
      " - 0s - loss: 0.0307 - acc: 0.9930 - val_loss: 0.5408 - val_acc: 0.8486\n",
      "Epoch 19/25\n",
      " - 0s - loss: 0.0305 - acc: 0.9918 - val_loss: 0.5460 - val_acc: 0.8522\n",
      "Epoch 20/25\n",
      " - 0s - loss: 0.0263 - acc: 0.9938 - val_loss: 0.5865 - val_acc: 0.8572\n",
      "Epoch 21/25\n",
      " - 0s - loss: 0.0216 - acc: 0.9948 - val_loss: 0.5639 - val_acc: 0.8528\n",
      "Epoch 22/25\n",
      " - 0s - loss: 0.0196 - acc: 0.9953 - val_loss: 0.5738 - val_acc: 0.8552\n",
      "Epoch 23/25\n",
      " - 0s - loss: 0.0159 - acc: 0.9965 - val_loss: 0.6080 - val_acc: 0.8574\n",
      "Epoch 24/25\n",
      " - 0s - loss: 0.0142 - acc: 0.9971 - val_loss: 0.6275 - val_acc: 0.8532\n",
      "Epoch 25/25\n",
      " - 0s - loss: 0.0134 - acc: 0.9970 - val_loss: 0.6419 - val_acc: 0.8532\n",
      "va acc: 0.85675\n",
      "te acc: 0.8532\n",
      "2018-10-01 23:22:41.267830 save dmd2v stack done!\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from datetime import datetime\n",
    "from sklearn.cross_validation import KFold\n",
    "from gensim.models import Doc2Vec\n",
    "from collections import OrderedDict\n",
    "\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Dropout, Activation\n",
    "from keras.optimizers import SGD, Adam, RMSprop\n",
    "from keras.utils import np_utils\n",
    "import re\n",
    "\n",
    "# ----------------------- myfunc -----------------------\n",
    "def myAcc(y_true,y_pred):\n",
    "    y_pred = np.argmax(y_pred,axis=1)\n",
    "    return np.mean(y_true == y_pred)\n",
    "\n",
    "# ----------------------- load dataset -----------------\n",
    "labeled_train = pd.read_csv('./data/labeledTrainData.tsv', sep='\\t', encoding='utf8')\n",
    "model = Doc2Vec.load('./data/dm_d2v_2.model')\n",
    "X_sp = np.array([model.docvecs[i] for i in range(25000)])\n",
    "labels = labeled_train['sentiment']\n",
    "\n",
    "# ----------------------dmd2v stack -------------------\n",
    "df_stack = pd.DataFrame(index=range(len(labeled_train)))\n",
    "TR = 20000\n",
    "n = 5\n",
    "\n",
    "X = X_sp[:TR]\n",
    "y = labels[:TR]\n",
    "X_te = X_sp[TR:]\n",
    "y_te = labels[TR:]\n",
    "\n",
    "num_class = len(pd.value_counts(labels))\n",
    "stack = np.zeros((X.shape[0],num_class))\n",
    "stack_te = np.zeros((X_te.shape[0],num_class))\n",
    "\n",
    "for k,(tr,va) in enumerate(KFold(len(y),n_folds=n)):\n",
    "    print('{} stack:{}/{}'.format(datetime.now(), k+1, n))\n",
    "    nb_classes = num_class\n",
    "    X_train = X[tr]\n",
    "    y_train = y[tr]\n",
    "    X_test = X_te\n",
    "    y_test = y_te\n",
    "    \n",
    "    X_train = X_train.astype('float32')\n",
    "    X_test = X_test.astype('float32')\n",
    "    Y_train = np_utils.to_categorical(y_train, nb_classes)\n",
    "    Y_test = np_utils.to_categorical(y_test, nb_classes)\n",
    "\n",
    "    model = Sequential()\n",
    "    model.add(Dense(300,input_shape=(X_train.shape[1],)))\n",
    "    model.add(Dropout(0.1))\n",
    "    model.add(Activation('tanh'))\n",
    "    model.add(Dense(nb_classes))\n",
    "    model.add(Activation('softmax'))\n",
    "\n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "                      optimizer='adadelta',\n",
    "                      metrics=['accuracy'])\n",
    "\n",
    "    history = model.fit(X_train, Y_train,shuffle=True,\n",
    "                            batch_size=128, nb_epoch=25,\n",
    "                            verbose=2, validation_data=(X_test, Y_test))\n",
    "    y_pred_va = model.predict_proba(X[va])\n",
    "    y_pred_te = model.predict_proba(X_te)\n",
    "    print('va acc:',myAcc(y[va],y_pred_va))\n",
    "    print('te acc:',myAcc(y_te,y_pred_te))\n",
    "    stack[va] += y_pred_va\n",
    "    stack_te += y_pred_te\n",
    "stack_te /= n\n",
    "stack_all = np.vstack([stack,stack_te])\n",
    "for l in range(stack_all.shape[1]):\n",
    "    df_stack['dmd2v_2_{}'.format(l)] = stack_all[:,l]\n",
    "        \n",
    "df_stack.to_csv('./data/dmd2v_stack_2.csv',encoding='utf8',index=None)\n",
    "print(datetime.now(),'save dmd2v stack done!')        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 2)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_stack.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dbowd2v_0</th>\n",
       "      <th>dbowd2v_1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.089950</td>\n",
       "      <td>0.910050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.004742</td>\n",
       "      <td>0.995258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.892094</td>\n",
       "      <td>0.107906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.999977</td>\n",
       "      <td>0.000023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.991922</td>\n",
       "      <td>0.008078</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   dbowd2v_0  dbowd2v_1\n",
       "0   0.089950   0.910050\n",
       "1   0.004742   0.995258\n",
       "2   0.892094   0.107906\n",
       "3   0.999977   0.000023\n",
       "4   0.991922   0.008078"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_stack.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3\n",
    "\n",
    "用全部数据进行Doc2vec的训练，以及对测试集进行预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_all = pd.concat([train,unlabeled_train,test]).fillna(0)\n",
    "data_all.to_csv('./data/data_all.csv',index=None,encoding='utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_all = pd.read_csv('./data/data_all.csv', encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"5814_8\"</td>\n",
       "      <td>\"With all this stuff going down at the moment ...</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"2381_9\"</td>\n",
       "      <td>\"\\\"The Classic War of the Worlds\\\" by Timothy ...</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"7759_3\"</td>\n",
       "      <td>\"The film starts with a manager (Nicholas Bell...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"3630_4\"</td>\n",
       "      <td>\"It must be assumed that those who praised thi...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"9495_8\"</td>\n",
       "      <td>\"Superbly trashy and wondrously unpretentious ...</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                             review  sentiment\n",
       "0  \"5814_8\"  \"With all this stuff going down at the moment ...        1.0\n",
       "1  \"2381_9\"  \"\\\"The Classic War of the Worlds\\\" by Timothy ...        1.0\n",
       "2  \"7759_3\"  \"The film starts with a manager (Nicholas Bell...        0.0\n",
       "3  \"3630_4\"  \"It must be assumed that those who praised thi...        0.0\n",
       "4  \"9495_8\"  \"Superbly trashy and wondrously unpretentious ...        1.0"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_all.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>99995</th>\n",
       "      <td>\"2155_10\"</td>\n",
       "      <td>\"Sony Pictures Classics, I'm looking at you! S...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99996</th>\n",
       "      <td>\"59_10\"</td>\n",
       "      <td>\"I always felt that Ms. Merkerson had never go...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99997</th>\n",
       "      <td>\"2531_1\"</td>\n",
       "      <td>\"I was so disappointed in this movie. I am ver...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99998</th>\n",
       "      <td>\"7772_8\"</td>\n",
       "      <td>\"From the opening sequence, filled with black ...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99999</th>\n",
       "      <td>\"11465_10\"</td>\n",
       "      <td>\"This is a great horror film for people who do...</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               id                                             review  \\\n",
       "99995   \"2155_10\"  \"Sony Pictures Classics, I'm looking at you! S...   \n",
       "99996     \"59_10\"  \"I always felt that Ms. Merkerson had never go...   \n",
       "99997    \"2531_1\"  \"I was so disappointed in this movie. I am ver...   \n",
       "99998    \"7772_8\"  \"From the opening sequence, filled with black ...   \n",
       "99999  \"11465_10\"  \"This is a great horror film for people who do...   \n",
       "\n",
       "       sentiment  \n",
       "99995        0.0  \n",
       "99996        0.0  \n",
       "99997        0.0  \n",
       "99998        0.0  \n",
       "99999        0.0  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_all.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stacking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['dbowd2v_0', 'dbowd2v_1', 'dbowd2v_2_0', 'dbowd2v_2_1', 'dmd2v_0',\n",
      "       'dmd2v_1', 'dmd2v_2_0', 'dmd2v_2_1'],\n",
      "      dtype='object')\n",
      "[0]\ttrain-merror:0.0975\teval-merror:0.1128\ttrain-acc:0.9025\teval-acc:0.8872\n",
      "Multiple eval metrics have been passed: 'eval-acc' will be used for early stopping.\n",
      "\n",
      "Will train until eval-acc hasn't improved in 25 rounds.\n",
      "[1]\ttrain-merror:0.0959\teval-merror:0.109\ttrain-acc:0.9041\teval-acc:0.891\n",
      "[2]\ttrain-merror:0.0946\teval-merror:0.1074\ttrain-acc:0.9054\teval-acc:0.8926\n",
      "[3]\ttrain-merror:0.0946\teval-merror:0.107\ttrain-acc:0.9054\teval-acc:0.893\n",
      "[4]\ttrain-merror:0.0937\teval-merror:0.1058\ttrain-acc:0.9063\teval-acc:0.8942\n",
      "[5]\ttrain-merror:0.09295\teval-merror:0.1032\ttrain-acc:0.90705\teval-acc:0.8968\n",
      "[6]\ttrain-merror:0.09295\teval-merror:0.1046\ttrain-acc:0.90705\teval-acc:0.8954\n",
      "[7]\ttrain-merror:0.09305\teval-merror:0.1036\ttrain-acc:0.90695\teval-acc:0.8964\n",
      "[8]\ttrain-merror:0.09335\teval-merror:0.1034\ttrain-acc:0.90665\teval-acc:0.8966\n",
      "[9]\ttrain-merror:0.0933\teval-merror:0.103\ttrain-acc:0.9067\teval-acc:0.897\n",
      "[10]\ttrain-merror:0.0928\teval-merror:0.1042\ttrain-acc:0.9072\teval-acc:0.8958\n",
      "[11]\ttrain-merror:0.09325\teval-merror:0.1034\ttrain-acc:0.90675\teval-acc:0.8966\n",
      "[12]\ttrain-merror:0.09315\teval-merror:0.1044\ttrain-acc:0.90685\teval-acc:0.8956\n",
      "[13]\ttrain-merror:0.0934\teval-merror:0.1044\ttrain-acc:0.9066\teval-acc:0.8956\n",
      "[14]\ttrain-merror:0.0924\teval-merror:0.1042\ttrain-acc:0.9076\teval-acc:0.8958\n",
      "[15]\ttrain-merror:0.09255\teval-merror:0.1044\ttrain-acc:0.90745\teval-acc:0.8956\n",
      "[16]\ttrain-merror:0.092\teval-merror:0.1044\ttrain-acc:0.908\teval-acc:0.8956\n",
      "[17]\ttrain-merror:0.0921\teval-merror:0.1042\ttrain-acc:0.9079\teval-acc:0.8958\n",
      "[18]\ttrain-merror:0.092\teval-merror:0.1044\ttrain-acc:0.908\teval-acc:0.8956\n",
      "[19]\ttrain-merror:0.09235\teval-merror:0.1038\ttrain-acc:0.90765\teval-acc:0.8962\n",
      "[20]\ttrain-merror:0.092\teval-merror:0.1032\ttrain-acc:0.908\teval-acc:0.8968\n",
      "[21]\ttrain-merror:0.09235\teval-merror:0.1042\ttrain-acc:0.90765\teval-acc:0.8958\n",
      "[22]\ttrain-merror:0.0927\teval-merror:0.1044\ttrain-acc:0.9073\teval-acc:0.8956\n",
      "[23]\ttrain-merror:0.0922\teval-merror:0.1042\ttrain-acc:0.9078\teval-acc:0.8958\n",
      "[24]\ttrain-merror:0.0925\teval-merror:0.1046\ttrain-acc:0.9075\teval-acc:0.8954\n",
      "[25]\ttrain-merror:0.09225\teval-merror:0.1042\ttrain-acc:0.90775\teval-acc:0.8958\n",
      "[26]\ttrain-merror:0.09225\teval-merror:0.1044\ttrain-acc:0.90775\teval-acc:0.8956\n",
      "[27]\ttrain-merror:0.09205\teval-merror:0.1046\ttrain-acc:0.90795\teval-acc:0.8954\n",
      "[28]\ttrain-merror:0.0921\teval-merror:0.104\ttrain-acc:0.9079\teval-acc:0.896\n",
      "[29]\ttrain-merror:0.09215\teval-merror:0.1044\ttrain-acc:0.90785\teval-acc:0.8956\n",
      "[30]\ttrain-merror:0.09245\teval-merror:0.1042\ttrain-acc:0.90755\teval-acc:0.8958\n",
      "[31]\ttrain-merror:0.09235\teval-merror:0.1044\ttrain-acc:0.90765\teval-acc:0.8956\n",
      "[32]\ttrain-merror:0.0921\teval-merror:0.1044\ttrain-acc:0.9079\teval-acc:0.8956\n",
      "[33]\ttrain-merror:0.0921\teval-merror:0.1048\ttrain-acc:0.9079\teval-acc:0.8952\n",
      "[34]\ttrain-merror:0.09215\teval-merror:0.1044\ttrain-acc:0.90785\teval-acc:0.8956\n",
      "Stopping. Best iteration:\n",
      "[9]\ttrain-merror:0.0933\teval-merror:0.103\ttrain-acc:0.9067\teval-acc:0.897\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import xgboost as xgb\n",
    "\n",
    "def xgb_acc_score(preds,dtrain):\n",
    "    y_true = dtrain.get_label()\n",
    "    y_pred = np.argmax(preds,axis=1)\n",
    "    return [('acc',np.mean(y_true == y_pred))]\n",
    "\n",
    "df_dbow = pd.read_csv('./data/dbowd2v_stack.csv')\n",
    "df_dbow2 = pd.read_csv('./data/dbowd2v_stack_2.csv')\n",
    "df_dm = pd.read_csv('./data/dmd2v_stack.csv')\n",
    "df_dm2 = pd.read_csv('./data/dmd2v_stack_2.csv')\n",
    "\n",
    "df = pd.concat([df_dbow, df_dbow2, df_dm, df_dm2], axis=1)\n",
    "print(df.columns)\n",
    "\n",
    "lable = np.array(labeled_train['sentiment'])\n",
    "num_class = len(pd.value_counts(labeled_train['sentiment']))\n",
    "TR = 20000\n",
    "seed = 10\n",
    "\n",
    "X = df.iloc[:TR]\n",
    "y = lable[:TR]\n",
    "X_te = df.iloc[TR:]\n",
    "y_te = lable[TR:]\n",
    "\n",
    "\n",
    "esr = 25\n",
    "evals = 1\n",
    "n_trees = 1000\n",
    "\n",
    "ss = 0.5\n",
    "mc = 0.8\n",
    "md = 7\n",
    "gm = 1\n",
    "# n_trees = 25\n",
    "\n",
    "params = {\n",
    "    \"objective\": \"multi:softprob\",\n",
    "    \"booster\": \"gbtree\",\n",
    "    # \"eval_metric\": \"merror\",\n",
    "    \"num_class\":num_class,\n",
    "    'max_depth':md,\n",
    "    'min_child_weight':mc,\n",
    "    'subsample':ss,\n",
    "    'colsample_bytree':1,\n",
    "    'gamma':gm,\n",
    "    \"eta\": 0.01,\n",
    "    \"lambda\":0,\n",
    "    'alpha':0,\n",
    "    \"silent\": 1,\n",
    "    # 'seed':seed,\n",
    "}\n",
    "\n",
    "dtrain = xgb.DMatrix(X, y)\n",
    "dvalid = xgb.DMatrix(X_te, y_te)\n",
    "watchlist = [(dtrain, 'train'), (dvalid, 'eval')]\n",
    "bst = xgb.train(params, dtrain, n_trees, evals=watchlist, feval=xgb_acc_score, maximize=True,\n",
    "                early_stopping_rounds=esr, verbose_eval=evals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}