![](\"https://github.com/fani-lab/Adila/raw/main/misc/flow.png\")
\n",
"\n",
"# Quickstart\n",
"The following is a sample run of `Adila` using a toy dataset modelled about the DBLP dataset. The arguments it employed are:\n",
@@ -33,16 +29,50 @@
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "vscode": {
- "languageId": "shellscript"
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[WinError 2] The system cannot find the file specified: 'src'\n",
+ "c:\\Users\\tea-n_\\Documents\\GitHub\\OpeNTF framework\\Adila\\src\n",
+ "Parallel run started ...\n",
+ "####################################################################################################\n",
+ "Reranking for the baseline ../output/toy.dblp.v12.json/bnn/t31.s11.m13.l[100].lr0.1.b4096.e20.s1/f0.test.pred ...\n",
+ "Loading popularity labels ...\n",
+ "Loading reranking results ...\n",
+ "Loading fairness evaluation results before and after reranking ...\n",
+ "####################################################################################################\n",
+ "Reranking for the baseline ../output/toy.dblp.v12.json/bnn/t31.s11.m13.l[100].lr0.1.b4096.e20.s1/f1.test.pred ...\n",
+ "Loading utility metric evaluation results before and after reranking ...\n",
+ "Loading popularity labels ...\n",
+ "Loading reranking results ...\n",
+ "Pipeline for the baseline ../output/toy.dblp.v12.json/bnn/t31.s11.m13.l[100].lr0.1.b4096.e20.s1/f0.test.pred completed by ![](\"./qr-code.png\")
"
+ ]
}
],
"metadata": {
+ "kernelspec": {
+ "display_name": "adila",
+ "language": "python",
+ "name": "python3"
+ },
"language_info": {
- "name": "python"
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.16"
}
},
"nbformat": 4,
diff --git a/docs/tutorial/umap24/jupyter/OpeNTF.ipynb b/docs/tutorial/umap24/jupyter/OpeNTF.ipynb
index b56fd621..309bcca3 100644
--- a/docs/tutorial/umap24/jupyter/OpeNTF.ipynb
+++ b/docs/tutorial/umap24/jupyter/OpeNTF.ipynb
@@ -4,55 +4,17 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "# What is OpeNTF?\n",
- "OpeNTF is an open-source framework hosting large-scale training datasets and canonical neural team formation models that are trained using fairness-aware and time-sensitive methods."
+ "# What is `OpeNTF`?\n",
+ "`OpeNTF` is an open-source framework hosting large-scale training datasets and canonical neural team formation models that are trained using fairness-aware and time-sensitive methods."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Prerequisite for OpeNTF\n",
- "Before using OpeNTF, the following libraries are needed as a prerequisite, in addition to the libraries in `requirements.txt`:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "vscode": {
- "languageId": "shellscript"
- }
- },
- "outputs": [],
- "source": [
- "pip install torch==1.9.0\n",
- "pip install pytrec-eval-terrier==0.5.2\n",
- "pip install gensim==3.8.3"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "vscode": {
- "languageId": "shellscript"
- }
- },
- "outputs": [],
- "source": [
- "git clone --recursive https://github.com/Fani-Lab/opentf\n",
- "cd opentf\n",
- "pip install -r requirements.txt"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Quickstart on OpeNTF\n",
+ "## Quickstart on `OpeNTF`\n",
"\n",
- "OpeNTF has the following required arguments:\n",
+ "`OpeNTF` has the following required arguments:\n",
"\n",
"- `-data`: the path of the input datasets.\n",
"- `-domain`: the domain the input dataset belongs in.\n",
@@ -60,7 +22,7 @@
"\n",
"As well, other optional arguments include:\n",
"- `-attribute`: the set of our sensitive attributes (e.g., popularity).\n",
- "- `-fairness`: fairness metrics for reranking algorithms, used to minimize popularity bias.\n",
+ "- `-fairness`: fairness metrics for reranking algorithms, used to minimize different bias.\n",
"- `-np-ratio`: desired ratio of non-popular experts after reranking.\n",
"- `-k_max`: cutoff for the reranking algorithms.\n",
"- `-filter`: remove outliers, if needed.\n",
@@ -68,21 +30,318 @@
"- `-exp_id`: ID of the experiment.\n",
"- `-output`: path of the baseline output.\n",
"\n",
- "The following is a sample run of the OpeNTF codebase using a toy dataset `toy.dblp.v12.json`, which is modelled after the DBLP dataset: a dataset consisting of authorship and skill information on more than 4 million Computer Science research publications. Two neural models (`feedforward` and `Bayesian`) are used in this quickstart."
+ "Note that some of the optional arguments on the top uses the extensions of `OpeNTF` (e.g., `Adila`).\n",
+ "\n",
+ "The following is a sample run of the `OpeNTF` codebase using a toy dataset `toy.dblp.v12.json`, which is modelled after the DBLP dataset: a dataset consisting of authorship and skill information on more than 4 million Computer Science research publications. Two neural models (`feedforward` and `Bayesian`) are used in this quickstart."
]
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "vscode": {
- "languageId": "shellscript"
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[WinError 2] The system cannot find the file specified: 'src'\n",
+ "c:\\Users\\tea-n_\\Documents\\GitHub\\OpeNTF framework\\OpeNTF\\src\n",
+ "Loading sparse matrices from ./../data/preprocessed/dblp/toy.dblp.v12.json/teamsvecs.pkl ...\n",
+ "Loading indexes pickle from ./../data/preprocessed/dblp/toy.dblp.v12.json/indexes.pkl ...\n",
+ "It took 0.0019981861114501953 seconds to load from the pickles.\n",
+ "It took 0.012996912002563477 seconds to load the sparse matrices.\n",
+ "Running for (dataset, model): (dblp, fnn) ... \n",
+ "Fold 0/2, Epoch 0/9, Minibatch 0/0, Phase train, Running Loss train 0.6056249141693115, Time 0.0507659912109375, Overall 2.9880211353302 \n",
+ "Fold 0/2, Epoch 0/9, Running Loss train 0.03562499495113597, Time 0.0507659912109375, Overall 2.9880211353302 \n",
+ "Fold 0/2, Epoch 0/9, Minibatch 0/0, Phase valid, Running Loss valid 0.5972533226013184, Time 0.05572986602783203, Overall 2.9929850101470947 \n",
+ "Fold 0/2, Epoch 0/9, Running Loss valid 0.06636148028903538, Time 0.05572986602783203, Overall 2.9929850101470947 \n",
+ "Fold 0/2, Epoch 1/9, Minibatch 0/0, Phase train, Running Loss train 0.5991436243057251, Time 0.07272839546203613, Overall 3.009983539581299 \n",
+ "Fold 0/2, Epoch 1/9, Running Loss train 0.035243742606219125, Time 0.07272839546203613, Overall 3.009983539581299 \n",
+ "Fold 0/2, Epoch 1/9, Minibatch 0/0, Phase valid, Running Loss valid 0.592308521270752, Time 0.07672905921936035, Overall 3.013984203338623 \n",
+ "Fold 0/2, Epoch 1/9, Running Loss valid 0.06581205791897243, Time 0.07672905921936035, Overall 3.013984203338623 \n",
+ "EarlyStopping counter: 1 out of 5\n",
+ "Fold 0/2, Epoch 2/9, Minibatch 0/0, Phase train, Running Loss train 0.592818021774292, Time 0.08708858489990234, Overall 3.024343729019165 \n",
+ "Fold 0/2, Epoch 2/9, Running Loss train 0.03487164833966423, Time 0.08708858489990234, Overall 3.024343729019165 \n",
+ "Fold 0/2, Epoch 2/9, Minibatch 0/0, Phase valid, Running Loss valid 0.5874701738357544, Time 0.09109020233154297, Overall 3.0283453464508057 \n",
+ "Fold 0/2, Epoch 2/9, Running Loss valid 0.06527446375952826, Time 0.09109020233154297, Overall 3.0283453464508057 \n",
+ "EarlyStopping counter: 2 out of 5\n",
+ "Fold 0/2, Epoch 3/9, Minibatch 0/0, Phase train, Running Loss train 0.5866307020187378, Time 0.10109424591064453, Overall 3.0383493900299072 \n",
+ "Fold 0/2, Epoch 3/9, Running Loss train 0.0345076883540434, Time 0.10109424591064453, Overall 3.0383493900299072 \n",
+ "Fold 0/2, Epoch 3/9, Minibatch 0/0, Phase valid, Running Loss valid 0.5827431082725525, Time 0.10508847236633301, Overall 3.0423436164855957 \n",
+ "Fold 0/2, Epoch 3/9, Running Loss valid 0.06474923425250584, Time 0.10508847236633301, Overall 3.0423436164855957 \n",
+ "EarlyStopping counter: 3 out of 5\n",
+ "Fold 0/2, Epoch 4/9, Minibatch 0/0, Phase train, Running Loss train 0.5805637240409851, Time 0.11522793769836426, Overall 3.052483081817627 \n",
+ "Fold 0/2, Epoch 4/9, Running Loss train 0.034150807296528536, Time 0.11522793769836426, Overall 3.052483081817627 \n",
+ "Fold 0/2, Epoch 4/9, Minibatch 0/0, Phase valid, Running Loss valid 0.5781247019767761, Time 0.11822772026062012, Overall 3.055482864379883 \n",
+ "Fold 0/2, Epoch 4/9, Running Loss valid 0.06423607799741957, Time 0.11922717094421387, Overall 3.0564823150634766 \n",
+ "EarlyStopping counter: 4 out of 5\n",
+ "Fold 0/2, Epoch 5/9, Minibatch 0/0, Phase train, Running Loss train 0.574617326259613, Time 0.12823057174682617, Overall 3.065485715866089 \n",
+ "Fold 0/2, Epoch 5/9, Running Loss train 0.03380101919174194, Time 0.12922906875610352, Overall 3.066484212875366 \n",
+ "Fold 0/2, Epoch 5/9, Minibatch 0/0, Phase valid, Running Loss valid 0.5736306309700012, Time 0.13222789764404297, Overall 3.0694830417633057 \n",
+ "Fold 0/2, Epoch 5/9, Running Loss valid 0.06373673677444458, Time 0.13222789764404297, Overall 3.0694830417633057 \n",
+ "EarlyStopping counter: 5 out of 5\n",
+ "Early Stopping Triggered at epoch: 5\n",
+ "Fold 1/2, Epoch 0/9, Minibatch 0/0, Phase train, Running Loss train 0.6933990716934204, Time 0.009009122848510742, Overall 3.0846405029296875 \n",
+ "Fold 1/2, Epoch 0/9, Running Loss train 0.04078818068784826, Time 0.009009122848510742, Overall 3.0846405029296875 \n",
+ "Fold 1/2, Epoch 0/9, Minibatch 0/0, Phase valid, Running Loss valid 0.6659319996833801, Time 0.012007474899291992, Overall 3.0876388549804688 \n",
+ "Fold 1/2, Epoch 0/9, Running Loss valid 0.07399244440926446, Time 0.01300811767578125, Overall 3.088639497756958 \n",
+ "Fold 1/2, Epoch 1/9, Minibatch 0/0, Phase train, Running Loss train 0.6839163899421692, Time 0.0290071964263916, Overall 3.1046385765075684 \n",
+ "Fold 1/2, Epoch 1/9, Running Loss train 0.040230375878951126, Time 0.0290071964263916, Overall 3.1046385765075684 \n",
+ "Fold 1/2, Epoch 1/9, Minibatch 0/0, Phase valid, Running Loss valid 0.6573002338409424, Time 0.033972978591918945, Overall 3.1096043586730957 \n",
+ "Fold 1/2, Epoch 1/9, Running Loss valid 0.07303335931566027, Time 0.033972978591918945, Overall 3.1096043586730957 \n",
+ "EarlyStopping counter: 1 out of 5\n",
+ "Fold 1/2, Epoch 2/9, Minibatch 0/0, Phase train, Running Loss train 0.6747457981109619, Time 0.044973134994506836, Overall 3.1206045150756836 \n",
+ "Fold 1/2, Epoch 2/9, Running Loss train 0.03969092930064482, Time 0.044973134994506836, Overall 3.1206045150756836 \n",
+ "Fold 1/2, Epoch 2/9, Minibatch 0/0, Phase valid, Running Loss valid 0.6489450335502625, Time 0.04804110527038574, Overall 3.1236724853515625 \n",
+ "Fold 1/2, Epoch 2/9, Running Loss valid 0.07210500372780694, Time 0.04804110527038574, Overall 3.1236724853515625 \n",
+ "EarlyStopping counter: 2 out of 5\n",
+ "Fold 1/2, Epoch 3/9, Minibatch 0/0, Phase train, Running Loss train 0.6658795475959778, Time 0.059162139892578125, Overall 3.134793519973755 \n",
+ "Fold 1/2, Epoch 3/9, Running Loss train 0.03916938515270457, Time 0.059162139892578125, Overall 3.134793519973755 \n",
+ "Fold 1/2, Epoch 3/9, Minibatch 0/0, Phase valid, Running Loss valid 0.640863835811615, Time 0.06316494941711426, Overall 3.138796329498291 \n",
+ "Fold 1/2, Epoch 3/9, Running Loss valid 0.07120709286795722, Time 0.06316494941711426, Overall 3.138796329498291 \n",
+ "EarlyStopping counter: 3 out of 5\n",
+ "Fold 1/2, Epoch 4/9, Minibatch 0/0, Phase train, Running Loss train 0.6573038697242737, Time 0.07298898696899414, Overall 3.148620367050171 \n",
+ "Fold 1/2, Epoch 4/9, Running Loss train 0.03866493351319257, Time 0.07298898696899414, Overall 3.148620367050171 \n",
+ "Fold 1/2, Epoch 4/9, Minibatch 0/0, Phase valid, Running Loss valid 0.6330206394195557, Time 0.0762319564819336, Overall 3.1518633365631104 \n",
+ "Fold 1/2, Epoch 4/9, Running Loss valid 0.07033562660217285, Time 0.0762319564819336, Overall 3.1518633365631104 \n",
+ "EarlyStopping counter: 4 out of 5\n",
+ "Fold 1/2, Epoch 5/9, Minibatch 0/0, Phase train, Running Loss train 0.6490036845207214, Time 0.08623027801513672, Overall 3.1618616580963135 \n",
+ "Fold 1/2, Epoch 5/9, Running Loss train 0.03817668732474832, Time 0.08723044395446777, Overall 3.1628618240356445 \n",
+ "Fold 1/2, Epoch 5/9, Minibatch 0/0, Phase valid, Running Loss valid 0.6254642009735107, Time 0.09022855758666992, Overall 3.1658599376678467 \n",
+ "Fold 1/2, Epoch 5/9, Running Loss valid 0.06949602233039008, Time 0.09022855758666992, Overall 3.1658599376678467 \n",
+ "EarlyStopping counter: 5 out of 5\n",
+ "Early Stopping Triggered at epoch: 5\n",
+ "Fold 2/2, Epoch 0/9, Minibatch 0/0, Phase train, Running Loss train 0.6977599263191223, Time 0.009129047393798828, Overall 3.1809446811676025 \n",
+ "Fold 2/2, Epoch 0/9, Running Loss train 0.03876444035106235, Time 0.009129047393798828, Overall 3.1809446811676025 \n",
+ "Fold 2/2, Epoch 0/9, Minibatch 0/0, Phase valid, Running Loss valid 0.7054514288902283, Time 0.01214146614074707, Overall 3.183957099914551 \n",
+ "Fold 2/2, Epoch 0/9, Running Loss valid 0.08818142861127853, Time 0.01214146614074707, Overall 3.183957099914551 \n",
+ "Fold 2/2, Epoch 1/9, Minibatch 0/0, Phase train, Running Loss train 0.687987744808197, Time 0.025124788284301758, Overall 3.1969404220581055 \n",
+ "Fold 2/2, Epoch 1/9, Running Loss train 0.038221541378233165, Time 0.025124788284301758, Overall 3.1969404220581055 \n",
+ "Fold 2/2, Epoch 1/9, Minibatch 0/0, Phase valid, Running Loss valid 0.6971001029014587, Time 0.02812790870666504, Overall 3.1999435424804688 \n",
+ "Fold 2/2, Epoch 1/9, Running Loss valid 0.08713751286268234, Time 0.02812790870666504, Overall 3.1999435424804688 \n",
+ "EarlyStopping counter: 1 out of 5\n",
+ "Fold 2/2, Epoch 2/9, Minibatch 0/0, Phase train, Running Loss train 0.678523600101471, Time 0.03809022903442383, Overall 3.2099058628082275 \n",
+ "Fold 2/2, Epoch 2/9, Running Loss train 0.03769575556119283, Time 0.03809022903442383, Overall 3.2099058628082275 \n",
+ "Fold 2/2, Epoch 2/9, Minibatch 0/0, Phase valid, Running Loss valid 0.6890474557876587, Time 0.04112529754638672, Overall 3.2129409313201904 \n",
+ "Fold 2/2, Epoch 2/9, Running Loss valid 0.08613093197345734, Time 0.04212832450866699, Overall 3.2139439582824707 \n",
+ "EarlyStopping counter: 2 out of 5\n",
+ "Fold 2/2, Epoch 3/9, Minibatch 0/0, Phase train, Running Loss train 0.6693738698959351, Time 0.05153536796569824, Overall 3.223351001739502 \n",
+ "Fold 2/2, Epoch 3/9, Running Loss train 0.037187437216440834, Time 0.05153536796569824, Overall 3.223351001739502 \n",
+ "Fold 2/2, Epoch 3/9, Minibatch 0/0, Phase valid, Running Loss valid 0.6812991499900818, Time 0.055664777755737305, Overall 3.227480411529541 \n",
+ "Fold 2/2, Epoch 3/9, Running Loss valid 0.08516239374876022, Time 0.055664777755737305, Overall 3.227480411529541 \n",
+ "EarlyStopping counter: 3 out of 5\n",
+ "Fold 2/2, Epoch 4/9, Minibatch 0/0, Phase train, Running Loss train 0.6605099439620972, Time 0.06554150581359863, Overall 3.2373571395874023 \n",
+ "Fold 2/2, Epoch 4/9, Running Loss train 0.036694996886783175, Time 0.06554150581359863, Overall 3.2383627891540527 \n",
+ "Fold 2/2, Epoch 4/9, Minibatch 0/0, Phase valid, Running Loss valid 0.6738382577896118, Time 0.06960129737854004, Overall 3.2414169311523438 \n",
+ "Fold 2/2, Epoch 4/9, Running Loss valid 0.08422978222370148, Time 0.06960129737854004, Overall 3.2414169311523438 \n",
+ "EarlyStopping counter: 4 out of 5\n",
+ "Fold 2/2, Epoch 5/9, Minibatch 0/0, Phase train, Running Loss train 0.6519139409065247, Time 0.07919692993164062, Overall 3.2510125637054443 \n",
+ "Fold 2/2, Epoch 5/9, Running Loss train 0.036217441161473594, Time 0.08025479316711426, Overall 3.252070426940918 \n",
+ "Fold 2/2, Epoch 5/9, Minibatch 0/0, Phase valid, Running Loss valid 0.6666646599769592, Time 0.0832357406616211, Overall 3.255051374435425 \n",
+ "Fold 2/2, Epoch 5/9, Running Loss valid 0.0833330824971199, Time 0.0832357406616211, Overall 3.255051374435425 \n",
+ "EarlyStopping counter: 5 out of 5\n",
+ "Early Stopping Triggered at epoch: 5\n",
+ "It took 3.2590525150299072 to train the model.\n",
+ "Figure(640x480)\n",
+ "Figure(640x480)\n",
+ "Figure(640x480)\n",
+ "Building pytrec_eval input for 5 instances ...\n",
+ "Evaluating {'map_cut_2,5,10', 'ndcg_cut_2,5,10', 'P_2,5,10', 'recall_2,5,10'} ...\n",
+ "Averaging ...\n",
+ "Building pytrec_eval input for 5 instances ...\n",
+ "Evaluating {'map_cut_2,5,10', 'ndcg_cut_2,5,10', 'P_2,5,10', 'recall_2,5,10'} ...\n",
+ "Averaging ...\n",
+ "Building pytrec_eval input for 5 instances ...\n",
+ "Evaluating {'map_cut_2,5,10', 'ndcg_cut_2,5,10', 'P_2,5,10', 'recall_2,5,10'} ...\n",
+ "Averaging ...\n",
+ "####################################################################################################\n",
+ "Reranking for the baseline ../output/toy.dblp.v12.json/fnn/t31.s10.m13.l[100].lr0.001.b128.e10.nns3.nsnone.lossSL/t31.s10.m13.l[100].lr0.001.b128.e10.nns3.nsnone.lossSL/f0.test.pred ...\n",
+ "Loading popularity labels ...\n",
+ "Loading reranking results ...\n",
+ "Loading fairness evaluation results before and after reranking ...\n",
+ "Loading fairness results failed! Evaluating fairness metric {'ndkl'} ...\n",
+ "Loading utility metric evaluation results before and after reranking ...\n",
+ "Pipeline for the baseline ../output/toy.dblp.v12.json/fnn/t31.s10.m13.l[100].lr0.001.b128.e10.nns3.nsnone.lossSL/t31.s10.m13.l[100].lr0.001.b128.e10.nns3.nsnone.lossSL/f0.test.pred completed by <_MainProcess name='MainProcess' parent=None started>! 0.06149601936340332\n",
+ "####################################################################################################\n",
+ "####################################################################################################\n",
+ "Reranking for the baseline ../output/toy.dblp.v12.json/fnn/t31.s10.m13.l[100].lr0.001.b128.e10.nns3.nsnone.lossSL/t31.s10.m13.l[100].lr0.001.b128.e10.nns3.nsnone.lossSL/f1.test.pred ...\n",
+ "Loading popularity labels ...\n",
+ "Loading reranking results ...\n",
+ "Loading fairness evaluation results before and after reranking ...\n",
+ "Loading fairness results failed! Evaluating fairness metric {'ndkl'} ...\n",
+ "Loading utility metric evaluation results before and after reranking ...\n",
+ "Pipeline for the baseline ../output/toy.dblp.v12.json/fnn/t31.s10.m13.l[100].lr0.001.b128.e10.nns3.nsnone.lossSL/t31.s10.m13.l[100].lr0.001.b128.e10.nns3.nsnone.lossSL/f1.test.pred completed by <_MainProcess name='MainProcess' parent=None started>! 0.040962934494018555\n",
+ "####################################################################################################\n",
+ "####################################################################################################\n",
+ "Reranking for the baseline ../output/toy.dblp.v12.json/fnn/t31.s10.m13.l[100].lr0.001.b128.e10.nns3.nsnone.lossSL/t31.s10.m13.l[100].lr0.001.b128.e10.nns3.nsnone.lossSL/f2.test.pred ...\n",
+ "Loading popularity labels ...\n",
+ "Loading reranking results ...\n",
+ "Loading fairness evaluation results before and after reranking ...\n",
+ "Loading fairness results failed! Evaluating fairness metric {'ndkl'} ...\n",
+ "Loading utility metric evaluation results before and after reranking ...\n",
+ "Pipeline for the baseline ../output/toy.dblp.v12.json/fnn/t31.s10.m13.l[100].lr0.001.b128.e10.nns3.nsnone.lossSL/t31.s10.m13.l[100].lr0.001.b128.e10.nns3.nsnone.lossSL/f2.test.pred completed by <_MainProcess name='MainProcess' parent=None started>! 0.04145026206970215\n",
+ "####################################################################################################\n",
+ "Running for (dataset, model): (dblp, bnn) ... \n",
+ "Fold 0/2, Epoch 0/4, Minibatch 0/0, Phase train, Running Loss train 9.701826095581055, Time 0.018190622329711914, Overall 0.021055936813354492 \n",
+ "Fold 0/2, Epoch 0/4, Running Loss train 0.5706956526812386, Time 0.018190622329711914, Overall 0.021055936813354492 \n",
+ "Fold 0/2, Epoch 0/4, Minibatch 0/0, Phase valid, Running Loss valid 6.872168064117432, Time 0.027338266372680664, Overall 0.030203580856323242 \n",
+ "Fold 0/2, Epoch 0/4, Running Loss valid 0.7635742293463813, Time 0.027338266372680664, Overall 0.030203580856323242 \n",
+ "Fold 0/2, Epoch 1/4, Minibatch 0/0, Phase train, Running Loss train 7.5233869552612305, Time 0.04839587211608887, Overall 0.051261186599731445 \n",
+ "Fold 0/2, Epoch 1/4, Running Loss train 0.4425521738388959, Time 0.04839587211608887, Overall 0.051261186599731445 \n",
+ "Fold 0/2, Epoch 1/4, Minibatch 0/0, Phase valid, Running Loss valid 5.3721466064453125, Time 0.05639481544494629, Overall 0.05926012992858887 \n",
+ "Fold 0/2, Epoch 1/4, Running Loss valid 0.5969051784939237, Time 0.05739259719848633, Overall 0.060257911682128906 \n",
+ "Fold 0/2, Epoch 2/4, Minibatch 0/0, Phase train, Running Loss train 8.192795753479004, Time 0.0774836540222168, Overall 0.08034896850585938 \n",
+ "Fold 0/2, Epoch 2/4, Running Loss train 0.48192916196935315, Time 0.0774836540222168, Overall 0.08034896850585938 \n",
+ "Fold 0/2, Epoch 2/4, Minibatch 0/0, Phase valid, Running Loss valid 3.359665632247925, Time 0.0865178108215332, Overall 0.08938312530517578 \n",
+ "Fold 0/2, Epoch 2/4, Running Loss valid 0.37329618136088055, Time 0.0865178108215332, Overall 0.08938312530517578 \n",
+ "Fold 0/2, Epoch 3/4, Minibatch 0/0, Phase train, Running Loss train 8.646369934082031, Time 0.1096503734588623, Overall 0.11251568794250488 \n",
+ "Fold 0/2, Epoch 3/4, Running Loss train 0.5086099961224724, Time 0.1096503734588623, Overall 0.11251568794250488 \n",
+ "Fold 0/2, Epoch 3/4, Minibatch 0/0, Phase valid, Running Loss valid 10.325803756713867, Time 0.11765170097351074, Overall 0.12051701545715332 \n",
+ "Fold 0/2, Epoch 3/4, Running Loss valid 1.147311528523763, Time 0.11868834495544434, Overall 0.12155365943908691 \n",
+ "EarlyStopping counter: 1 out of 5\n",
+ "Fold 0/2, Epoch 4/4, Minibatch 0/0, Phase train, Running Loss train 8.802574157714844, Time 0.13765859603881836, Overall 0.14052391052246094 \n",
+ "Fold 0/2, Epoch 4/4, Running Loss train 0.517798479865579, Time 0.138657808303833, Overall 0.14152312278747559 \n",
+ "Fold 0/2, Epoch 4/4, Minibatch 0/0, Phase valid, Running Loss valid 6.550690650939941, Time 0.14665746688842773, Overall 0.1495227813720703 \n",
+ "Fold 0/2, Epoch 4/4, Running Loss valid 0.7278545167711046, Time 0.14665746688842773, Overall 0.1495227813720703 \n",
+ "EarlyStopping counter: 2 out of 5\n",
+ "Fold 1/2, Epoch 0/4, Minibatch 0/0, Phase train, Running Loss train 8.19212818145752, Time 0.016184568405151367, Overall 0.17570805549621582 \n",
+ "Fold 1/2, Epoch 0/4, Running Loss train 0.48188989302691293, Time 0.016184568405151367, Overall 0.17570805549621582 \n",
+ "Fold 1/2, Epoch 0/4, Minibatch 0/0, Phase valid, Running Loss valid 10.479596138000488, Time 0.02418231964111328, Overall 0.18370580673217773 \n",
+ "Fold 1/2, Epoch 0/4, Running Loss valid 1.1643995708889432, Time 0.025183677673339844, Overall 0.1847071647644043 \n",
+ "Fold 1/2, Epoch 1/4, Minibatch 0/0, Phase train, Running Loss train 7.664950370788574, Time 0.04637622833251953, Overall 0.20589971542358398 \n",
+ "Fold 1/2, Epoch 1/4, Running Loss train 0.45087943357579846, Time 0.04637622833251953, Overall 0.20589971542358398 \n",
+ "Fold 1/2, Epoch 1/4, Minibatch 0/0, Phase valid, Running Loss valid 5.143964767456055, Time 0.05441546440124512, Overall 0.21393895149230957 \n",
+ "Fold 1/2, Epoch 1/4, Running Loss valid 0.5715516408284506, Time 0.05441546440124512, Overall 0.21393895149230957 \n",
+ "Fold 1/2, Epoch 2/4, Minibatch 0/0, Phase train, Running Loss train 7.0461907386779785, Time 0.07616400718688965, Overall 0.2356874942779541 \n",
+ "Fold 1/2, Epoch 2/4, Running Loss train 0.41448180815752816, Time 0.07616400718688965, Overall 0.2356874942779541 \n",
+ "Fold 1/2, Epoch 2/4, Minibatch 0/0, Phase valid, Running Loss valid 5.6814775466918945, Time 0.0841226577758789, Overall 0.24364614486694336 \n",
+ "Fold 1/2, Epoch 2/4, Running Loss valid 0.6312752829657661, Time 0.0841226577758789, Overall 0.24364614486694336 \n",
+ "EarlyStopping counter: 1 out of 5\n",
+ "Fold 1/2, Epoch 3/4, Minibatch 0/0, Phase train, Running Loss train 9.323131561279297, Time 0.10512852668762207, Overall 0.2646520137786865 \n",
+ "Fold 1/2, Epoch 3/4, Running Loss train 0.5484195036046645, Time 0.10512852668762207, Overall 0.2646520137786865 \n",
+ "Fold 1/2, Epoch 3/4, Minibatch 0/0, Phase valid, Running Loss valid 5.848703384399414, Time 0.11430501937866211, Overall 0.27382850646972656 \n",
+ "Fold 1/2, Epoch 3/4, Running Loss valid 0.6498559315999349, Time 0.11430501937866211, Overall 0.27382850646972656 \n",
+ "EarlyStopping counter: 2 out of 5\n",
+ "Fold 1/2, Epoch 4/4, Minibatch 0/0, Phase train, Running Loss train 8.630220413208008, Time 0.13330769538879395, Overall 0.2928311824798584 \n",
+ "Fold 1/2, Epoch 4/4, Running Loss train 0.5076600243063534, Time 0.13330769538879395, Overall 0.2928311824798584 \n",
+ "Fold 1/2, Epoch 4/4, Minibatch 0/0, Phase valid, Running Loss valid 6.053482532501221, Time 0.1417984962463379, Overall 0.30132198333740234 \n",
+ "Fold 1/2, Epoch 4/4, Running Loss valid 0.6726091702779134, Time 0.1417984962463379, Overall 0.30132198333740234 \n",
+ "EarlyStopping counter: 3 out of 5\n",
+ "Fold 2/2, Epoch 0/4, Minibatch 0/0, Phase train, Running Loss train 13.120939254760742, Time 0.016001462936401367, Overall 0.32532310485839844 \n",
+ "Fold 2/2, Epoch 0/4, Running Loss train 0.7289410697089301, Time 0.016962766647338867, Overall 0.32628440856933594 \n",
+ "Fold 2/2, Epoch 0/4, Minibatch 0/0, Phase valid, Running Loss valid 7.2940826416015625, Time 0.02499985694885254, Overall 0.3343214988708496 \n",
+ "Fold 2/2, Epoch 0/4, Running Loss valid 0.9117603302001953, Time 0.02499985694885254, Overall 0.3343214988708496 \n",
+ "Fold 2/2, Epoch 1/4, Minibatch 0/0, Phase train, Running Loss train 6.706620693206787, Time 0.04700016975402832, Overall 0.3563218116760254 \n",
+ "Fold 2/2, Epoch 1/4, Running Loss train 0.3725900385114882, Time 0.04700016975402832, Overall 0.3563218116760254 \n",
+ "Fold 2/2, Epoch 1/4, Minibatch 0/0, Phase valid, Running Loss valid 5.9946818351745605, Time 0.05800151824951172, Overall 0.3673231601715088 \n",
+ "Fold 2/2, Epoch 1/4, Running Loss valid 0.7493352293968201, Time 0.05800151824951172, Overall 0.3673231601715088 \n",
+ "Fold 2/2, Epoch 2/4, Minibatch 0/0, Phase train, Running Loss train 6.208798408508301, Time 0.07899999618530273, Overall 0.3883216381072998 \n",
+ "Fold 2/2, Epoch 2/4, Running Loss train 0.34493324491712785, Time 0.07899999618530273, Overall 0.3883216381072998 \n",
+ "Fold 2/2, Epoch 2/4, Minibatch 0/0, Phase valid, Running Loss valid 6.477299213409424, Time 0.0873870849609375, Overall 0.39670872688293457 \n",
+ "Fold 2/2, Epoch 2/4, Running Loss valid 0.809662401676178, Time 0.08847355842590332, Overall 0.3977952003479004 \n",
+ "EarlyStopping counter: 1 out of 5\n",
+ "Fold 2/2, Epoch 3/4, Minibatch 0/0, Phase train, Running Loss train 6.142394065856934, Time 0.10651612281799316, Overall 0.41583776473999023 \n",
+ "Fold 2/2, Epoch 3/4, Running Loss train 0.34124411476982963, Time 0.10651612281799316, Overall 0.41583776473999023 \n",
+ "Fold 2/2, Epoch 3/4, Minibatch 0/0, Phase valid, Running Loss valid 6.238161563873291, Time 0.11433148384094238, Overall 0.42365312576293945 \n",
+ "Fold 2/2, Epoch 3/4, Running Loss valid 0.7797701954841614, Time 0.11433148384094238, Overall 0.42365312576293945 \n",
+ "EarlyStopping counter: 2 out of 5\n",
+ "Fold 2/2, Epoch 4/4, Minibatch 0/0, Phase train, Running Loss train 5.8609395027160645, Time 0.13337326049804688, Overall 0.44269490242004395 \n",
+ "Fold 2/2, Epoch 4/4, Running Loss train 0.3256077501508925, Time 0.13337326049804688, Overall 0.44269490242004395 \n",
+ "Fold 2/2, Epoch 4/4, Minibatch 0/0, Phase valid, Running Loss valid 6.935448169708252, Time 0.14133477210998535, Overall 0.4506564140319824 \n",
+ "Fold 2/2, Epoch 4/4, Running Loss valid 0.8669310212135315, Time 0.14133477210998535, Overall 0.4506564140319824 \n",
+ "EarlyStopping counter: 3 out of 5\n",
+ "It took 0.4577000141143799 to train the model.\n",
+ "Figure(640x480)\n",
+ "Figure(640x480)\n",
+ "Figure(640x480)\n",
+ "Building pytrec_eval input for 5 instances ...\n",
+ "Evaluating {'map_cut_2,5,10', 'ndcg_cut_2,5,10', 'P_2,5,10', 'recall_2,5,10'} ...\n",
+ "Averaging ...\n",
+ "Building pytrec_eval input for 5 instances ...\n",
+ "Evaluating {'map_cut_2,5,10', 'ndcg_cut_2,5,10', 'P_2,5,10', 'recall_2,5,10'} ...\n",
+ "Averaging ...\n",
+ "Building pytrec_eval input for 5 instances ...\n",
+ "Evaluating {'map_cut_2,5,10', 'ndcg_cut_2,5,10', 'P_2,5,10', 'recall_2,5,10'} ...\n",
+ "Averaging ...\n",
+ "####################################################################################################\n",
+ "Reranking for the baseline ../output/toy.dblp.v12.json/bnn/t31.s10.m13.l[128].lr0.1.b128.e5.nns3.nsunigram_b.s1.lossSL/t31.s10.m13.l[128].lr0.1.b128.e5.nns3.nsunigram_b.s1.lossSL/f0.test.pred ...\n",
+ "Loading popularity labels ...\n",
+ "Loading reranking results ...\n",
+ "Loading fairness evaluation results before and after reranking ...\n",
+ "Loading fairness results failed! Evaluating fairness metric {'ndkl'} ...\n",
+ "Loading utility metric evaluation results before and after reranking ...\n",
+ "Pipeline for the baseline ../output/toy.dblp.v12.json/bnn/t31.s10.m13.l[128].lr0.1.b128.e5.nns3.nsunigram_b.s1.lossSL/t31.s10.m13.l[128].lr0.1.b128.e5.nns3.nsunigram_b.s1.lossSL/f0.test.pred completed by <_MainProcess name='MainProcess' parent=None started>! 0.04403567314147949\n",
+ "####################################################################################################\n",
+ "####################################################################################################\n",
+ "Reranking for the baseline ../output/toy.dblp.v12.json/bnn/t31.s10.m13.l[128].lr0.1.b128.e5.nns3.nsunigram_b.s1.lossSL/t31.s10.m13.l[128].lr0.1.b128.e5.nns3.nsunigram_b.s1.lossSL/f1.test.pred ...\n",
+ "Loading popularity labels ...\n",
+ "Loading reranking results ...\n",
+ "Loading fairness evaluation results before and after reranking ...\n",
+ "Loading fairness results failed! Evaluating fairness metric {'ndkl'} ...\n",
+ "Loading utility metric evaluation results before and after reranking ...\n",
+ "Pipeline for the baseline ../output/toy.dblp.v12.json/bnn/t31.s10.m13.l[128].lr0.1.b128.e5.nns3.nsunigram_b.s1.lossSL/t31.s10.m13.l[128].lr0.1.b128.e5.nns3.nsunigram_b.s1.lossSL/f1.test.pred completed by <_MainProcess name='MainProcess' parent=None started>! 0.041002511978149414\n",
+ "####################################################################################################\n",
+ "####################################################################################################\n",
+ "Reranking for the baseline ../output/toy.dblp.v12.json/bnn/t31.s10.m13.l[128].lr0.1.b128.e5.nns3.nsunigram_b.s1.lossSL/t31.s10.m13.l[128].lr0.1.b128.e5.nns3.nsunigram_b.s1.lossSL/f2.test.pred ...\n",
+ "Loading popularity labels ...\n",
+ "Loading reranking results ...\n",
+ "Loading fairness evaluation results before and after reranking ...\n",
+ "Loading fairness results failed! Evaluating fairness metric {'ndkl'} ...\n",
+ "Loading utility metric evaluation results before and after reranking ...\n",
+ "Pipeline for the baseline ../output/toy.dblp.v12.json/bnn/t31.s10.m13.l[128].lr0.1.b128.e5.nns3.nsunigram_b.s1.lossSL/t31.s10.m13.l[128].lr0.1.b128.e5.nns3.nsunigram_b.s1.lossSL/f2.test.pred completed by <_MainProcess name='MainProcess' parent=None started>! 0.0410001277923584\n",
+ "####################################################################################################\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "c:\\Users\\tea-n_\\anaconda3\\envs\\opentf\\lib\\site-packages\\torch\\optim\\lr_scheduler.py:28: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate.\n",
+ " warnings.warn(\"The verbose parameter is deprecated. Please use get_last_lr() \"\n",
+ "\n",
+ " 0%| | 0/5 [00:00
\n",
+ "
![](\"./new-class-diagram.png\")
\n",
"\n",
"To integrate a new model into the baseline, follow the `Ntf` class. Ideally, only the `learn()` method should be overriden, with `eval()` remaining the same for fair comparison."
]
@@ -305,39 +566,63 @@
"metadata": {},
"outputs": [],
"source": [
- "import numpy as np\n",
- "import keras\n",
- "import pandas as pd\n",
- "\n",
+ "from sklearn.neural_network import MLPClassifier\n",
"from mdl.ntf import Ntf\n",
- "from mdl.cds import TFDataset\n",
- "from cmn.team import Team\n",
- "from cmn.tools import merge_teams_by_skills\n",
- "from mdl.cds import SuperlossDataset\n",
- "from mdl.superloss import SuperLoss\n",
+ "import pickle\n",
"\n",
"class Random(Ntf):\n",
" def __init__(self):\n",
" super(Random, self).__init__()\n",
" \n",
" def init(self):\n",
- " self.model = keras.Sequential()\n",
+ " self.model = MLPClassifier() # empty network\n",
+ " return self\n",
"\n",
" def learn(self, splits, indexes, vecs, params, prev_model, output):\n",
- " input_size = vecs['skill'].shape[1]\n",
- " output_size = len(indexes['i2c'])\n",
+ " # splits, vecs, and indexes are components of the preprocessed data\n",
+ " # params come from param.py\n",
"\n",
" for foldidx in splits['folds'].keys():\n",
- " self.init(input_size=input_size, output_size=output_size)\n",
- " if prev_model: keras.saving.load_model((prev_model[foldidx]))\n",
- " \n",
- " keras.saving.save_model(self.model, f\"{output}/state_dict_model.f{foldidx}.pt\")"
+ " self.init()\n",
+ " with open(f\"{output}/state_dict_model.f{foldidx}.pt\",\"wb\") as f: pickle.dump(self.model,f)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Additional Resources\n",
+ "- [`OpeNTF` codebase](https://github.com/fani-lab/OpeNTF/tree/main)\n",
+ "- [`Adila` codebase](https://github.com/fani-lab/adila)\n",
+ "- [`vivaFemme` codebase](https://github.com/fani-lab/OpeNTF/tree/vivaFemme)\n",
+ "- [Streaming Training Strategy codebase](https://github.com/fani-lab/OpeNTF/tree/ecir24)\n",
+ "- [Tutorial Website and Materials](https://fani-lab.github.io/OpeNTF/tutorial/umap24/)\n",
+ " - [`OpeNTF` paper](https://doi.org/10.1145/3511808.3557526)\n",
+ " - [`Adila` paper](https://doi.org/10.1007/978-3-031-37249-0_9)\n",
+ " - [`vivaFemme` paper](https://hosseinfani.github.io/res/papers/2024_BIAS_SIGIR_vivaFemme_Mitigating_Gender_Bias_in_Neural_Team_Recommendation_via_Female-Advocate_Loss_Regularization.pdf)\n",
+ " - [Streaming Training Strategy paper](https://link.springer.com/chapter/10.1007/978-3-031-56027-9_20)\n",
+ "\n",
+ ""
]
}
],
"metadata": {
+ "kernelspec": {
+ "display_name": "opentf",
+ "language": "python",
+ "name": "python3"
+ },
"language_info": {
- "name": "python"
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.19"
}
},
"nbformat": 4,
diff --git a/docs/tutorial/umap24/jupyter/Temporal.ipynb b/docs/tutorial/umap24/jupyter/Temporal.ipynb
new file mode 100644
index 00000000..0f44a0de
--- /dev/null
+++ b/docs/tutorial/umap24/jupyter/Temporal.ipynb
@@ -0,0 +1,907 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Temporal Neural Team Formation\n",
+ "Unlike `Adila` and `vivaFemme`, this extension of the `OpeNTF` framework focuses on the temporal aspect of neural team formation.\n",
+ "\n",
+ "Current neural team formation methods often overlook the temporality of the experts within teams, such as the evolution of the expert network and skillsets. These methods often assumes teams to be independent and identically distributed (bag of teams). \n",
+ "\n",
+ "Specifically, this extension propose a streaming training strategy that considers this temporal aspect in neural-based team formation methods. A latent space models the experts' vector position in terms of skills and collaboration ties, allowing the models to also predict future vector positions. These information are used for the neural models to predict future successful teams, taking into consideration of the predicted collaboration ties and evolved skillsets.\n",
+ "\n",
+ "![](\"./streaming-training.png\")
\n",
+ "\n",
+ "For models within the baseline, models with the `t` prefix are trained with said strategy. For example, `tntf` is the subclass of `ntf`.\n",
+ "\n",
+ "# Quickstart\n",
+ "The extension follows the same format as `OpeNTF`. To run the system, navigate to the `ecir24` branch first."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[WinError 2] The system cannot find the file specified: 'src'\n",
+ "c:\\Users\\tea-n_\\Documents\\GitHub\\OpeNTF framework\\Temporal\\src\n",
+ "Loading sparse matrices from ./../data/preprocessed/dblp/toy.dblp.v12.json/teamsvecs.pkl ...\n",
+ "Loading indexes pickle from ./../data/preprocessed/dblp/toy.dblp.v12.json/indexes.pkl ...\n",
+ "It took 0.0897526741027832 seconds to load from the pickles.\n",
+ "It took 0.0899958610534668 seconds to load the sparse matrices.\n",
+ "Running for (dataset, model): (dblp, bnn) ... \n",
+ "Fold 0/4, Epoch 0/19, Minibatch 0/0, Phase train, Running Loss train 311.3135070800781, Time 0.0373692512512207, Overall 3.0487139225006104 \n",
+ "Fold 0/4, Epoch 0/19, Running Loss train 15.565675354003906, Time 0.0373692512512207, Overall 3.0487139225006104 \n",
+ "Fold 0/4, Epoch 0/19, Minibatch 0/0, Phase valid, Running Loss valid 98.90568542480469, Time 0.04854321479797363, Overall 3.0598878860473633 \n",
+ "Fold 0/4, Epoch 0/19, Running Loss valid 16.484280904134113, Time 0.04954648017883301, Overall 3.0608911514282227 \n",
+ "Fold 0/4, Epoch 1/19, Minibatch 0/0, Phase train, Running Loss train 267.15325927734375, Time 0.08184194564819336, Overall 3.093186616897583 \n",
+ "Fold 0/4, Epoch 1/19, Running Loss train 13.357662963867188, Time 0.08184194564819336, Overall 3.093186616897583 \n",
+ "Fold 0/4, Epoch 1/19, Minibatch 0/0, Phase valid, Running Loss valid 69.3178939819336, Time 0.09201169013977051, Overall 3.10335636138916 \n",
+ "Fold 0/4, Epoch 1/19, Running Loss valid 11.552982330322266, Time 0.09301066398620605, Overall 3.1043553352355957 \n",
+ "Fold 0/4, Epoch 2/19, Minibatch 0/0, Phase train, Running Loss train 249.93606567382812, Time 0.12201094627380371, Overall 3.1333556175231934 \n",
+ "Fold 0/4, Epoch 2/19, Running Loss train 12.496803283691406, Time 0.12201094627380371, Overall 3.1333556175231934 \n",
+ "Fold 0/4, Epoch 2/19, Minibatch 0/0, Phase valid, Running Loss valid 70.90058135986328, Time 0.1330108642578125, Overall 3.144355535507202 \n",
+ "Fold 0/4, Epoch 2/19, Running Loss valid 11.816763559977213, Time 0.1330108642578125, Overall 3.144355535507202 \n",
+ "Fold 0/4, Epoch 3/19, Minibatch 0/0, Phase train, Running Loss train 385.6123962402344, Time 0.16219019889831543, Overall 3.173534870147705 \n",
+ "Fold 0/4, Epoch 3/19, Running Loss train 19.280619812011718, Time 0.16219019889831543, Overall 3.173534870147705 \n",
+ "Fold 0/4, Epoch 3/19, Minibatch 0/0, Phase valid, Running Loss valid 137.45327758789062, Time 0.17350268363952637, Overall 3.184847354888916 \n",
+ "Fold 0/4, Epoch 3/19, Running Loss valid 22.90887959798177, Time 0.17350268363952637, Overall 3.184847354888916 \n",
+ "Fold 0/4, Epoch 4/19, Minibatch 0/0, Phase train, Running Loss train 322.2874755859375, Time 0.20266938209533691, Overall 3.2140140533447266 \n",
+ "Fold 0/4, Epoch 4/19, Running Loss train 16.114373779296876, Time 0.20266938209533691, Overall 3.2140140533447266 \n",
+ "Fold 0/4, Epoch 4/19, Minibatch 0/0, Phase valid, Running Loss valid 95.04570007324219, Time 0.21267032623291016, Overall 3.2240149974823 \n",
+ "Fold 0/4, Epoch 4/19, Running Loss valid 15.840950012207031, Time 0.213670015335083, Overall 3.2250146865844727 \n",
+ "Fold 0/4, Epoch 5/19, Minibatch 0/0, Phase train, Running Loss train 360.58197021484375, Time 0.24289298057556152, Overall 3.254237651824951 \n",
+ "Fold 0/4, Epoch 5/19, Running Loss train 18.029098510742188, Time 0.24289298057556152, Overall 3.254237651824951 \n",
+ "Fold 0/4, Epoch 5/19, Minibatch 0/0, Phase valid, Running Loss valid 38.85438919067383, Time 0.2540547847747803, Overall 3.26539945602417 \n",
+ "Fold 0/4, Epoch 5/19, Running Loss valid 6.475731531778972, Time 0.2540547847747803, Overall 3.26539945602417 \n",
+ "Fold 0/4, Epoch 6/19, Minibatch 0/0, Phase train, Running Loss train 375.97003173828125, Time 0.2873706817626953, Overall 3.298715353012085 \n",
+ "Fold 0/4, Epoch 6/19, Running Loss train 18.798501586914064, Time 0.2873706817626953, Overall 3.298715353012085 \n",
+ "Fold 0/4, Epoch 6/19, Minibatch 0/0, Phase valid, Running Loss valid 121.81012725830078, Time 0.29990577697753906, Overall 3.3112504482269287 \n",
+ "Fold 0/4, Epoch 6/19, Running Loss valid 20.301687876383465, Time 0.29990577697753906, Overall 3.3112504482269287 \n",
+ "Fold 0/4, Epoch 7/19, Minibatch 0/0, Phase train, Running Loss train 227.5210723876953, Time 0.33030104637145996, Overall 3.3416457176208496 \n",
+ "Fold 0/4, Epoch 7/19, Running Loss train 11.376053619384766, Time 0.33030104637145996, Overall 3.3416457176208496 \n",
+ "Fold 0/4, Epoch 7/19, Minibatch 0/0, Phase valid, Running Loss valid 106.88151550292969, Time 0.34150195121765137, Overall 3.352846622467041 \n",
+ "Fold 0/4, Epoch 7/19, Running Loss valid 17.81358591715495, Time 0.3425455093383789, Overall 3.3538901805877686 \n",
+ "Fold 0/4, Epoch 8/19, Minibatch 0/0, Phase train, Running Loss train 233.68914794921875, Time 0.3713979721069336, Overall 3.3827426433563232 \n",
+ "Fold 0/4, Epoch 8/19, Running Loss train 11.684457397460937, Time 0.372438907623291, Overall 3.3837835788726807 \n",
+ "Fold 0/4, Epoch 8/19, Minibatch 0/0, Phase valid, Running Loss valid 69.06804656982422, Time 0.383434534072876, Overall 3.3947792053222656 \n",
+ "Fold 0/4, Epoch 8/19, Running Loss valid 11.511341094970703, Time 0.383434534072876, Overall 3.3947792053222656 \n",
+ "Fold 0/4, Epoch 9/19, Minibatch 0/0, Phase train, Running Loss train 185.38743591308594, Time 0.4135715961456299, Overall 3.4249162673950195 \n",
+ "Fold 0/4, Epoch 9/19, Running Loss train 9.269371795654298, Time 0.4135715961456299, Overall 3.4249162673950195 \n",
+ "Fold 0/4, Epoch 9/19, Minibatch 0/0, Phase valid, Running Loss valid 69.32157897949219, Time 0.4235715866088867, Overall 3.4349162578582764 \n",
+ "Fold 0/4, Epoch 9/19, Running Loss valid 11.553596496582031, Time 0.4235715866088867, Overall 3.4349162578582764 \n",
+ "Fold 0/4, Epoch 10/19, Minibatch 0/0, Phase train, Running Loss train 251.98092651367188, Time 0.4583256244659424, Overall 3.469670295715332 \n",
+ "Fold 0/4, Epoch 10/19, Running Loss train 12.599046325683593, Time 0.4583256244659424, Overall 3.469670295715332 \n",
+ "Fold 0/4, Epoch 10/19, Minibatch 0/0, Phase valid, Running Loss valid 88.49652099609375, Time 0.4703686237335205, Overall 3.48171329498291 \n",
+ "Fold 0/4, Epoch 10/19, Running Loss valid 14.749420166015625, Time 0.4704747200012207, Overall 3.4818193912506104 \n",
+ "Fold 0/4, Epoch 11/19, Minibatch 0/0, Phase train, Running Loss train 262.6159362792969, Time 0.49866747856140137, Overall 3.510012149810791 \n",
+ "Fold 0/4, Epoch 11/19, Running Loss train 13.130796813964844, Time 0.49866747856140137, Overall 3.510012149810791 \n",
+ "Fold 0/4, Epoch 11/19, Minibatch 0/0, Phase valid, Running Loss valid 30.562679290771484, Time 0.5102908611297607, Overall 3.5216355323791504 \n",
+ "Fold 0/4, Epoch 11/19, Running Loss valid 5.093779881795247, Time 0.5102908611297607, Overall 3.5216355323791504 \n",
+ "Fold 0/4, Epoch 12/19, Minibatch 0/0, Phase train, Running Loss train 121.70864868164062, Time 0.5402348041534424, Overall 3.551579475402832 \n",
+ "Fold 0/4, Epoch 12/19, Running Loss train 6.085432434082032, Time 0.5402348041534424, Overall 3.551579475402832 \n",
+ "Fold 0/4, Epoch 12/19, Minibatch 0/0, Phase valid, Running Loss valid 59.493865966796875, Time 0.5519816875457764, Overall 3.563326358795166 \n",
+ "Fold 0/4, Epoch 12/19, Running Loss valid 9.915644327799479, Time 0.5519816875457764, Overall 3.563326358795166 \n",
+ "Fold 0/4, Epoch 13/19, Minibatch 0/0, Phase train, Running Loss train 212.9227752685547, Time 0.5823390483856201, Overall 3.5936837196350098 \n",
+ "Fold 0/4, Epoch 13/19, Running Loss train 10.646138763427734, Time 0.5823390483856201, Overall 3.5936837196350098 \n",
+ "Fold 0/4, Epoch 13/19, Minibatch 0/0, Phase valid, Running Loss valid 43.08524703979492, Time 0.59385085105896, Overall 3.6051955223083496 \n",
+ "Fold 0/4, Epoch 13/19, Running Loss valid 7.180874506632487, Time 0.59385085105896, Overall 3.6051955223083496 \n",
+ "Fold 0/4, Epoch 14/19, Minibatch 0/0, Phase train, Running Loss train 173.87652587890625, Time 0.6245613098144531, Overall 3.6359059810638428 \n",
+ "Fold 0/4, Epoch 14/19, Running Loss train 8.693826293945312, Time 0.6255567073822021, Overall 3.636901378631592 \n",
+ "Fold 0/4, Epoch 14/19, Minibatch 0/0, Phase valid, Running Loss valid 37.356422424316406, Time 0.6375594139099121, Overall 3.6489040851593018 \n",
+ "Fold 0/4, Epoch 14/19, Running Loss valid 6.226070404052734, Time 0.6375594139099121, Overall 3.6489040851593018 \n",
+ "Fold 0/4, Epoch 15/19, Minibatch 0/0, Phase train, Running Loss train 230.41909790039062, Time 0.6735243797302246, Overall 3.6848690509796143 \n",
+ "Fold 0/4, Epoch 15/19, Running Loss train 11.520954895019532, Time 0.6735243797302246, Overall 3.6848690509796143 \n",
+ "Fold 0/4, Epoch 15/19, Minibatch 0/0, Phase valid, Running Loss valid 63.21870422363281, Time 0.6855218410491943, Overall 3.696866512298584 \n",
+ "Fold 0/4, Epoch 15/19, Running Loss valid 10.536450703938803, Time 0.6865220069885254, Overall 3.697866678237915 \n",
+ "Fold 0/4, Epoch 16/19, Minibatch 0/0, Phase train, Running Loss train 244.5215606689453, Time 0.7195217609405518, Overall 3.7308664321899414 \n",
+ "Fold 0/4, Epoch 16/19, Running Loss train 12.226078033447266, Time 0.7195217609405518, Overall 3.7308664321899414 \n",
+ "Fold 0/4, Epoch 16/19, Minibatch 0/0, Phase valid, Running Loss valid 37.26233673095703, Time 0.7301459312438965, Overall 3.741490602493286 \n",
+ "Fold 0/4, Epoch 16/19, Running Loss valid 6.210389455159505, Time 0.7311515808105469, Overall 3.7424962520599365 \n",
+ "Fold 0/4, Epoch 17/19, Minibatch 0/0, Phase train, Running Loss train 229.6649627685547, Time 0.7588589191436768, Overall 3.7702035903930664 \n",
+ "Fold 0/4, Epoch 17/19, Running Loss train 11.483248138427735, Time 0.7588589191436768, Overall 3.7702035903930664 \n",
+ "Fold 0/4, Epoch 17/19, Minibatch 0/0, Phase valid, Running Loss valid 30.66425895690918, Time 0.7702169418334961, Overall 3.7815616130828857 \n",
+ "Fold 0/4, Epoch 17/19, Running Loss valid 5.11070982615153, Time 0.770308256149292, Overall 3.7816529273986816 \n",
+ "Fold 0/4, Epoch 18/19, Minibatch 0/0, Phase train, Running Loss train 237.21221923828125, Time 0.7988123893737793, Overall 3.810157060623169 \n",
+ "Fold 0/4, Epoch 18/19, Running Loss train 11.860610961914062, Time 0.7998149394989014, Overall 3.811159610748291 \n",
+ "Fold 0/4, Epoch 18/19, Minibatch 0/0, Phase valid, Running Loss valid 74.39366912841797, Time 0.8098163604736328, Overall 3.8211610317230225 \n",
+ "Fold 0/4, Epoch 18/19, Running Loss valid 12.398944854736328, Time 0.8098163604736328, Overall 3.8211610317230225 \n",
+ "Fold 0/4, Epoch 19/19, Minibatch 0/0, Phase train, Running Loss train 159.50411987304688, Time 0.839747428894043, Overall 3.8510921001434326 \n",
+ "Fold 0/4, Epoch 19/19, Running Loss train 7.975205993652343, Time 0.839747428894043, Overall 3.8510921001434326 \n",
+ "Fold 0/4, Epoch 19/19, Minibatch 0/0, Phase valid, Running Loss valid 76.92823028564453, Time 0.8543143272399902, Overall 3.86565899848938 \n",
+ "Fold 0/4, Epoch 19/19, Running Loss valid 12.821371714274088, Time 0.8543143272399902, Overall 3.86565899848938 \n",
+ "Fold 1/4, Epoch 0/19, Minibatch 0/0, Phase train, Running Loss train 319.09130859375, Time 0.029447078704833984, Overall 3.9114768505096436 \n",
+ "Fold 1/4, Epoch 0/19, Running Loss train 15.19482421875, Time 0.029447078704833984, Overall 3.9114768505096436 \n",
+ "Fold 1/4, Epoch 0/19, Minibatch 0/0, Phase valid, Running Loss valid 29.68271255493164, Time 0.04044485092163086, Overall 3.9224746227264404 \n",
+ "Fold 1/4, Epoch 0/19, Running Loss valid 5.936542510986328, Time 0.04044485092163086, Overall 3.9224746227264404 \n",
+ "Fold 1/4, Epoch 1/19, Minibatch 0/0, Phase train, Running Loss train 307.4185791015625, Time 0.08613848686218262, Overall 3.968168258666992 \n",
+ "Fold 1/4, Epoch 1/19, Running Loss train 14.638979957217263, Time 0.08613848686218262, Overall 3.968168258666992 \n",
+ "Fold 1/4, Epoch 1/19, Minibatch 0/0, Phase valid, Running Loss valid 88.34307098388672, Time 0.10435342788696289, Overall 3.9863831996917725 \n",
+ "Fold 1/4, Epoch 1/19, Running Loss valid 17.668614196777344, Time 0.10435342788696289, Overall 3.9863831996917725 \n",
+ "Fold 1/4, Epoch 2/19, Minibatch 0/0, Phase train, Running Loss train 233.86709594726562, Time 0.14341139793395996, Overall 4.0254411697387695 \n",
+ "Fold 1/4, Epoch 2/19, Running Loss train 11.13652837844122, Time 0.14341139793395996, Overall 4.0254411697387695 \n",
+ "Fold 1/4, Epoch 2/19, Minibatch 0/0, Phase valid, Running Loss valid 103.17047882080078, Time 0.1534106731414795, Overall 4.035440444946289 \n",
+ "Fold 1/4, Epoch 2/19, Running Loss valid 20.634095764160158, Time 0.1534106731414795, Overall 4.035440444946289 \n",
+ "Fold 1/4, Epoch 3/19, Minibatch 0/0, Phase train, Running Loss train 321.0475769042969, Time 0.18282270431518555, Overall 4.064852476119995 \n",
+ "Fold 1/4, Epoch 3/19, Running Loss train 15.287979852585565, Time 0.1838221549987793, Overall 4.065851926803589 \n",
+ "Fold 1/4, Epoch 3/19, Minibatch 0/0, Phase valid, Running Loss valid 83.61467742919922, Time 0.19382190704345703, Overall 4.075851678848267 \n",
+ "Fold 1/4, Epoch 3/19, Running Loss valid 16.722935485839844, Time 0.19382190704345703, Overall 4.075851678848267 \n",
+ "Fold 1/4, Epoch 4/19, Minibatch 0/0, Phase train, Running Loss train 323.4862976074219, Time 0.22208499908447266, Overall 4.104114770889282 \n",
+ "Fold 1/4, Epoch 4/19, Running Loss train 15.404109409877233, Time 0.2230527400970459, Overall 4.1050825119018555 \n",
+ "Fold 1/4, Epoch 4/19, Minibatch 0/0, Phase valid, Running Loss valid 68.5489501953125, Time 0.23227572441101074, Overall 4.11430549621582 \n",
+ "Fold 1/4, Epoch 4/19, Running Loss valid 13.7097900390625, Time 0.23227572441101074, Overall 4.11430549621582 \n",
+ "Fold 1/4, Epoch 5/19, Minibatch 0/0, Phase train, Running Loss train 326.1994323730469, Time 0.2617368698120117, Overall 4.143766641616821 \n",
+ "Fold 1/4, Epoch 5/19, Running Loss train 15.533306303478422, Time 0.2617368698120117, Overall 4.143766641616821 \n",
+ "Fold 1/4, Epoch 5/19, Minibatch 0/0, Phase valid, Running Loss valid 64.61378479003906, Time 0.27084898948669434, Overall 4.152878761291504 \n",
+ "Fold 1/4, Epoch 5/19, Running Loss valid 12.922756958007813, Time 0.27084898948669434, Overall 4.152878761291504 \n",
+ "Fold 1/4, Epoch 6/19, Minibatch 0/0, Phase train, Running Loss train 226.34429931640625, Time 0.3011620044708252, Overall 4.183191776275635 \n",
+ "Fold 1/4, Epoch 6/19, Running Loss train 10.778299967447916, Time 0.3011620044708252, Overall 4.183191776275635 \n",
+ "Fold 1/4, Epoch 6/19, Minibatch 0/0, Phase valid, Running Loss valid 93.72303009033203, Time 0.31114721298217773, Overall 4.193176984786987 \n",
+ "Fold 1/4, Epoch 6/19, Running Loss valid 18.744606018066406, Time 0.31114721298217773, Overall 4.193176984786987 \n",
+ "Fold 1/4, Epoch 7/19, Minibatch 0/0, Phase train, Running Loss train 273.42742919921875, Time 0.3408372402191162, Overall 4.222867012023926 \n",
+ "Fold 1/4, Epoch 7/19, Running Loss train 13.020353771391369, Time 0.3408372402191162, Overall 4.222867012023926 \n",
+ "Fold 1/4, Epoch 7/19, Minibatch 0/0, Phase valid, Running Loss valid 48.30699920654297, Time 0.35010719299316406, Overall 4.232136964797974 \n",
+ "Fold 1/4, Epoch 7/19, Running Loss valid 9.661399841308594, Time 0.35010719299316406, Overall 4.232136964797974 \n",
+ "Fold 1/4, Epoch 8/19, Minibatch 0/0, Phase train, Running Loss train 231.58108520507812, Time 0.3795895576477051, Overall 4.261619329452515 \n",
+ "Fold 1/4, Epoch 8/19, Running Loss train 11.027670724051339, Time 0.3795895576477051, Overall 4.261619329452515 \n",
+ "Fold 1/4, Epoch 8/19, Minibatch 0/0, Phase valid, Running Loss valid 115.33173370361328, Time 0.38955235481262207, Overall 4.271582126617432 \n",
+ "Fold 1/4, Epoch 8/19, Running Loss valid 23.066346740722658, Time 0.39061450958251953, Overall 4.272644281387329 \n",
+ "Fold 1/4, Epoch 9/19, Minibatch 0/0, Phase train, Running Loss train 222.4735107421875, Time 0.42112302780151367, Overall 4.303152799606323 \n",
+ "Fold 1/4, Epoch 9/19, Running Loss train 10.593976702008929, Time 0.42112302780151367, Overall 4.303152799606323 \n",
+ "Fold 1/4, Epoch 9/19, Minibatch 0/0, Phase valid, Running Loss valid 28.84703254699707, Time 0.43172240257263184, Overall 4.313752174377441 \n",
+ "Fold 1/4, Epoch 9/19, Running Loss valid 5.769406509399414, Time 0.43172240257263184, Overall 4.313752174377441 \n",
+ "Fold 1/4, Epoch 10/19, Minibatch 0/0, Phase train, Running Loss train 133.0574188232422, Time 0.4638636112213135, Overall 4.345893383026123 \n",
+ "Fold 1/4, Epoch 10/19, Running Loss train 6.336067563011532, Time 0.4638636112213135, Overall 4.345893383026123 \n",
+ "Fold 1/4, Epoch 10/19, Minibatch 0/0, Phase valid, Running Loss valid 79.88641357421875, Time 0.4740266799926758, Overall 4.356056451797485 \n",
+ "Fold 1/4, Epoch 10/19, Running Loss valid 15.97728271484375, Time 0.475034236907959, Overall 4.3570640087127686 \n",
+ "Fold 1/4, Epoch 11/19, Minibatch 0/0, Phase train, Running Loss train 236.9170379638672, Time 0.512016773223877, Overall 4.3940465450286865 \n",
+ "Fold 1/4, Epoch 11/19, Running Loss train 11.281763712565104, Time 0.512016773223877, Overall 4.3940465450286865 \n",
+ "Fold 1/4, Epoch 11/19, Minibatch 0/0, Phase valid, Running Loss valid 47.821075439453125, Time 0.523017168045044, Overall 4.4050469398498535 \n",
+ "Fold 1/4, Epoch 11/19, Running Loss valid 9.564215087890625, Time 0.523017168045044, Overall 4.4050469398498535 \n",
+ "Fold 1/4, Epoch 12/19, Minibatch 0/0, Phase train, Running Loss train 202.40614318847656, Time 0.5558984279632568, Overall 4.437928199768066 \n",
+ "Fold 1/4, Epoch 12/19, Running Loss train 9.638387770879836, Time 0.5558984279632568, Overall 4.437928199768066 \n",
+ "Fold 1/4, Epoch 12/19, Minibatch 0/0, Phase valid, Running Loss valid 68.35807800292969, Time 0.5669338703155518, Overall 4.448963642120361 \n",
+ "Fold 1/4, Epoch 12/19, Running Loss valid 13.671615600585938, Time 0.5669338703155518, Overall 4.448963642120361 \n",
+ "Fold 1/4, Epoch 13/19, Minibatch 0/0, Phase train, Running Loss train 204.5157470703125, Time 0.5969338417053223, Overall 4.478963613510132 \n",
+ "Fold 1/4, Epoch 13/19, Running Loss train 9.73884509858631, Time 0.5969338417053223, Overall 4.478963613510132 \n",
+ "Fold 1/4, Epoch 13/19, Minibatch 0/0, Phase valid, Running Loss valid 51.10749053955078, Time 0.6068985462188721, Overall 4.488928318023682 \n",
+ "Fold 1/4, Epoch 13/19, Running Loss valid 10.221498107910156, Time 0.6068985462188721, Overall 4.488928318023682 \n",
+ "Fold 1/4, Epoch 14/19, Minibatch 0/0, Phase train, Running Loss train 219.52896118164062, Time 0.6326746940612793, Overall 4.514704465866089 \n",
+ "Fold 1/4, Epoch 14/19, Running Loss train 10.453760056268601, Time 0.6326746940612793, Overall 4.514704465866089 \n",
+ "Fold 1/4, Epoch 14/19, Minibatch 0/0, Phase valid, Running Loss valid 22.572038650512695, Time 0.647716760635376, Overall 4.5297465324401855 \n",
+ "Fold 1/4, Epoch 14/19, Running Loss valid 4.514407730102539, Time 0.647716760635376, Overall 4.5297465324401855 \n",
+ "Fold 1/4, Epoch 15/19, Minibatch 0/0, Phase train, Running Loss train 235.83526611328125, Time 0.6801528930664062, Overall 4.562182664871216 \n",
+ "Fold 1/4, Epoch 15/19, Running Loss train 11.230250767299108, Time 0.6811230182647705, Overall 4.56315279006958 \n",
+ "Fold 1/4, Epoch 15/19, Minibatch 0/0, Phase valid, Running Loss valid 61.23127746582031, Time 0.6901402473449707, Overall 4.57217001914978 \n",
+ "Fold 1/4, Epoch 15/19, Running Loss valid 12.246255493164062, Time 0.6901402473449707, Overall 4.57217001914978 \n",
+ "Fold 1/4, Epoch 16/19, Minibatch 0/0, Phase train, Running Loss train 109.59815979003906, Time 0.7197482585906982, Overall 4.601778030395508 \n",
+ "Fold 1/4, Epoch 16/19, Running Loss train 5.21895999000186, Time 0.7197482585906982, Overall 4.601778030395508 \n",
+ "Fold 1/4, Epoch 16/19, Minibatch 0/0, Phase valid, Running Loss valid 88.19727325439453, Time 0.7300295829772949, Overall 4.6120593547821045 \n",
+ "Fold 1/4, Epoch 16/19, Running Loss valid 17.639454650878907, Time 0.7300295829772949, Overall 4.6120593547821045 \n",
+ "Fold 1/4, Epoch 17/19, Minibatch 0/0, Phase train, Running Loss train 247.18582153320312, Time 0.7610154151916504, Overall 4.64304518699646 \n",
+ "Fold 1/4, Epoch 17/19, Running Loss train 11.770753406343006, Time 0.7620849609375, Overall 4.64411473274231 \n",
+ "Fold 1/4, Epoch 17/19, Minibatch 0/0, Phase valid, Running Loss valid 40.05247116088867, Time 0.7720208168029785, Overall 4.654050588607788 \n",
+ "Fold 1/4, Epoch 17/19, Running Loss valid 8.010494232177734, Time 0.7720208168029785, Overall 4.654050588607788 \n",
+ "Fold 1/4, Epoch 18/19, Minibatch 0/0, Phase train, Running Loss train 184.01002502441406, Time 0.803257942199707, Overall 4.685287714004517 \n",
+ "Fold 1/4, Epoch 18/19, Running Loss train 8.762382144019718, Time 0.803257942199707, Overall 4.685287714004517 \n",
+ "Fold 1/4, Epoch 18/19, Minibatch 0/0, Phase valid, Running Loss valid 46.39714813232422, Time 0.8142645359039307, Overall 4.69629430770874 \n",
+ "Fold 1/4, Epoch 18/19, Running Loss valid 9.279429626464843, Time 0.8142645359039307, Overall 4.69629430770874 \n",
+ "Fold 1/4, Epoch 19/19, Minibatch 0/0, Phase train, Running Loss train 133.8429412841797, Time 0.8454403877258301, Overall 4.72747015953064 \n",
+ "Fold 1/4, Epoch 19/19, Running Loss train 6.373473394484747, Time 0.8464453220367432, Overall 4.728475093841553 \n",
+ "Fold 1/4, Epoch 19/19, Minibatch 0/0, Phase valid, Running Loss valid 22.802549362182617, Time 0.8560645580291748, Overall 4.738094329833984 \n",
+ "Fold 1/4, Epoch 19/19, Running Loss valid 4.560509872436524, Time 0.8560645580291748, Overall 4.738094329833984 \n",
+ "Fold 2/4, Epoch 0/19, Minibatch 0/0, Phase train, Running Loss train 283.9684143066406, Time 0.029220104217529297, Overall 4.7771315574646 \n",
+ "Fold 2/4, Epoch 0/19, Running Loss train 13.522305443173362, Time 0.029220104217529297, Overall 4.7771315574646 \n",
+ "Fold 2/4, Epoch 0/19, Minibatch 0/0, Phase valid, Running Loss valid 54.25079345703125, Time 0.03848838806152344, Overall 4.786399841308594 \n",
+ "Fold 2/4, Epoch 0/19, Running Loss valid 10.85015869140625, Time 0.03848838806152344, Overall 4.786399841308594 \n",
+ "Fold 2/4, Epoch 1/19, Minibatch 0/0, Phase train, Running Loss train 270.843017578125, Time 0.06868481636047363, Overall 4.816596269607544 \n",
+ "Fold 2/4, Epoch 1/19, Running Loss train 12.897286551339286, Time 0.06868481636047363, Overall 4.816596269607544 \n",
+ "Fold 2/4, Epoch 1/19, Minibatch 0/0, Phase valid, Running Loss valid 63.85279846191406, Time 0.0786886215209961, Overall 4.826600074768066 \n",
+ "Fold 2/4, Epoch 1/19, Running Loss valid 12.770559692382813, Time 0.0786886215209961, Overall 4.826600074768066 \n",
+ "Fold 2/4, Epoch 2/19, Minibatch 0/0, Phase train, Running Loss train 284.4129638671875, Time 0.11091089248657227, Overall 4.858822345733643 \n",
+ "Fold 2/4, Epoch 2/19, Running Loss train 13.543474469866071, Time 0.11091089248657227, Overall 4.858822345733643 \n",
+ "Fold 2/4, Epoch 2/19, Minibatch 0/0, Phase valid, Running Loss valid 82.72183990478516, Time 0.12191081047058105, Overall 4.869822263717651 \n",
+ "Fold 2/4, Epoch 2/19, Running Loss valid 16.54436798095703, Time 0.12191081047058105, Overall 4.869822263717651 \n",
+ "Fold 2/4, Epoch 3/19, Minibatch 0/0, Phase train, Running Loss train 309.48089599609375, Time 0.161482572555542, Overall 4.909394025802612 \n",
+ "Fold 2/4, Epoch 3/19, Running Loss train 14.737185523623513, Time 0.161482572555542, Overall 4.909394025802612 \n",
+ "Fold 2/4, Epoch 3/19, Minibatch 0/0, Phase valid, Running Loss valid 96.23108673095703, Time 0.172715425491333, Overall 4.920626878738403 \n",
+ "Fold 2/4, Epoch 3/19, Running Loss valid 19.246217346191408, Time 0.172715425491333, Overall 4.920626878738403 \n",
+ "Fold 2/4, Epoch 4/19, Minibatch 0/0, Phase train, Running Loss train 445.317138671875, Time 0.20254755020141602, Overall 4.950459003448486 \n",
+ "Fold 2/4, Epoch 4/19, Running Loss train 21.205578031994047, Time 0.20308732986450195, Overall 4.950998783111572 \n",
+ "Fold 2/4, Epoch 4/19, Minibatch 0/0, Phase valid, Running Loss valid 79.60504150390625, Time 0.21448493003845215, Overall 4.9623963832855225 \n",
+ "Fold 2/4, Epoch 4/19, Running Loss valid 15.92100830078125, Time 0.21448493003845215, Overall 4.9623963832855225 \n",
+ "Fold 2/4, Epoch 5/19, Minibatch 0/0, Phase train, Running Loss train 348.8349609375, Time 0.25059056282043457, Overall 4.998502016067505 \n",
+ "Fold 2/4, Epoch 5/19, Running Loss train 16.611188616071427, Time 0.25059056282043457, Overall 4.998502016067505 \n",
+ "Fold 2/4, Epoch 5/19, Minibatch 0/0, Phase valid, Running Loss valid 55.42896270751953, Time 0.25998353958129883, Overall 5.007894992828369 \n",
+ "Fold 2/4, Epoch 5/19, Running Loss valid 11.085792541503906, Time 0.25998353958129883, Overall 5.007894992828369 \n",
+ "Fold 2/4, Epoch 6/19, Minibatch 0/0, Phase train, Running Loss train 295.18585205078125, Time 0.29114627838134766, Overall 5.039057731628418 \n",
+ "Fold 2/4, Epoch 6/19, Running Loss train 14.056469145275297, Time 0.29114627838134766, Overall 5.039057731628418 \n",
+ "Fold 2/4, Epoch 6/19, Minibatch 0/0, Phase valid, Running Loss valid 93.25776672363281, Time 0.30097365379333496, Overall 5.048885107040405 \n",
+ "Fold 2/4, Epoch 6/19, Running Loss valid 18.651553344726562, Time 0.30152130126953125, Overall 5.049432754516602 \n",
+ "Fold 2/4, Epoch 7/19, Minibatch 0/0, Phase train, Running Loss train 370.6454162597656, Time 0.3348548412322998, Overall 5.08276629447937 \n",
+ "Fold 2/4, Epoch 7/19, Running Loss train 17.649781726655505, Time 0.3348548412322998, Overall 5.08276629447937 \n",
+ "Fold 2/4, Epoch 7/19, Minibatch 0/0, Phase valid, Running Loss valid 123.43555450439453, Time 0.34590768814086914, Overall 5.0938191413879395 \n",
+ "Fold 2/4, Epoch 7/19, Running Loss valid 24.687110900878906, Time 0.34590768814086914, Overall 5.0938191413879395 \n",
+ "Fold 2/4, Epoch 8/19, Minibatch 0/0, Phase train, Running Loss train 326.4887390136719, Time 0.3763387203216553, Overall 5.124250173568726 \n",
+ "Fold 2/4, Epoch 8/19, Running Loss train 15.547082810174851, Time 0.3763387203216553, Overall 5.124250173568726 \n",
+ "Fold 2/4, Epoch 8/19, Minibatch 0/0, Phase valid, Running Loss valid 72.78269958496094, Time 0.3879725933074951, Overall 5.135884046554565 \n",
+ "Fold 2/4, Epoch 8/19, Running Loss valid 14.556539916992188, Time 0.3879725933074951, Overall 5.135884046554565 \n",
+ "Fold 2/4, Epoch 9/19, Minibatch 0/0, Phase train, Running Loss train 289.95355224609375, Time 0.4222543239593506, Overall 5.170165777206421 \n",
+ "Fold 2/4, Epoch 9/19, Running Loss train 13.80731201171875, Time 0.4222543239593506, Overall 5.170165777206421 \n",
+ "Fold 2/4, Epoch 9/19, Minibatch 0/0, Phase valid, Running Loss valid 57.10198211669922, Time 0.4341466426849365, Overall 5.182058095932007 \n",
+ "Fold 2/4, Epoch 9/19, Running Loss valid 11.420396423339843, Time 0.4341466426849365, Overall 5.182058095932007 \n",
+ "Fold 2/4, Epoch 10/19, Minibatch 0/0, Phase train, Running Loss train 269.69744873046875, Time 0.4653770923614502, Overall 5.2132885456085205 \n",
+ "Fold 2/4, Epoch 10/19, Running Loss train 12.842735653831845, Time 0.46640992164611816, Overall 5.2143213748931885 \n",
+ "Fold 2/4, Epoch 10/19, Minibatch 0/0, Phase valid, Running Loss valid 61.86549377441406, Time 0.47637057304382324, Overall 5.2242820262908936 \n",
+ "Fold 2/4, Epoch 10/19, Running Loss valid 12.373098754882813, Time 0.47637057304382324, Overall 5.2242820262908936 \n",
+ "Fold 2/4, Epoch 11/19, Minibatch 0/0, Phase train, Running Loss train 268.4837646484375, Time 0.5072064399719238, Overall 5.255117893218994 \n",
+ "Fold 2/4, Epoch 11/19, Running Loss train 12.784941173735119, Time 0.5072064399719238, Overall 5.255117893218994 \n",
+ "Fold 2/4, Epoch 11/19, Minibatch 0/0, Phase valid, Running Loss valid 52.4472541809082, Time 0.5184876918792725, Overall 5.266399145126343 \n",
+ "Fold 2/4, Epoch 11/19, Running Loss valid 10.489450836181641, Time 0.5194869041442871, Overall 5.267398357391357 \n",
+ "Fold 2/4, Epoch 12/19, Minibatch 0/0, Phase train, Running Loss train 287.0235900878906, Time 0.5566325187683105, Overall 5.304543972015381 \n",
+ "Fold 2/4, Epoch 12/19, Running Loss train 13.667790004185267, Time 0.5566325187683105, Overall 5.304543972015381 \n",
+ "Fold 2/4, Epoch 12/19, Minibatch 0/0, Phase valid, Running Loss valid 41.685569763183594, Time 0.568669319152832, Overall 5.316580772399902 \n",
+ "Fold 2/4, Epoch 12/19, Running Loss valid 8.337113952636718, Time 0.568669319152832, Overall 5.316580772399902 \n",
+ "Fold 2/4, Epoch 13/19, Minibatch 0/0, Phase train, Running Loss train 125.11067962646484, Time 0.6030466556549072, Overall 5.3509581089019775 \n",
+ "Fold 2/4, Epoch 13/19, Running Loss train 5.95765141078404, Time 0.6030466556549072, Overall 5.3509581089019775 \n",
+ "Fold 2/4, Epoch 13/19, Minibatch 0/0, Phase valid, Running Loss valid 69.70402526855469, Time 0.614048957824707, Overall 5.361960411071777 \n",
+ "Fold 2/4, Epoch 13/19, Running Loss valid 13.940805053710937, Time 0.614048957824707, Overall 5.361960411071777 \n",
+ "Fold 2/4, Epoch 14/19, Minibatch 0/0, Phase train, Running Loss train 252.17723083496094, Time 0.6644270420074463, Overall 5.412338495254517 \n",
+ "Fold 2/4, Epoch 14/19, Running Loss train 12.008439563569569, Time 0.6644270420074463, Overall 5.412338495254517 \n",
+ "Fold 2/4, Epoch 14/19, Minibatch 0/0, Phase valid, Running Loss valid 39.99421691894531, Time 0.6754286289215088, Overall 5.423340082168579 \n",
+ "Fold 2/4, Epoch 14/19, Running Loss valid 7.998843383789063, Time 0.6754286289215088, Overall 5.423340082168579 \n",
+ "Fold 2/4, Epoch 15/19, Minibatch 0/0, Phase train, Running Loss train 283.14263916015625, Time 0.707427978515625, Overall 5.455339431762695 \n",
+ "Fold 2/4, Epoch 15/19, Running Loss train 13.482982817150297, Time 0.707427978515625, Overall 5.455339431762695 \n",
+ "Fold 2/4, Epoch 15/19, Minibatch 0/0, Phase valid, Running Loss valid 35.51783752441406, Time 0.7184274196624756, Overall 5.466338872909546 \n",
+ "Fold 2/4, Epoch 15/19, Running Loss valid 7.103567504882813, Time 0.7184274196624756, Overall 5.466338872909546 \n",
+ "Fold 2/4, Epoch 16/19, Minibatch 0/0, Phase train, Running Loss train 192.06735229492188, Time 0.7534627914428711, Overall 5.501374244689941 \n",
+ "Fold 2/4, Epoch 16/19, Running Loss train 9.14606439499628, Time 0.7534627914428711, Overall 5.501374244689941 \n",
+ "Fold 2/4, Epoch 16/19, Minibatch 0/0, Phase valid, Running Loss valid 44.02018737792969, Time 0.7644212245941162, Overall 5.5123326778411865 \n",
+ "Fold 2/4, Epoch 16/19, Running Loss valid 8.804037475585938, Time 0.7654213905334473, Overall 5.513332843780518 \n",
+ "Fold 2/4, Epoch 17/19, Minibatch 0/0, Phase train, Running Loss train 185.5213623046875, Time 0.8224208354949951, Overall 5.570332288742065 \n",
+ "Fold 2/4, Epoch 17/19, Running Loss train 8.8343505859375, Time 0.8224208354949951, Overall 5.570332288742065 \n",
+ "Fold 2/4, Epoch 17/19, Minibatch 0/0, Phase valid, Running Loss valid 59.7275390625, Time 0.8344211578369141, Overall 5.582332611083984 \n",
+ "Fold 2/4, Epoch 17/19, Running Loss valid 11.9455078125, Time 0.8344211578369141, Overall 5.582332611083984 \n",
+ "Fold 2/4, Epoch 18/19, Minibatch 0/0, Phase train, Running Loss train 149.49176025390625, Time 0.8674211502075195, Overall 5.61533260345459 \n",
+ "Fold 2/4, Epoch 18/19, Running Loss train 7.118655250186012, Time 0.8674211502075195, Overall 5.61533260345459 \n",
+ "Fold 2/4, Epoch 18/19, Minibatch 0/0, Phase valid, Running Loss valid 46.520626068115234, Time 0.8784260749816895, Overall 5.62633752822876 \n",
+ "Fold 2/4, Epoch 18/19, Running Loss valid 9.304125213623047, Time 0.8784260749816895, Overall 5.62633752822876 \n",
+ "Fold 2/4, Epoch 19/19, Minibatch 0/0, Phase train, Running Loss train 157.23046875, Time 0.909940242767334, Overall 5.657851696014404 \n",
+ "Fold 2/4, Epoch 19/19, Running Loss train 7.487165178571429, Time 0.909940242767334, Overall 5.657851696014404 \n",
+ "Fold 2/4, Epoch 19/19, Minibatch 0/0, Phase valid, Running Loss valid 74.27267456054688, Time 0.9199361801147461, Overall 5.667847633361816 \n",
+ "Fold 2/4, Epoch 19/19, Running Loss valid 14.854534912109376, Time 0.9199361801147461, Overall 5.667847633361816 \n",
+ "Fold 3/4, Epoch 0/19, Minibatch 0/0, Phase train, Running Loss train 345.1056823730469, Time 0.02792072296142578, Overall 5.706114053726196 \n",
+ "Fold 3/4, Epoch 0/19, Running Loss train 16.433603922526043, Time 0.02792072296142578, Overall 5.706114053726196 \n",
+ "Fold 3/4, Epoch 0/19, Minibatch 0/0, Phase valid, Running Loss valid 74.28764343261719, Time 0.038166046142578125, Overall 5.716359376907349 \n",
+ "Fold 3/4, Epoch 0/19, Running Loss valid 14.857528686523438, Time 0.038166046142578125, Overall 5.716359376907349 \n",
+ "Fold 3/4, Epoch 1/19, Minibatch 0/0, Phase train, Running Loss train 293.286865234375, Time 0.0694725513458252, Overall 5.747665882110596 \n",
+ "Fold 3/4, Epoch 1/19, Running Loss train 13.966041201636905, Time 0.0694725513458252, Overall 5.747665882110596 \n",
+ "Fold 3/4, Epoch 1/19, Minibatch 0/0, Phase valid, Running Loss valid 40.048465728759766, Time 0.07964658737182617, Overall 5.757839918136597 \n",
+ "Fold 3/4, Epoch 1/19, Running Loss valid 8.009693145751953, Time 0.08057641983032227, Overall 5.758769750595093 \n",
+ "Fold 3/4, Epoch 2/19, Minibatch 0/0, Phase train, Running Loss train 314.8412780761719, Time 0.11184120178222656, Overall 5.790034532546997 \n",
+ "Fold 3/4, Epoch 2/19, Running Loss train 14.992441813151041, Time 0.11288094520568848, Overall 5.791074275970459 \n",
+ "Fold 3/4, Epoch 2/19, Minibatch 0/0, Phase valid, Running Loss valid 46.11180877685547, Time 0.12287282943725586, Overall 5.801066160202026 \n",
+ "Fold 3/4, Epoch 2/19, Running Loss valid 9.222361755371093, Time 0.12287282943725586, Overall 5.801066160202026 \n",
+ "Fold 3/4, Epoch 3/19, Minibatch 0/0, Phase train, Running Loss train 274.5566711425781, Time 0.1544170379638672, Overall 5.832610368728638 \n",
+ "Fold 3/4, Epoch 3/19, Running Loss train 13.074127197265625, Time 0.1544170379638672, Overall 5.832610368728638 \n",
+ "Fold 3/4, Epoch 3/19, Minibatch 0/0, Phase valid, Running Loss valid 73.82421112060547, Time 0.16541719436645508, Overall 5.843610525131226 \n",
+ "Fold 3/4, Epoch 3/19, Running Loss valid 14.764842224121093, Time 0.16541719436645508, Overall 5.843610525131226 \n",
+ "Fold 3/4, Epoch 4/19, Minibatch 0/0, Phase train, Running Loss train 381.44146728515625, Time 0.19826841354370117, Overall 5.876461744308472 \n",
+ "Fold 3/4, Epoch 4/19, Running Loss train 18.16387939453125, Time 0.19826841354370117, Overall 5.876461744308472 \n",
+ "Fold 3/4, Epoch 4/19, Minibatch 0/0, Phase valid, Running Loss valid 58.846519470214844, Time 0.20956969261169434, Overall 5.887763023376465 \n",
+ "Fold 3/4, Epoch 4/19, Running Loss valid 11.769303894042968, Time 0.20956969261169434, Overall 5.887763023376465 \n",
+ "Fold 3/4, Epoch 5/19, Minibatch 0/0, Phase train, Running Loss train 264.5203857421875, Time 0.24682855606079102, Overall 5.9250218868255615 \n",
+ "Fold 3/4, Epoch 5/19, Running Loss train 12.596208844866071, Time 0.24682855606079102, Overall 5.9250218868255615 \n",
+ "Fold 3/4, Epoch 5/19, Minibatch 0/0, Phase valid, Running Loss valid 76.337646484375, Time 0.2581298351287842, Overall 5.936323165893555 \n",
+ "Fold 3/4, Epoch 5/19, Running Loss valid 15.267529296875, Time 0.2581298351287842, Overall 5.936323165893555 \n",
+ "Fold 3/4, Epoch 6/19, Minibatch 0/0, Phase train, Running Loss train 216.62884521484375, Time 0.29661083221435547, Overall 5.974804162979126 \n",
+ "Fold 3/4, Epoch 6/19, Running Loss train 10.31565929594494, Time 0.29661083221435547, Overall 5.974804162979126 \n",
+ "Fold 3/4, Epoch 6/19, Minibatch 0/0, Phase valid, Running Loss valid 31.53200340270996, Time 0.30845212936401367, Overall 5.986645460128784 \n",
+ "Fold 3/4, Epoch 6/19, Running Loss valid 6.3064006805419925, Time 0.30845212936401367, Overall 5.986645460128784 \n",
+ "Fold 3/4, Epoch 7/19, Minibatch 0/0, Phase train, Running Loss train 250.02200317382812, Time 0.34145164489746094, Overall 6.0196449756622314 \n",
+ "Fold 3/4, Epoch 7/19, Running Loss train 11.905809674944196, Time 0.34145164489746094, Overall 6.0196449756622314 \n",
+ "Fold 3/4, Epoch 7/19, Minibatch 0/0, Phase valid, Running Loss valid 67.86028289794922, Time 0.351487398147583, Overall 6.0296807289123535 \n",
+ "Fold 3/4, Epoch 7/19, Running Loss valid 13.572056579589844, Time 0.3524904251098633, Overall 6.030683755874634 \n",
+ "Fold 3/4, Epoch 8/19, Minibatch 0/0, Phase train, Running Loss train 163.09503173828125, Time 0.38632941246032715, Overall 6.064522743225098 \n",
+ "Fold 3/4, Epoch 8/19, Running Loss train 7.7664300827752974, Time 0.38632941246032715, Overall 6.064522743225098 \n",
+ "Fold 3/4, Epoch 8/19, Minibatch 0/0, Phase valid, Running Loss valid 57.410240173339844, Time 0.3963010311126709, Overall 6.074494361877441 \n",
+ "Fold 3/4, Epoch 8/19, Running Loss valid 11.482048034667969, Time 0.3963010311126709, Overall 6.074494361877441 \n",
+ "Fold 3/4, Epoch 9/19, Minibatch 0/0, Phase train, Running Loss train 368.7311096191406, Time 0.42726683616638184, Overall 6.105460166931152 \n",
+ "Fold 3/4, Epoch 9/19, Running Loss train 17.558624267578125, Time 0.42726683616638184, Overall 6.105460166931152 \n",
+ "Fold 3/4, Epoch 9/19, Minibatch 0/0, Phase valid, Running Loss valid 47.40684127807617, Time 0.4384152889251709, Overall 6.116608619689941 \n",
+ "Fold 3/4, Epoch 9/19, Running Loss valid 9.481368255615234, Time 0.4384152889251709, Overall 6.116608619689941 \n",
+ "Fold 3/4, Epoch 10/19, Minibatch 0/0, Phase train, Running Loss train 296.4657287597656, Time 0.47141528129577637, Overall 6.149608612060547 \n",
+ "Fold 3/4, Epoch 10/19, Running Loss train 14.117415655226935, Time 0.47141528129577637, Overall 6.149608612060547 \n",
+ "Fold 3/4, Epoch 10/19, Minibatch 0/0, Phase valid, Running Loss valid 80.32162475585938, Time 0.48238515853881836, Overall 6.160578489303589 \n",
+ "Fold 3/4, Epoch 10/19, Running Loss valid 16.064324951171876, Time 0.48238515853881836, Overall 6.160578489303589 \n",
+ "Fold 3/4, Epoch 11/19, Minibatch 0/0, Phase train, Running Loss train 227.9687957763672, Time 0.5153825283050537, Overall 6.193575859069824 \n",
+ "Fold 3/4, Epoch 11/19, Running Loss train 10.855656941731771, Time 0.5153825283050537, Overall 6.193575859069824 \n",
+ "Fold 3/4, Epoch 11/19, Minibatch 0/0, Phase valid, Running Loss valid 33.964195251464844, Time 0.5263781547546387, Overall 6.204571485519409 \n",
+ "Fold 3/4, Epoch 11/19, Running Loss valid 6.792839050292969, Time 0.5263781547546387, Overall 6.204571485519409 \n",
+ "Fold 3/4, Epoch 12/19, Minibatch 0/0, Phase train, Running Loss train 202.17547607421875, Time 0.5593826770782471, Overall 6.237576007843018 \n",
+ "Fold 3/4, Epoch 12/19, Running Loss train 9.627403622581845, Time 0.5593826770782471, Overall 6.237576007843018 \n",
+ "Fold 3/4, Epoch 12/19, Minibatch 0/0, Phase valid, Running Loss valid 80.26263427734375, Time 0.570380687713623, Overall 6.2485740184783936 \n",
+ "Fold 3/4, Epoch 12/19, Running Loss valid 16.05252685546875, Time 0.570380687713623, Overall 6.2485740184783936 \n",
+ "Fold 3/4, Epoch 13/19, Minibatch 0/0, Phase train, Running Loss train 237.56607055664062, Time 0.602731466293335, Overall 6.2809247970581055 \n",
+ "Fold 3/4, Epoch 13/19, Running Loss train 11.312670026506696, Time 0.602731466293335, Overall 6.2809247970581055 \n",
+ "Fold 3/4, Epoch 13/19, Minibatch 0/0, Phase valid, Running Loss valid 44.15294647216797, Time 0.6138348579406738, Overall 6.292028188705444 \n",
+ "Fold 3/4, Epoch 13/19, Running Loss valid 8.830589294433594, Time 0.6138348579406738, Overall 6.292028188705444 \n",
+ "Fold 3/4, Epoch 14/19, Minibatch 0/0, Phase train, Running Loss train 269.99639892578125, Time 0.6450014114379883, Overall 6.323194742202759 \n",
+ "Fold 3/4, Epoch 14/19, Running Loss train 12.856971377418155, Time 0.6460459232330322, Overall 6.324239253997803 \n",
+ "Fold 3/4, Epoch 14/19, Minibatch 0/0, Phase valid, Running Loss valid 45.65902328491211, Time 0.6537978649139404, Overall 6.331991195678711 \n",
+ "Fold 3/4, Epoch 14/19, Running Loss valid 9.131804656982421, Time 0.6537978649139404, Overall 6.331991195678711 \n",
+ "Fold 3/4, Epoch 15/19, Minibatch 0/0, Phase train, Running Loss train 128.41725158691406, Time 0.6862523555755615, Overall 6.364445686340332 \n",
+ "Fold 3/4, Epoch 15/19, Running Loss train 6.1151072184244795, Time 0.6862523555755615, Overall 6.364445686340332 \n",
+ "Fold 3/4, Epoch 15/19, Minibatch 0/0, Phase valid, Running Loss valid 46.60124588012695, Time 0.6962549686431885, Overall 6.374448299407959 \n",
+ "Fold 3/4, Epoch 15/19, Running Loss valid 9.32024917602539, Time 0.6972537040710449, Overall 6.375447034835815 \n",
+ "Fold 3/4, Epoch 16/19, Minibatch 0/0, Phase train, Running Loss train 188.31240844726562, Time 0.7276227474212646, Overall 6.405816078186035 \n",
+ "Fold 3/4, Epoch 16/19, Running Loss train 8.967257545107888, Time 0.7286114692687988, Overall 6.406804800033569 \n",
+ "Fold 3/4, Epoch 16/19, Minibatch 0/0, Phase valid, Running Loss valid 30.548398971557617, Time 0.7391514778137207, Overall 6.417344808578491 \n",
+ "Fold 3/4, Epoch 16/19, Running Loss valid 6.109679794311523, Time 0.7391514778137207, Overall 6.417344808578491 \n",
+ "Fold 3/4, Epoch 17/19, Minibatch 0/0, Phase train, Running Loss train 82.09529876708984, Time 0.7704441547393799, Overall 6.44863748550415 \n",
+ "Fold 3/4, Epoch 17/19, Running Loss train 3.9092999412899925, Time 0.7704441547393799, Overall 6.44863748550415 \n",
+ "Fold 3/4, Epoch 17/19, Minibatch 0/0, Phase valid, Running Loss valid 81.87535095214844, Time 0.7814486026763916, Overall 6.459641933441162 \n",
+ "Fold 3/4, Epoch 17/19, Running Loss valid 16.37507019042969, Time 0.7814486026763916, Overall 6.459641933441162 \n",
+ "Fold 3/4, Epoch 18/19, Minibatch 0/0, Phase train, Running Loss train 128.8899383544922, Time 0.8144412040710449, Overall 6.492634534835815 \n",
+ "Fold 3/4, Epoch 18/19, Running Loss train 6.137616112118676, Time 0.8144412040710449, Overall 6.492634534835815 \n",
+ "Fold 3/4, Epoch 18/19, Minibatch 0/0, Phase valid, Running Loss valid 43.13569641113281, Time 0.8244411945343018, Overall 6.502634525299072 \n",
+ "Fold 3/4, Epoch 18/19, Running Loss valid 8.627139282226562, Time 0.8244411945343018, Overall 6.502634525299072 \n",
+ "Fold 3/4, Epoch 19/19, Minibatch 0/0, Phase train, Running Loss train 244.5829620361328, Time 0.858905553817749, Overall 6.5370988845825195 \n",
+ "Fold 3/4, Epoch 19/19, Running Loss train 11.646807716006325, Time 0.858905553817749, Overall 6.5370988845825195 \n",
+ "Fold 3/4, Epoch 19/19, Minibatch 0/0, Phase valid, Running Loss valid 38.02728271484375, Time 0.8699052333831787, Overall 6.548098564147949 \n",
+ "Fold 3/4, Epoch 19/19, Running Loss valid 7.60545654296875, Time 0.8699052333831787, Overall 6.548098564147949 \n",
+ "Fold 4/4, Epoch 0/19, Minibatch 0/0, Phase train, Running Loss train 270.8803405761719, Time 0.027253389358520508, Overall 6.585127592086792 \n",
+ "Fold 4/4, Epoch 0/19, Running Loss train 12.899063836960565, Time 0.02825474739074707, Overall 6.5861289501190186 \n",
+ "Fold 4/4, Epoch 0/19, Minibatch 0/0, Phase valid, Running Loss valid 57.89813995361328, Time 0.0382542610168457, Overall 6.596128463745117 \n",
+ "Fold 4/4, Epoch 0/19, Running Loss valid 11.579627990722656, Time 0.039255380630493164, Overall 6.597129583358765 \n",
+ "Fold 4/4, Epoch 1/19, Minibatch 0/0, Phase train, Running Loss train 246.2553253173828, Time 0.07059717178344727, Overall 6.628471374511719 \n",
+ "Fold 4/4, Epoch 1/19, Running Loss train 11.726444062732515, Time 0.07059717178344727, Overall 6.628471374511719 \n",
+ "Fold 4/4, Epoch 1/19, Minibatch 0/0, Phase valid, Running Loss valid 72.75984954833984, Time 0.08174419403076172, Overall 6.639618396759033 \n",
+ "Fold 4/4, Epoch 1/19, Running Loss valid 14.551969909667969, Time 0.08174419403076172, Overall 6.639618396759033 \n",
+ "Fold 4/4, Epoch 2/19, Minibatch 0/0, Phase train, Running Loss train 278.0937194824219, Time 0.11234092712402344, Overall 6.670215129852295 \n",
+ "Fold 4/4, Epoch 2/19, Running Loss train 13.242558070591517, Time 0.11234092712402344, Overall 6.670215129852295 \n",
+ "Fold 4/4, Epoch 2/19, Minibatch 0/0, Phase valid, Running Loss valid 72.7728271484375, Time 0.12404346466064453, Overall 6.681917667388916 \n",
+ "Fold 4/4, Epoch 2/19, Running Loss valid 14.5545654296875, Time 0.12404346466064453, Overall 6.681917667388916 \n",
+ "Fold 4/4, Epoch 3/19, Minibatch 0/0, Phase train, Running Loss train 222.35459899902344, Time 0.15599322319030762, Overall 6.713867425918579 \n",
+ "Fold 4/4, Epoch 3/19, Running Loss train 10.588314238048735, Time 0.15599322319030762, Overall 6.713867425918579 \n",
+ "Fold 4/4, Epoch 3/19, Minibatch 0/0, Phase valid, Running Loss valid 74.1617431640625, Time 0.16700387001037598, Overall 6.7248780727386475 \n",
+ "Fold 4/4, Epoch 3/19, Running Loss valid 14.8323486328125, Time 0.16700387001037598, Overall 6.7248780727386475 \n",
+ "Fold 4/4, Epoch 4/19, Minibatch 0/0, Phase train, Running Loss train 341.50714111328125, Time 0.19799399375915527, Overall 6.755868196487427 \n",
+ "Fold 4/4, Epoch 4/19, Running Loss train 16.262244814918155, Time 0.19799399375915527, Overall 6.755868196487427 \n",
+ "Fold 4/4, Epoch 4/19, Minibatch 0/0, Phase valid, Running Loss valid 77.46047973632812, Time 0.20849370956420898, Overall 6.7663679122924805 \n",
+ "Fold 4/4, Epoch 4/19, Running Loss valid 15.492095947265625, Time 0.20849370956420898, Overall 6.7663679122924805 \n",
+ "Fold 4/4, Epoch 5/19, Minibatch 0/0, Phase train, Running Loss train 220.587158203125, Time 0.23949599266052246, Overall 6.797370195388794 \n",
+ "Fold 4/4, Epoch 5/19, Running Loss train 10.504150390625, Time 0.23949599266052246, Overall 6.797370195388794 \n",
+ "Fold 4/4, Epoch 5/19, Minibatch 0/0, Phase valid, Running Loss valid 86.21421813964844, Time 0.25059986114501953, Overall 6.808474063873291 \n",
+ "Fold 4/4, Epoch 5/19, Running Loss valid 17.242843627929688, Time 0.25059986114501953, Overall 6.808474063873291 \n",
+ "Fold 4/4, Epoch 6/19, Minibatch 0/0, Phase train, Running Loss train 231.66419982910156, Time 0.2827630043029785, Overall 6.84063720703125 \n",
+ "Fold 4/4, Epoch 6/19, Running Loss train 11.03162856329055, Time 0.2827630043029785, Overall 6.84063720703125 \n",
+ "Fold 4/4, Epoch 6/19, Minibatch 0/0, Phase valid, Running Loss valid 86.43109130859375, Time 0.2928352355957031, Overall 6.850709438323975 \n",
+ "Fold 4/4, Epoch 6/19, Running Loss valid 17.28621826171875, Time 0.2928352355957031, Overall 6.850709438323975 \n",
+ "Fold 4/4, Epoch 7/19, Minibatch 0/0, Phase train, Running Loss train 175.728271484375, Time 0.3242611885070801, Overall 6.882135391235352 \n",
+ "Fold 4/4, Epoch 7/19, Running Loss train 8.368012927827381, Time 0.3242611885070801, Overall 6.882135391235352 \n",
+ "Fold 4/4, Epoch 7/19, Minibatch 0/0, Phase valid, Running Loss valid 105.3039321899414, Time 0.33628296852111816, Overall 6.89415717124939 \n",
+ "Fold 4/4, Epoch 7/19, Running Loss valid 21.06078643798828, Time 0.33628296852111816, Overall 6.89415717124939 \n",
+ "Fold 4/4, Epoch 8/19, Minibatch 0/0, Phase train, Running Loss train 264.690673828125, Time 0.3721904754638672, Overall 6.930064678192139 \n",
+ "Fold 4/4, Epoch 8/19, Running Loss train 12.604317801339286, Time 0.3721904754638672, Overall 6.930064678192139 \n",
+ "Fold 4/4, Epoch 8/19, Minibatch 0/0, Phase valid, Running Loss valid 111.57740783691406, Time 0.386333703994751, Overall 6.9442079067230225 \n",
+ "Fold 4/4, Epoch 8/19, Running Loss valid 22.31548156738281, Time 0.386333703994751, Overall 6.945209503173828 \n",
+ "Fold 4/4, Epoch 9/19, Minibatch 0/0, Phase train, Running Loss train 272.2957458496094, Time 0.4218604564666748, Overall 6.979734659194946 \n",
+ "Fold 4/4, Epoch 9/19, Running Loss train 12.966464088076638, Time 0.42286062240600586, Overall 6.980734825134277 \n",
+ "Fold 4/4, Epoch 9/19, Minibatch 0/0, Phase valid, Running Loss valid 61.778411865234375, Time 0.43400001525878906, Overall 6.9918742179870605 \n",
+ "Fold 4/4, Epoch 9/19, Running Loss valid 12.355682373046875, Time 0.43400001525878906, Overall 6.9918742179870605 \n",
+ "Fold 4/4, Epoch 10/19, Minibatch 0/0, Phase train, Running Loss train 124.22348022460938, Time 0.4688413143157959, Overall 7.026715517044067 \n",
+ "Fold 4/4, Epoch 10/19, Running Loss train 5.915403820219494, Time 0.4688413143157959, Overall 7.026715517044067 \n",
+ "Fold 4/4, Epoch 10/19, Minibatch 0/0, Phase valid, Running Loss valid 126.7549057006836, Time 0.47916555404663086, Overall 7.037039756774902 \n",
+ "Fold 4/4, Epoch 10/19, Running Loss valid 25.350981140136717, Time 0.47916555404663086, Overall 7.037039756774902 \n",
+ "Fold 4/4, Epoch 11/19, Minibatch 0/0, Phase train, Running Loss train 185.1405487060547, Time 0.5106408596038818, Overall 7.068515062332153 \n",
+ "Fold 4/4, Epoch 11/19, Running Loss train 8.816216605050224, Time 0.5106408596038818, Overall 7.068515062332153 \n",
+ "Fold 4/4, Epoch 11/19, Minibatch 0/0, Phase valid, Running Loss valid 82.51553344726562, Time 0.5226407051086426, Overall 7.080514907836914 \n",
+ "Fold 4/4, Epoch 11/19, Running Loss valid 16.503106689453126, Time 0.5226407051086426, Overall 7.080514907836914 \n",
+ "Fold 4/4, Epoch 12/19, Minibatch 0/0, Phase train, Running Loss train 270.6172180175781, Time 0.5538477897644043, Overall 7.111721992492676 \n",
+ "Fold 4/4, Epoch 12/19, Running Loss train 12.886534191313244, Time 0.5538477897644043, Overall 7.111721992492676 \n",
+ "Fold 4/4, Epoch 12/19, Minibatch 0/0, Phase valid, Running Loss valid 80.2857666015625, Time 0.5651562213897705, Overall 7.123030424118042 \n",
+ "Fold 4/4, Epoch 12/19, Running Loss valid 16.0571533203125, Time 0.5651562213897705, Overall 7.123030424118042 \n",
+ "Fold 4/4, Epoch 13/19, Minibatch 0/0, Phase train, Running Loss train 138.1797332763672, Time 0.5969119071960449, Overall 7.154786109924316 \n",
+ "Fold 4/4, Epoch 13/19, Running Loss train 6.579987298874628, Time 0.5979125499725342, Overall 7.155786752700806 \n",
+ "Fold 4/4, Epoch 13/19, Minibatch 0/0, Phase valid, Running Loss valid 80.07384490966797, Time 0.6089119911193848, Overall 7.166786193847656 \n",
+ "Fold 4/4, Epoch 13/19, Running Loss valid 16.014768981933592, Time 0.6089119911193848, Overall 7.166786193847656 \n",
+ "Fold 4/4, Epoch 14/19, Minibatch 0/0, Phase train, Running Loss train 210.91873168945312, Time 0.6460769176483154, Overall 7.203951120376587 \n",
+ "Fold 4/4, Epoch 14/19, Running Loss train 10.043749128069196, Time 0.6460769176483154, Overall 7.203951120376587 \n",
+ "Fold 4/4, Epoch 14/19, Minibatch 0/0, Phase valid, Running Loss valid 95.47752380371094, Time 0.6543197631835938, Overall 7.212193965911865 \n",
+ "Fold 4/4, Epoch 14/19, Running Loss valid 19.095504760742188, Time 0.6543197631835938, Overall 7.212193965911865 \n",
+ "Fold 4/4, Epoch 15/19, Minibatch 0/0, Phase train, Running Loss train 240.2045440673828, Time 0.6931560039520264, Overall 7.251030206680298 \n",
+ "Fold 4/4, Epoch 15/19, Running Loss train 11.438311622256325, Time 0.6939349174499512, Overall 7.251809120178223 \n",
+ "Fold 4/4, Epoch 15/19, Minibatch 0/0, Phase valid, Running Loss valid 91.93415832519531, Time 0.7071313858032227, Overall 7.265005588531494 \n",
+ "Fold 4/4, Epoch 15/19, Running Loss valid 18.38683166503906, Time 0.7071313858032227, Overall 7.265005588531494 \n",
+ "Fold 4/4, Epoch 16/19, Minibatch 0/0, Phase train, Running Loss train 189.84376525878906, Time 0.7438554763793945, Overall 7.301729679107666 \n",
+ "Fold 4/4, Epoch 16/19, Running Loss train 9.040179298037575, Time 0.7438554763793945, Overall 7.301729679107666 \n",
+ "Fold 4/4, Epoch 16/19, Minibatch 0/0, Phase valid, Running Loss valid 56.034019470214844, Time 0.7585043907165527, Overall 7.316378593444824 \n",
+ "Fold 4/4, Epoch 16/19, Running Loss valid 11.206803894042968, Time 0.7585043907165527, Overall 7.316378593444824 \n",
+ "Fold 4/4, Epoch 17/19, Minibatch 0/0, Phase train, Running Loss train 190.76541137695312, Time 0.799030065536499, Overall 7.3569042682647705 \n",
+ "Fold 4/4, Epoch 17/19, Running Loss train 9.084067208426339, Time 0.799030065536499, Overall 7.3569042682647705 \n",
+ "Fold 4/4, Epoch 17/19, Minibatch 0/0, Phase valid, Running Loss valid 99.03899383544922, Time 0.8121697902679443, Overall 7.370558738708496 \n",
+ "Fold 4/4, Epoch 17/19, Running Loss valid 19.807798767089842, Time 0.8127298355102539, Overall 7.370604038238525 \n",
+ "Fold 4/4, Epoch 18/19, Minibatch 0/0, Phase train, Running Loss train 206.193115234375, Time 0.8538713455200195, Overall 7.411745548248291 \n",
+ "Fold 4/4, Epoch 18/19, Running Loss train 9.818719773065476, Time 0.8538713455200195, Overall 7.411745548248291 \n",
+ "Fold 4/4, Epoch 18/19, Minibatch 0/0, Phase valid, Running Loss valid 71.90238952636719, Time 0.8680891990661621, Overall 7.425963401794434 \n",
+ "Fold 4/4, Epoch 18/19, Running Loss valid 14.380477905273438, Time 0.8680891990661621, Overall 7.425963401794434 \n",
+ "Fold 4/4, Epoch 19/19, Minibatch 0/0, Phase train, Running Loss train 179.0325927734375, Time 0.9084174633026123, Overall 7.466291666030884 \n",
+ "Fold 4/4, Epoch 19/19, Running Loss train 8.525361560639881, Time 0.9084174633026123, Overall 7.466291666030884 \n",
+ "Fold 4/4, Epoch 19/19, Minibatch 0/0, Phase valid, Running Loss valid 44.92753219604492, Time 0.9194145202636719, Overall 7.477288722991943 \n",
+ "Fold 4/4, Epoch 19/19, Running Loss valid 8.985506439208985, Time 0.9194145202636719, Overall 7.477288722991943 \n",
+ "It took 7.4831109046936035 to train the model.\n",
+ "Figure(640x480)\n",
+ "Figure(640x480)\n",
+ "Figure(640x480)\n",
+ "Figure(640x480)\n",
+ "Figure(640x480)\n",
+ "Running for (dataset, model): (dblp, tbnn) ... \n",
+ "Fold 0/4, Epoch 0/19, Minibatch 0/0, Phase train, Running Loss train 220.37966918945312, Time 0.02385258674621582, Overall 0.025847911834716797 \n",
+ "Fold 0/4, Epoch 0/19, Running Loss train 18.364972432454426, Time 0.02385258674621582, Overall 0.025847911834716797 \n",
+ "Fold 0/4, Epoch 0/19, Minibatch 0/0, Phase valid, Running Loss valid 54.05299377441406, Time 0.03483843803405762, Overall 0.036833763122558594 \n",
+ "Fold 0/4, Epoch 0/19, Running Loss valid 13.513248443603516, Time 0.03483843803405762, Overall 0.036833763122558594 \n",
+ "Fold 0/4, Epoch 1/19, Minibatch 0/0, Phase train, Running Loss train 149.54063415527344, Time 0.05891084671020508, Overall 0.060906171798706055 \n",
+ "Fold 0/4, Epoch 1/19, Running Loss train 12.461719512939453, Time 0.05891084671020508, Overall 0.060906171798706055 \n",
+ "Fold 0/4, Epoch 1/19, Minibatch 0/0, Phase valid, Running Loss valid 22.217761993408203, Time 0.06884479522705078, Overall 0.07084012031555176 \n",
+ "Fold 0/4, Epoch 1/19, Running Loss valid 5.554440498352051, Time 0.06884479522705078, Overall 0.07084012031555176 \n",
+ "Fold 0/4, Epoch 2/19, Minibatch 0/0, Phase train, Running Loss train 121.35899353027344, Time 0.09197163581848145, Overall 0.09396696090698242 \n",
+ "Fold 0/4, Epoch 2/19, Running Loss train 10.11324946085612, Time 0.09197163581848145, Overall 0.09396696090698242 \n",
+ "Fold 0/4, Epoch 2/19, Minibatch 0/0, Phase valid, Running Loss valid 58.73306655883789, Time 0.10209035873413086, Overall 0.10408568382263184 \n",
+ "Fold 0/4, Epoch 2/19, Running Loss valid 14.683266639709473, Time 0.10209035873413086, Overall 0.10408568382263184 \n",
+ "Fold 0/4, Epoch 3/19, Minibatch 0/0, Phase train, Running Loss train 182.0093994140625, Time 0.1284024715423584, Overall 0.13039779663085938 \n",
+ "Fold 0/4, Epoch 3/19, Running Loss train 15.167449951171875, Time 0.1284024715423584, Overall 0.13039779663085938 \n",
+ "Fold 0/4, Epoch 3/19, Minibatch 0/0, Phase valid, Running Loss valid 79.49810028076172, Time 0.13744139671325684, Overall 0.1394367218017578 \n",
+ "Fold 0/4, Epoch 3/19, Running Loss valid 19.87452507019043, Time 0.13744139671325684, Overall 0.1394367218017578 \n",
+ "Fold 0/4, Epoch 4/19, Minibatch 0/0, Phase train, Running Loss train 146.17286682128906, Time 0.16043996810913086, Overall 0.16243529319763184 \n",
+ "Fold 0/4, Epoch 4/19, Running Loss train 12.181072235107422, Time 0.16043996810913086, Overall 0.16243529319763184 \n",
+ "Fold 0/4, Epoch 4/19, Minibatch 0/0, Phase valid, Running Loss valid 29.213436126708984, Time 0.16944193840026855, Overall 0.17243742942810059 \n",
+ "Fold 0/4, Epoch 4/19, Running Loss valid 7.303359031677246, Time 0.1704421043395996, Overall 0.17243742942810059 \n",
+ "Fold 0/4, Epoch 5/19, Minibatch 0/0, Phase train, Running Loss train 131.99887084960938, Time 0.1934053897857666, Overall 0.19540071487426758 \n",
+ "Fold 0/4, Epoch 5/19, Running Loss train 10.999905904134115, Time 0.1934053897857666, Overall 0.19540071487426758 \n",
+ "Fold 0/4, Epoch 5/19, Minibatch 0/0, Phase valid, Running Loss valid 110.39471435546875, Time 0.20340275764465332, Overall 0.2053980827331543 \n",
+ "Fold 0/4, Epoch 5/19, Running Loss valid 27.598678588867188, Time 0.20340275764465332, Overall 0.2053980827331543 \n",
+ "Fold 0/4, Epoch 6/19, Minibatch 0/0, Phase train, Running Loss train 150.42572021484375, Time 0.227402925491333, Overall 0.22939825057983398 \n",
+ "Fold 0/4, Epoch 6/19, Running Loss train 12.535476684570312, Time 0.22840404510498047, Overall 0.23039937019348145 \n",
+ "Fold 0/4, Epoch 6/19, Minibatch 0/0, Phase valid, Running Loss valid 35.894248962402344, Time 0.2384021282196045, Overall 0.24039745330810547 \n",
+ "Fold 0/4, Epoch 6/19, Running Loss valid 8.973562240600586, Time 0.2384021282196045, Overall 0.24039745330810547 \n",
+ "Fold 0/4, Epoch 7/19, Minibatch 0/0, Phase train, Running Loss train 132.61795043945312, Time 0.26940131187438965, Overall 0.2713966369628906 \n",
+ "Fold 0/4, Epoch 7/19, Running Loss train 11.051495869954428, Time 0.26940131187438965, Overall 0.2713966369628906 \n",
+ "Fold 0/4, Epoch 7/19, Minibatch 0/0, Phase valid, Running Loss valid 33.127235412597656, Time 0.28015708923339844, Overall 0.2821524143218994 \n",
+ "Fold 0/4, Epoch 7/19, Running Loss valid 8.281808853149414, Time 0.28015708923339844, Overall 0.2821524143218994 \n",
+ "Fold 0/4, Epoch 8/19, Minibatch 0/0, Phase train, Running Loss train 149.59239196777344, Time 0.3067195415496826, Overall 0.3087148666381836 \n",
+ "Fold 0/4, Epoch 8/19, Running Loss train 12.46603266398112, Time 0.3067195415496826, Overall 0.3087148666381836 \n",
+ "Fold 0/4, Epoch 8/19, Minibatch 0/0, Phase valid, Running Loss valid 21.732248306274414, Time 0.31780052185058594, Overall 0.3197958469390869 \n",
+ "Fold 0/4, Epoch 8/19, Running Loss valid 5.4330620765686035, Time 0.31780052185058594, Overall 0.3197958469390869 \n",
+ "Fold 0/4, Epoch 9/19, Minibatch 0/0, Phase train, Running Loss train 187.32518005371094, Time 0.34343457221984863, Overall 0.3454298973083496 \n",
+ "Fold 0/4, Epoch 9/19, Running Loss train 15.610431671142578, Time 0.34343457221984863, Overall 0.3454298973083496 \n",
+ "Fold 0/4, Epoch 9/19, Minibatch 0/0, Phase valid, Running Loss valid 46.808589935302734, Time 0.3569505214691162, Overall 0.3589458465576172 \n",
+ "Fold 0/4, Epoch 9/19, Running Loss valid 11.702147483825684, Time 0.3569505214691162, Overall 0.3589458465576172 \n",
+ "Fold 0/4, Epoch 10/19, Minibatch 0/0, Phase train, Running Loss train 98.97306060791016, Time 0.3859109878540039, Overall 0.3879063129425049 \n",
+ "Fold 0/4, Epoch 10/19, Running Loss train 8.24775505065918, Time 0.3859109878540039, Overall 0.3879063129425049 \n",
+ "Fold 0/4, Epoch 10/19, Minibatch 0/0, Phase valid, Running Loss valid 13.391858100891113, Time 0.3959052562713623, Overall 0.3979005813598633 \n",
+ "Fold 0/4, Epoch 10/19, Running Loss valid 3.3479645252227783, Time 0.3959052562713623, Overall 0.3979005813598633 \n",
+ "Fold 0/4, Epoch 11/19, Minibatch 0/0, Phase train, Running Loss train 145.80435180664062, Time 0.42290663719177246, Overall 0.42490196228027344 \n",
+ "Fold 0/4, Epoch 11/19, Running Loss train 12.150362650553385, Time 0.42290663719177246, Overall 0.42490196228027344 \n",
+ "Fold 0/4, Epoch 11/19, Minibatch 0/0, Phase valid, Running Loss valid 38.999534606933594, Time 0.43390703201293945, Overall 0.43590235710144043 \n",
+ "Fold 0/4, Epoch 11/19, Running Loss valid 9.749883651733398, Time 0.43390703201293945, Overall 0.43590235710144043 \n",
+ "Fold 0/4, Epoch 12/19, Minibatch 0/0, Phase train, Running Loss train 159.315185546875, Time 0.45790600776672363, Overall 0.4599013328552246 \n",
+ "Fold 0/4, Epoch 12/19, Running Loss train 13.276265462239584, Time 0.45790600776672363, Overall 0.4599013328552246 \n",
+ "Fold 0/4, Epoch 12/19, Minibatch 0/0, Phase valid, Running Loss valid 26.264419555664062, Time 0.46790385246276855, Overall 0.46989917755126953 \n",
+ "Fold 0/4, Epoch 12/19, Running Loss valid 6.566104888916016, Time 0.4689064025878906, Overall 0.4709017276763916 \n",
+ "Fold 0/4, Epoch 13/19, Minibatch 0/0, Phase train, Running Loss train 140.43067932128906, Time 0.4938483238220215, Overall 0.49584364891052246 \n",
+ "Fold 0/4, Epoch 13/19, Running Loss train 11.702556610107422, Time 0.4938483238220215, Overall 0.49584364891052246 \n",
+ "Fold 0/4, Epoch 13/19, Minibatch 0/0, Phase valid, Running Loss valid 35.62059020996094, Time 0.5038456916809082, Overall 0.5058410167694092 \n",
+ "Fold 0/4, Epoch 13/19, Running Loss valid 8.905147552490234, Time 0.5038456916809082, Overall 0.5058410167694092 \n",
+ "Fold 0/4, Epoch 14/19, Minibatch 0/0, Phase train, Running Loss train 180.31912231445312, Time 0.5278456211090088, Overall 0.5298409461975098 \n",
+ "Fold 0/4, Epoch 14/19, Running Loss train 15.026593526204428, Time 0.5278456211090088, Overall 0.5298409461975098 \n",
+ "Fold 0/4, Epoch 14/19, Minibatch 0/0, Phase valid, Running Loss valid 8.003910064697266, Time 0.5368845462799072, Overall 0.5388798713684082 \n",
+ "Fold 0/4, Epoch 14/19, Running Loss valid 2.0009775161743164, Time 0.5368845462799072, Overall 0.5388798713684082 \n",
+ "Fold 0/4, Epoch 15/19, Minibatch 0/0, Phase train, Running Loss train 132.35400390625, Time 0.5608468055725098, Overall 0.5628421306610107 \n",
+ "Fold 0/4, Epoch 15/19, Running Loss train 11.029500325520834, Time 0.5608468055725098, Overall 0.5628421306610107 \n",
+ "Fold 0/4, Epoch 15/19, Minibatch 0/0, Phase valid, Running Loss valid 19.50863265991211, Time 0.5708856582641602, Overall 0.5728809833526611 \n",
+ "Fold 0/4, Epoch 15/19, Running Loss valid 4.877158164978027, Time 0.5708856582641602, Overall 0.5728809833526611 \n",
+ "Fold 0/4, Epoch 16/19, Minibatch 0/0, Phase train, Running Loss train 189.90126037597656, Time 0.5988924503326416, Overall 0.6008877754211426 \n",
+ "Fold 0/4, Epoch 16/19, Running Loss train 15.82510503133138, Time 0.5988924503326416, Overall 0.6008877754211426 \n",
+ "Fold 0/4, Epoch 16/19, Minibatch 0/0, Phase valid, Running Loss valid 49.77595138549805, Time 0.6081297397613525, Overall 0.6101250648498535 \n",
+ "Fold 0/4, Epoch 16/19, Running Loss valid 12.443987846374512, Time 0.6081297397613525, Overall 0.6101250648498535 \n",
+ "Fold 0/4, Epoch 17/19, Minibatch 0/0, Phase train, Running Loss train 222.95948791503906, Time 0.6321296691894531, Overall 0.6341249942779541 \n",
+ "Fold 0/4, Epoch 17/19, Running Loss train 18.579957326253254, Time 0.6321296691894531, Overall 0.6341249942779541 \n",
+ "Fold 0/4, Epoch 17/19, Minibatch 0/0, Phase valid, Running Loss valid 45.22557830810547, Time 0.6411302089691162, Overall 0.6431255340576172 \n",
+ "Fold 0/4, Epoch 17/19, Running Loss valid 11.306394577026367, Time 0.6411302089691162, Overall 0.6431255340576172 \n",
+ "Fold 0/4, Epoch 18/19, Minibatch 0/0, Phase train, Running Loss train 215.9012451171875, Time 0.6661326885223389, Overall 0.6681280136108398 \n",
+ "Fold 0/4, Epoch 18/19, Running Loss train 17.991770426432293, Time 0.6661326885223389, Overall 0.6681280136108398 \n",
+ "Fold 0/4, Epoch 18/19, Minibatch 0/0, Phase valid, Running Loss valid 53.447288513183594, Time 0.6751291751861572, Overall 0.6771245002746582 \n",
+ "Fold 0/4, Epoch 18/19, Running Loss valid 13.361822128295898, Time 0.6751291751861572, Overall 0.6771245002746582 \n",
+ "Fold 0/4, Epoch 19/19, Minibatch 0/0, Phase train, Running Loss train 121.39436340332031, Time 0.6980969905853271, Overall 0.7000923156738281 \n",
+ "Fold 0/4, Epoch 19/19, Running Loss train 10.116196950276693, Time 0.6980969905853271, Overall 0.7000923156738281 \n",
+ "Fold 0/4, Epoch 19/19, Minibatch 0/0, Phase valid, Running Loss valid 35.64196014404297, Time 0.7080976963043213, Overall 0.7100930213928223 \n",
+ "Fold 0/4, Epoch 19/19, Running Loss valid 8.910490036010742, Time 0.7080976963043213, Overall 0.7100930213928223 \n",
+ "Fold 1/4, Epoch 0/19, Minibatch 0/0, Phase train, Running Loss train 143.24639892578125, Time 0.022992610931396484, Overall 0.7440850734710693 \n",
+ "Fold 1/4, Epoch 0/19, Running Loss train 11.018953763521635, Time 0.022992610931396484, Overall 0.7440850734710693 \n",
+ "Fold 1/4, Epoch 0/19, Minibatch 0/0, Phase valid, Running Loss valid 68.03410339355469, Time 0.03299379348754883, Overall 0.7540862560272217 \n",
+ "Fold 1/4, Epoch 0/19, Running Loss valid 22.67803446451823, Time 0.03299379348754883, Overall 0.7540862560272217 \n",
+ "Fold 1/4, Epoch 1/19, Minibatch 0/0, Phase train, Running Loss train 153.31979370117188, Time 0.05900073051452637, Overall 0.7800931930541992 \n",
+ "Fold 1/4, Epoch 1/19, Running Loss train 11.793830284705528, Time 0.05900073051452637, Overall 0.7800931930541992 \n",
+ "Fold 1/4, Epoch 1/19, Minibatch 0/0, Phase valid, Running Loss valid 41.99846267700195, Time 0.06803202629089355, Overall 0.7891244888305664 \n",
+ "Fold 1/4, Epoch 1/19, Running Loss valid 13.99948755900065, Time 0.06803202629089355, Overall 0.7891244888305664 \n",
+ "Fold 1/4, Epoch 2/19, Minibatch 0/0, Phase train, Running Loss train 101.59265899658203, Time 0.09268307685852051, Overall 0.8137755393981934 \n",
+ "Fold 1/4, Epoch 2/19, Running Loss train 7.814819922814002, Time 0.09268307685852051, Overall 0.8137755393981934 \n",
+ "Fold 1/4, Epoch 2/19, Minibatch 0/0, Phase valid, Running Loss valid 50.51361083984375, Time 0.10167241096496582, Overall 0.8227648735046387 \n",
+ "Fold 1/4, Epoch 2/19, Running Loss valid 16.837870279947918, Time 0.10167241096496582, Overall 0.8227648735046387 \n",
+ "Fold 1/4, Epoch 3/19, Minibatch 0/0, Phase train, Running Loss train 204.93313598632812, Time 0.12848305702209473, Overall 0.8495755195617676 \n",
+ "Fold 1/4, Epoch 3/19, Running Loss train 15.764087383563702, Time 0.12848305702209473, Overall 0.8495755195617676 \n",
+ "Fold 1/4, Epoch 3/19, Minibatch 0/0, Phase valid, Running Loss valid 27.207908630371094, Time 0.1375746726989746, Overall 0.8586671352386475 \n",
+ "Fold 1/4, Epoch 3/19, Running Loss valid 9.069302876790365, Time 0.1375746726989746, Overall 0.8586671352386475 \n",
+ "Fold 1/4, Epoch 4/19, Minibatch 0/0, Phase train, Running Loss train 103.553466796875, Time 0.16741323471069336, Overall 0.8885056972503662 \n",
+ "Fold 1/4, Epoch 4/19, Running Loss train 7.9656512920673075, Time 0.16741323471069336, Overall 0.8885056972503662 \n",
+ "Fold 1/4, Epoch 4/19, Minibatch 0/0, Phase valid, Running Loss valid 26.640058517456055, Time 0.17760086059570312, Overall 0.898693323135376 \n",
+ "Fold 1/4, Epoch 4/19, Running Loss valid 8.880019505818685, Time 0.17841076850891113, Overall 0.899503231048584 \n",
+ "Fold 1/4, Epoch 5/19, Minibatch 0/0, Phase train, Running Loss train 189.2102813720703, Time 0.20267653465270996, Overall 0.9237689971923828 \n",
+ "Fold 1/4, Epoch 5/19, Running Loss train 14.554637028620792, Time 0.20267653465270996, Overall 0.9237689971923828 \n",
+ "Fold 1/4, Epoch 5/19, Minibatch 0/0, Phase valid, Running Loss valid 51.37736511230469, Time 0.21143484115600586, Overall 0.9325273036956787 \n",
+ "Fold 1/4, Epoch 5/19, Running Loss valid 17.12578837076823, Time 0.2124326229095459, Overall 0.9335250854492188 \n",
+ "Fold 1/4, Epoch 6/19, Minibatch 0/0, Phase train, Running Loss train 175.477783203125, Time 0.23661041259765625, Overall 0.9577028751373291 \n",
+ "Fold 1/4, Epoch 6/19, Running Loss train 13.498291015625, Time 0.23661041259765625, Overall 0.9577028751373291 \n",
+ "Fold 1/4, Epoch 6/19, Minibatch 0/0, Phase valid, Running Loss valid 53.45769500732422, Time 0.2455747127532959, Overall 0.9666671752929688 \n",
+ "Fold 1/4, Epoch 6/19, Running Loss valid 17.819231669108074, Time 0.2455747127532959, Overall 0.9666671752929688 \n",
+ "Fold 1/4, Epoch 7/19, Minibatch 0/0, Phase train, Running Loss train 234.42359924316406, Time 0.27075767517089844, Overall 0.9918501377105713 \n",
+ "Fold 1/4, Epoch 7/19, Running Loss train 18.032584557166466, Time 0.27075767517089844, Overall 0.9918501377105713 \n",
+ "Fold 1/4, Epoch 7/19, Minibatch 0/0, Phase valid, Running Loss valid 76.25323486328125, Time 0.279829740524292, Overall 1.0009222030639648 \n",
+ "Fold 1/4, Epoch 7/19, Running Loss valid 25.417744954427082, Time 0.279829740524292, Overall 1.0009222030639648 \n",
+ "Fold 1/4, Epoch 8/19, Minibatch 0/0, Phase train, Running Loss train 147.52513122558594, Time 0.30521607398986816, Overall 1.026308536529541 \n",
+ "Fold 1/4, Epoch 8/19, Running Loss train 11.348087017352764, Time 0.30521607398986816, Overall 1.026308536529541 \n",
+ "Fold 1/4, Epoch 8/19, Minibatch 0/0, Phase valid, Running Loss valid 59.18119430541992, Time 0.3172450065612793, Overall 1.0383374691009521 \n",
+ "Fold 1/4, Epoch 8/19, Running Loss valid 19.72706476847331, Time 0.31824445724487305, Overall 1.039336919784546 \n",
+ "Fold 1/4, Epoch 9/19, Minibatch 0/0, Phase train, Running Loss train 164.4113006591797, Time 0.3432934284210205, Overall 1.0643858909606934 \n",
+ "Fold 1/4, Epoch 9/19, Running Loss train 12.647023127629208, Time 0.3432934284210205, Overall 1.0643858909606934 \n",
+ "Fold 1/4, Epoch 9/19, Minibatch 0/0, Phase valid, Running Loss valid 27.751352310180664, Time 0.3527226448059082, Overall 1.073815107345581 \n",
+ "Fold 1/4, Epoch 9/19, Running Loss valid 9.25045077006022, Time 0.3527226448059082, Overall 1.073815107345581 \n",
+ "Fold 1/4, Epoch 10/19, Minibatch 0/0, Phase train, Running Loss train 97.6352767944336, Time 0.37741732597351074, Overall 1.0985097885131836 \n",
+ "Fold 1/4, Epoch 10/19, Running Loss train 7.510405907264123, Time 0.37741732597351074, Overall 1.0985097885131836 \n",
+ "Fold 1/4, Epoch 10/19, Minibatch 0/0, Phase valid, Running Loss valid 42.76206970214844, Time 0.3864154815673828, Overall 1.1075079441070557 \n",
+ "Fold 1/4, Epoch 10/19, Running Loss valid 14.254023234049479, Time 0.3864154815673828, Overall 1.1075079441070557 \n",
+ "Fold 1/4, Epoch 11/19, Minibatch 0/0, Phase train, Running Loss train 113.28530883789062, Time 0.4127223491668701, Overall 1.133814811706543 \n",
+ "Fold 1/4, Epoch 11/19, Running Loss train 8.714254525991587, Time 0.4127223491668701, Overall 1.133814811706543 \n",
+ "Fold 1/4, Epoch 11/19, Minibatch 0/0, Phase valid, Running Loss valid 56.55455017089844, Time 0.4206821918487549, Overall 1.1417746543884277 \n",
+ "Fold 1/4, Epoch 11/19, Running Loss valid 18.851516723632812, Time 0.42173314094543457, Overall 1.1428256034851074 \n",
+ "Fold 1/4, Epoch 12/19, Minibatch 0/0, Phase train, Running Loss train 177.08370971679688, Time 0.4457216262817383, Overall 1.1668140888214111 \n",
+ "Fold 1/4, Epoch 12/19, Running Loss train 13.62182382436899, Time 0.44672417640686035, Overall 1.1678166389465332 \n",
+ "Fold 1/4, Epoch 12/19, Minibatch 0/0, Phase valid, Running Loss valid 58.8853645324707, Time 0.454681396484375, Overall 1.1757738590240479 \n",
+ "Fold 1/4, Epoch 12/19, Running Loss valid 19.628454844156902, Time 0.454681396484375, Overall 1.1757738590240479 \n",
+ "Fold 1/4, Epoch 13/19, Minibatch 0/0, Phase train, Running Loss train 71.85808563232422, Time 0.47968268394470215, Overall 1.200775146484375 \n",
+ "Fold 1/4, Epoch 13/19, Running Loss train 5.527545048640325, Time 0.4806861877441406, Overall 1.2017786502838135 \n",
+ "Fold 1/4, Epoch 13/19, Minibatch 0/0, Phase valid, Running Loss valid 62.09322738647461, Time 0.4887230396270752, Overall 1.209815502166748 \n",
+ "Fold 1/4, Epoch 13/19, Running Loss valid 20.697742462158203, Time 0.4887230396270752, Overall 1.209815502166748 \n",
+ "Fold 1/4, Epoch 14/19, Minibatch 0/0, Phase train, Running Loss train 132.32553100585938, Time 0.5146865844726562, Overall 1.235779047012329 \n",
+ "Fold 1/4, Epoch 14/19, Running Loss train 10.178887000450722, Time 0.5146865844726562, Overall 1.235779047012329 \n",
+ "Fold 1/4, Epoch 14/19, Minibatch 0/0, Phase valid, Running Loss valid 65.70878601074219, Time 0.5237219333648682, Overall 1.244814395904541 \n",
+ "Fold 1/4, Epoch 14/19, Running Loss valid 21.902928670247395, Time 0.5237219333648682, Overall 1.244814395904541 \n",
+ "Fold 1/4, Epoch 15/19, Minibatch 0/0, Phase train, Running Loss train 157.72842407226562, Time 0.5497233867645264, Overall 1.2708158493041992 \n",
+ "Fold 1/4, Epoch 15/19, Running Loss train 12.132955697866587, Time 0.5497233867645264, Overall 1.2708158493041992 \n",
+ "Fold 1/4, Epoch 15/19, Minibatch 0/0, Phase valid, Running Loss valid 64.66592407226562, Time 0.558687686920166, Overall 1.2797801494598389 \n",
+ "Fold 1/4, Epoch 15/19, Running Loss valid 21.555308024088543, Time 0.558687686920166, Overall 1.2797801494598389 \n",
+ "Fold 1/4, Epoch 16/19, Minibatch 0/0, Phase train, Running Loss train 54.230648040771484, Time 0.584681510925293, Overall 1.3057739734649658 \n",
+ "Fold 1/4, Epoch 16/19, Running Loss train 4.171588310828576, Time 0.584681510925293, Overall 1.3057739734649658 \n",
+ "Fold 1/4, Epoch 16/19, Minibatch 0/0, Phase valid, Running Loss valid 46.54447555541992, Time 0.5936851501464844, Overall 1.3147776126861572 \n",
+ "Fold 1/4, Epoch 16/19, Running Loss valid 15.514825185139975, Time 0.5946831703186035, Overall 1.3157756328582764 \n",
+ "Fold 1/4, Epoch 17/19, Minibatch 0/0, Phase train, Running Loss train 105.17176055908203, Time 0.619682788848877, Overall 1.3407752513885498 \n",
+ "Fold 1/4, Epoch 17/19, Running Loss train 8.090135427621695, Time 0.619682788848877, Overall 1.3407752513885498 \n",
+ "Fold 1/4, Epoch 17/19, Minibatch 0/0, Phase valid, Running Loss valid 26.436908721923828, Time 0.6288290023803711, Overall 1.349921464920044 \n",
+ "Fold 1/4, Epoch 17/19, Running Loss valid 8.812302907307943, Time 0.6288290023803711, Overall 1.349921464920044 \n",
+ "Fold 1/4, Epoch 18/19, Minibatch 0/0, Phase train, Running Loss train 147.8332977294922, Time 0.6528253555297852, Overall 1.373917818069458 \n",
+ "Fold 1/4, Epoch 18/19, Running Loss train 11.37179213303786, Time 0.6528253555297852, Overall 1.373917818069458 \n",
+ "Fold 1/4, Epoch 18/19, Minibatch 0/0, Phase valid, Running Loss valid 87.29964447021484, Time 0.6617891788482666, Overall 1.3828816413879395 \n",
+ "Fold 1/4, Epoch 18/19, Running Loss valid 29.099881490071613, Time 0.6617891788482666, Overall 1.3828816413879395 \n",
+ "Fold 1/4, Epoch 19/19, Minibatch 0/0, Phase train, Running Loss train 116.48267364501953, Time 0.6867897510528564, Overall 1.4078822135925293 \n",
+ "Fold 1/4, Epoch 19/19, Running Loss train 8.960205665001503, Time 0.6867897510528564, Overall 1.4078822135925293 \n",
+ "Fold 1/4, Epoch 19/19, Minibatch 0/0, Phase valid, Running Loss valid 56.616233825683594, Time 0.6958255767822266, Overall 1.4169180393218994 \n",
+ "Fold 1/4, Epoch 19/19, Running Loss valid 18.87207794189453, Time 0.6958255767822266, Overall 1.4169180393218994 \n",
+ "Fold 2/4, Epoch 0/19, Minibatch 0/0, Phase train, Running Loss train 188.22210693359375, Time 0.020031213760375977, Overall 1.4459116458892822 \n",
+ "Fold 2/4, Epoch 0/19, Running Loss train 14.478623610276442, Time 0.020031213760375977, Overall 1.4459116458892822 \n",
+ "Fold 2/4, Epoch 0/19, Minibatch 0/0, Phase valid, Running Loss valid 25.42812728881836, Time 0.02904224395751953, Overall 1.4549226760864258 \n",
+ "Fold 2/4, Epoch 0/19, Running Loss valid 8.47604242960612, Time 0.02904224395751953, Overall 1.4549226760864258 \n",
+ "Fold 2/4, Epoch 1/19, Minibatch 0/0, Phase train, Running Loss train 92.89910125732422, Time 0.05230069160461426, Overall 1.4781811237335205 \n",
+ "Fold 2/4, Epoch 1/19, Running Loss train 7.146084712101863, Time 0.05230069160461426, Overall 1.4781811237335205 \n",
+ "Fold 2/4, Epoch 1/19, Minibatch 0/0, Phase valid, Running Loss valid 28.455171585083008, Time 0.06132364273071289, Overall 1.4872040748596191 \n",
+ "Fold 2/4, Epoch 1/19, Running Loss valid 9.48505719502767, Time 0.06132364273071289, Overall 1.4872040748596191 \n",
+ "Fold 2/4, Epoch 2/19, Minibatch 0/0, Phase train, Running Loss train 155.53329467773438, Time 0.08526492118835449, Overall 1.5111453533172607 \n",
+ "Fold 2/4, Epoch 2/19, Running Loss train 11.964099590594952, Time 0.08526492118835449, Overall 1.5111453533172607 \n",
+ "Fold 2/4, Epoch 2/19, Minibatch 0/0, Phase valid, Running Loss valid 43.7566032409668, Time 0.09430122375488281, Overall 1.520181655883789 \n",
+ "Fold 2/4, Epoch 2/19, Running Loss valid 14.5855344136556, Time 0.09430122375488281, Overall 1.520181655883789 \n",
+ "Fold 2/4, Epoch 3/19, Minibatch 0/0, Phase train, Running Loss train 134.365234375, Time 0.11826157569885254, Overall 1.5441420078277588 \n",
+ "Fold 2/4, Epoch 3/19, Running Loss train 10.335787259615385, Time 0.11826157569885254, Overall 1.5441420078277588 \n",
+ "Fold 2/4, Epoch 3/19, Minibatch 0/0, Phase valid, Running Loss valid 32.00360870361328, Time 0.1272602081298828, Overall 1.553140640258789 \n",
+ "Fold 2/4, Epoch 3/19, Running Loss valid 10.667869567871094, Time 0.1272602081298828, Overall 1.553140640258789 \n",
+ "Fold 2/4, Epoch 4/19, Minibatch 0/0, Phase train, Running Loss train 188.45669555664062, Time 0.1532602310180664, Overall 1.5791406631469727 \n",
+ "Fold 2/4, Epoch 4/19, Running Loss train 14.496668888972355, Time 0.1532602310180664, Overall 1.5791406631469727 \n",
+ "Fold 2/4, Epoch 4/19, Minibatch 0/0, Phase valid, Running Loss valid 17.06161117553711, Time 0.1622607707977295, Overall 1.5881412029266357 \n",
+ "Fold 2/4, Epoch 4/19, Running Loss valid 5.687203725179036, Time 0.1622607707977295, Overall 1.5881412029266357 \n",
+ "Fold 2/4, Epoch 5/19, Minibatch 0/0, Phase train, Running Loss train 250.71546936035156, Time 0.18726348876953125, Overall 1.6131439208984375 \n",
+ "Fold 2/4, Epoch 5/19, Running Loss train 19.28580533541166, Time 0.18726348876953125, Overall 1.6131439208984375 \n",
+ "Fold 2/4, Epoch 5/19, Minibatch 0/0, Phase valid, Running Loss valid 11.606767654418945, Time 0.1962890625, Overall 1.6221694946289062 \n",
+ "Fold 2/4, Epoch 5/19, Running Loss valid 3.868922551472982, Time 0.1962890625, Overall 1.6221694946289062 \n",
+ "Fold 2/4, Epoch 6/19, Minibatch 0/0, Phase train, Running Loss train 140.74143981933594, Time 0.2193005084991455, Overall 1.6451809406280518 \n",
+ "Fold 2/4, Epoch 6/19, Running Loss train 10.82626460148738, Time 0.22026348114013672, Overall 1.646143913269043 \n",
+ "Fold 2/4, Epoch 6/19, Minibatch 0/0, Phase valid, Running Loss valid 37.543678283691406, Time 0.22854089736938477, Overall 1.654421329498291 \n",
+ "Fold 2/4, Epoch 6/19, Running Loss valid 12.514559427897135, Time 0.22854089736938477, Overall 1.654421329498291 \n",
+ "Fold 2/4, Epoch 7/19, Minibatch 0/0, Phase train, Running Loss train 157.45712280273438, Time 0.2523066997528076, Overall 1.6781871318817139 \n",
+ "Fold 2/4, Epoch 7/19, Running Loss train 12.112086369441105, Time 0.25249290466308594, Overall 1.6783733367919922 \n",
+ "Fold 2/4, Epoch 7/19, Minibatch 0/0, Phase valid, Running Loss valid 39.14799118041992, Time 0.26053881645202637, Overall 1.6864192485809326 \n",
+ "Fold 2/4, Epoch 7/19, Running Loss valid 13.049330393473307, Time 0.26053881645202637, Overall 1.6864192485809326 \n",
+ "Fold 2/4, Epoch 8/19, Minibatch 0/0, Phase train, Running Loss train 182.251953125, Time 0.2848491668701172, Overall 1.7107295989990234 \n",
+ "Fold 2/4, Epoch 8/19, Running Loss train 14.019381009615385, Time 0.2848491668701172, Overall 1.7107295989990234 \n",
+ "Fold 2/4, Epoch 8/19, Minibatch 0/0, Phase valid, Running Loss valid 33.55611038208008, Time 0.2938530445098877, Overall 1.719733476638794 \n",
+ "Fold 2/4, Epoch 8/19, Running Loss valid 11.185370127360025, Time 0.2938530445098877, Overall 1.719733476638794 \n",
+ "Fold 2/4, Epoch 9/19, Minibatch 0/0, Phase train, Running Loss train 133.43484497070312, Time 0.31984686851501465, Overall 1.745727300643921 \n",
+ "Fold 2/4, Epoch 9/19, Running Loss train 10.26421884390024, Time 0.31984686851501465, Overall 1.745727300643921 \n",
+ "Fold 2/4, Epoch 9/19, Minibatch 0/0, Phase valid, Running Loss valid 19.15946388244629, Time 0.3288459777832031, Overall 1.7547264099121094 \n",
+ "Fold 2/4, Epoch 9/19, Running Loss valid 6.38648796081543, Time 0.3288459777832031, Overall 1.7547264099121094 \n",
+ "Fold 2/4, Epoch 10/19, Minibatch 0/0, Phase train, Running Loss train 127.42709350585938, Time 0.3640174865722656, Overall 1.7898979187011719 \n",
+ "Fold 2/4, Epoch 10/19, Running Loss train 9.802084115835337, Time 0.3640174865722656, Overall 1.7898979187011719 \n",
+ "Fold 2/4, Epoch 10/19, Minibatch 0/0, Phase valid, Running Loss valid 30.757869720458984, Time 0.3761591911315918, Overall 1.802039623260498 \n",
+ "Fold 2/4, Epoch 10/19, Running Loss valid 10.252623240152994, Time 0.3761591911315918, Overall 1.802039623260498 \n",
+ "Fold 2/4, Epoch 11/19, Minibatch 0/0, Phase train, Running Loss train 79.99552917480469, Time 0.4038388729095459, Overall 1.8297193050384521 \n",
+ "Fold 2/4, Epoch 11/19, Running Loss train 6.153502244215745, Time 0.4038388729095459, Overall 1.8297193050384521 \n",
+ "Fold 2/4, Epoch 11/19, Minibatch 0/0, Phase valid, Running Loss valid 22.445987701416016, Time 0.4128403663635254, Overall 1.8387207984924316 \n",
+ "Fold 2/4, Epoch 11/19, Running Loss valid 7.481995900472005, Time 0.41383886337280273, Overall 1.839719295501709 \n",
+ "Fold 2/4, Epoch 12/19, Minibatch 0/0, Phase train, Running Loss train 116.212646484375, Time 0.4487171173095703, Overall 1.8745975494384766 \n",
+ "Fold 2/4, Epoch 12/19, Running Loss train 8.939434344951923, Time 0.4487171173095703, Overall 1.8745975494384766 \n",
+ "Fold 2/4, Epoch 12/19, Minibatch 0/0, Phase valid, Running Loss valid 36.823143005371094, Time 0.4581766128540039, Overall 1.8840570449829102 \n",
+ "Fold 2/4, Epoch 12/19, Running Loss valid 12.274381001790365, Time 0.4581766128540039, Overall 1.8840570449829102 \n",
+ "Fold 2/4, Epoch 13/19, Minibatch 0/0, Phase train, Running Loss train 143.07899475097656, Time 0.48307251930236816, Overall 1.9089529514312744 \n",
+ "Fold 2/4, Epoch 13/19, Running Loss train 11.00607651930589, Time 0.48307251930236816, Overall 1.9089529514312744 \n",
+ "Fold 2/4, Epoch 13/19, Minibatch 0/0, Phase valid, Running Loss valid 9.73843002319336, Time 0.49207162857055664, Overall 1.917952060699463 \n",
+ "Fold 2/4, Epoch 13/19, Running Loss valid 3.246143341064453, Time 0.49207162857055664, Overall 1.917952060699463 \n",
+ "Fold 2/4, Epoch 14/19, Minibatch 0/0, Phase train, Running Loss train 124.80842590332031, Time 0.5170810222625732, Overall 1.9429614543914795 \n",
+ "Fold 2/4, Epoch 14/19, Running Loss train 9.600648146409254, Time 0.5170810222625732, Overall 1.9429614543914795 \n",
+ "Fold 2/4, Epoch 14/19, Minibatch 0/0, Phase valid, Running Loss valid 39.44697570800781, Time 0.5267333984375, Overall 1.9526138305664062 \n",
+ "Fold 2/4, Epoch 14/19, Running Loss valid 13.148991902669271, Time 0.5267333984375, Overall 1.9526138305664062 \n",
+ "Fold 2/4, Epoch 15/19, Minibatch 0/0, Phase train, Running Loss train 82.21380615234375, Time 0.5506973266601562, Overall 1.9765777587890625 \n",
+ "Fold 2/4, Epoch 15/19, Running Loss train 6.324138934795673, Time 0.5516958236694336, Overall 1.9775762557983398 \n",
+ "Fold 2/4, Epoch 15/19, Minibatch 0/0, Phase valid, Running Loss valid 10.006136894226074, Time 0.560734748840332, Overall 1.9866151809692383 \n",
+ "Fold 2/4, Epoch 15/19, Running Loss valid 3.335378964742025, Time 0.5616958141326904, Overall 1.9875762462615967 \n",
+ "Fold 2/4, Epoch 16/19, Minibatch 0/0, Phase train, Running Loss train 64.39615631103516, Time 0.5886998176574707, Overall 2.014580249786377 \n",
+ "Fold 2/4, Epoch 16/19, Running Loss train 4.953550485464243, Time 0.5886998176574707, Overall 2.014580249786377 \n",
+ "Fold 2/4, Epoch 16/19, Minibatch 0/0, Phase valid, Running Loss valid 34.03740692138672, Time 0.5997004508972168, Overall 2.025580883026123 \n",
+ "Fold 2/4, Epoch 16/19, Running Loss valid 11.345802307128906, Time 0.5997004508972168, Overall 2.025580883026123 \n",
+ "Fold 2/4, Epoch 17/19, Minibatch 0/0, Phase train, Running Loss train 149.45712280273438, Time 0.6264023780822754, Overall 2.0522828102111816 \n",
+ "Fold 2/4, Epoch 17/19, Running Loss train 11.49670175405649, Time 0.6274042129516602, Overall 2.0532846450805664 \n",
+ "Fold 2/4, Epoch 17/19, Minibatch 0/0, Phase valid, Running Loss valid 27.49114227294922, Time 0.6355555057525635, Overall 2.0614359378814697 \n",
+ "Fold 2/4, Epoch 17/19, Running Loss valid 9.163714090983072, Time 0.6355555057525635, Overall 2.0614359378814697 \n",
+ "Fold 2/4, Epoch 18/19, Minibatch 0/0, Phase train, Running Loss train 107.77920532226562, Time 0.6609275341033936, Overall 2.0868079662323 \n",
+ "Fold 2/4, Epoch 18/19, Running Loss train 8.29070810171274, Time 0.6609275341033936, Overall 2.0868079662323 \n",
+ "Fold 2/4, Epoch 18/19, Minibatch 0/0, Phase valid, Running Loss valid 22.910139083862305, Time 0.6689245700836182, Overall 2.0948050022125244 \n",
+ "Fold 2/4, Epoch 18/19, Running Loss valid 7.636713027954102, Time 0.6689245700836182, Overall 2.0948050022125244 \n",
+ "Fold 2/4, Epoch 19/19, Minibatch 0/0, Phase train, Running Loss train 134.6446533203125, Time 0.69706130027771, Overall 2.122941732406616 \n",
+ "Fold 2/4, Epoch 19/19, Running Loss train 10.357281024639423, Time 0.69706130027771, Overall 2.122941732406616 \n",
+ "Fold 2/4, Epoch 19/19, Minibatch 0/0, Phase valid, Running Loss valid 7.223106384277344, Time 0.7050580978393555, Overall 2.1309385299682617 \n",
+ "Fold 2/4, Epoch 19/19, Running Loss valid 2.4077021280924478, Time 0.7060575485229492, Overall 2.1319379806518555 \n",
+ "Fold 3/4, Epoch 0/19, Minibatch 0/0, Phase train, Running Loss train 235.2511749267578, Time 0.026021480560302734, Overall 2.1722986698150635 \n",
+ "Fold 3/4, Epoch 0/19, Running Loss train 18.096244225135216, Time 0.026021480560302734, Overall 2.1722986698150635 \n",
+ "Fold 3/4, Epoch 0/19, Minibatch 0/0, Phase valid, Running Loss valid 69.53622436523438, Time 0.03543829917907715, Overall 2.181715488433838 \n",
+ "Fold 3/4, Epoch 0/19, Running Loss valid 23.178741455078125, Time 0.03543829917907715, Overall 2.181715488433838 \n",
+ "Fold 3/4, Epoch 1/19, Minibatch 0/0, Phase train, Running Loss train 190.74822998046875, Time 0.060552358627319336, Overall 2.20682954788208 \n",
+ "Fold 3/4, Epoch 1/19, Running Loss train 14.672940767728365, Time 0.060552358627319336, Overall 2.20682954788208 \n",
+ "Fold 3/4, Epoch 1/19, Minibatch 0/0, Phase valid, Running Loss valid 30.674692153930664, Time 0.06950998306274414, Overall 2.215787172317505 \n",
+ "Fold 3/4, Epoch 1/19, Running Loss valid 10.224897384643555, Time 0.06950998306274414, Overall 2.215787172317505 \n",
+ "Fold 3/4, Epoch 2/19, Minibatch 0/0, Phase train, Running Loss train 252.19219970703125, Time 0.09309101104736328, Overall 2.239368200302124 \n",
+ "Fold 3/4, Epoch 2/19, Running Loss train 19.399399977463943, Time 0.09309101104736328, Overall 2.239368200302124 \n",
+ "Fold 3/4, Epoch 2/19, Minibatch 0/0, Phase valid, Running Loss valid 37.449153900146484, Time 0.10206031799316406, Overall 2.248337507247925 \n",
+ "Fold 3/4, Epoch 2/19, Running Loss valid 12.483051300048828, Time 0.10206031799316406, Overall 2.248337507247925 \n",
+ "Fold 3/4, Epoch 3/19, Minibatch 0/0, Phase train, Running Loss train 131.9229736328125, Time 0.1254892349243164, Overall 2.271766424179077 \n",
+ "Fold 3/4, Epoch 3/19, Running Loss train 10.147921048677885, Time 0.1254892349243164, Overall 2.271766424179077 \n",
+ "Fold 3/4, Epoch 3/19, Minibatch 0/0, Phase valid, Running Loss valid 34.53129959106445, Time 0.13472247123718262, Overall 2.2809996604919434 \n",
+ "Fold 3/4, Epoch 3/19, Running Loss valid 11.510433197021484, Time 0.13497114181518555, Overall 2.2812483310699463 \n",
+ "Fold 3/4, Epoch 4/19, Minibatch 0/0, Phase train, Running Loss train 157.80264282226562, Time 0.15857219696044922, Overall 2.30484938621521 \n",
+ "Fold 3/4, Epoch 4/19, Running Loss train 12.138664832481972, Time 0.15857219696044922, Overall 2.30484938621521 \n",
+ "Fold 3/4, Epoch 4/19, Minibatch 0/0, Phase valid, Running Loss valid 40.73909378051758, Time 0.16658926010131836, Overall 2.312866449356079 \n",
+ "Fold 3/4, Epoch 4/19, Running Loss valid 13.579697926839193, Time 0.16658926010131836, Overall 2.312866449356079 \n",
+ "Fold 3/4, Epoch 5/19, Minibatch 0/0, Phase train, Running Loss train 73.5020751953125, Time 0.19022440910339355, Overall 2.3365015983581543 \n",
+ "Fold 3/4, Epoch 5/19, Running Loss train 5.6540057842548075, Time 0.19022440910339355, Overall 2.3374643325805664 \n",
+ "Fold 3/4, Epoch 5/19, Minibatch 0/0, Phase valid, Running Loss valid 48.32692337036133, Time 0.19922709465026855, Overall 2.3455042839050293 \n",
+ "Fold 3/4, Epoch 5/19, Running Loss valid 16.10897445678711, Time 0.19922709465026855, Overall 2.3455042839050293 \n",
+ "Fold 3/4, Epoch 6/19, Minibatch 0/0, Phase train, Running Loss train 143.3633575439453, Time 0.22268342971801758, Overall 2.3689606189727783 \n",
+ "Fold 3/4, Epoch 6/19, Running Loss train 11.027950580303486, Time 0.22268342971801758, Overall 2.3689606189727783 \n",
+ "Fold 3/4, Epoch 6/19, Minibatch 0/0, Phase valid, Running Loss valid 39.17284393310547, Time 0.2308950424194336, Overall 2.3771722316741943 \n",
+ "Fold 3/4, Epoch 6/19, Running Loss valid 13.05761464436849, Time 0.2308950424194336, Overall 2.3771722316741943 \n",
+ "Fold 3/4, Epoch 7/19, Minibatch 0/0, Phase train, Running Loss train 217.56857299804688, Time 0.25809788703918457, Overall 2.4043750762939453 \n",
+ "Fold 3/4, Epoch 7/19, Running Loss train 16.736044076772835, Time 0.25809788703918457, Overall 2.4043750762939453 \n",
+ "Fold 3/4, Epoch 7/19, Minibatch 0/0, Phase valid, Running Loss valid 47.37115478515625, Time 0.26624608039855957, Overall 2.4125232696533203 \n",
+ "Fold 3/4, Epoch 7/19, Running Loss valid 15.790384928385416, Time 0.26624608039855957, Overall 2.4125232696533203 \n",
+ "Fold 3/4, Epoch 8/19, Minibatch 0/0, Phase train, Running Loss train 212.71543884277344, Time 0.2917134761810303, Overall 2.437990665435791 \n",
+ "Fold 3/4, Epoch 8/19, Running Loss train 16.362726064828728, Time 0.2927525043487549, Overall 2.4390296936035156 \n",
+ "Fold 3/4, Epoch 8/19, Minibatch 0/0, Phase valid, Running Loss valid 46.650657653808594, Time 0.30071449279785156, Overall 2.4469916820526123 \n",
+ "Fold 3/4, Epoch 8/19, Running Loss valid 15.550219217936197, Time 0.30071449279785156, Overall 2.4469916820526123 \n",
+ "Fold 3/4, Epoch 9/19, Minibatch 0/0, Phase train, Running Loss train 172.74009704589844, Time 0.32602858543395996, Overall 2.4723057746887207 \n",
+ "Fold 3/4, Epoch 9/19, Running Loss train 13.287699772761417, Time 0.32602858543395996, Overall 2.4723057746887207 \n",
+ "Fold 3/4, Epoch 9/19, Minibatch 0/0, Phase valid, Running Loss valid 24.515403747558594, Time 0.3362588882446289, Overall 2.4825360774993896 \n",
+ "Fold 3/4, Epoch 9/19, Running Loss valid 8.171801249186197, Time 0.3362588882446289, Overall 2.4825360774993896 \n",
+ "Fold 3/4, Epoch 10/19, Minibatch 0/0, Phase train, Running Loss train 148.35494995117188, Time 0.36325955390930176, Overall 2.5095367431640625 \n",
+ "Fold 3/4, Epoch 10/19, Running Loss train 11.411919227013222, Time 0.36325955390930176, Overall 2.5095367431640625 \n",
+ "Fold 3/4, Epoch 10/19, Minibatch 0/0, Phase valid, Running Loss valid 31.7917423248291, Time 0.37225914001464844, Overall 2.518536329269409 \n",
+ "Fold 3/4, Epoch 10/19, Running Loss valid 10.597247441609701, Time 0.37225914001464844, Overall 2.518536329269409 \n",
+ "Fold 3/4, Epoch 11/19, Minibatch 0/0, Phase train, Running Loss train 97.50045013427734, Time 0.399486780166626, Overall 2.5457639694213867 \n",
+ "Fold 3/4, Epoch 11/19, Running Loss train 7.500034625713642, Time 0.399486780166626, Overall 2.5457639694213867 \n",
+ "Fold 3/4, Epoch 11/19, Minibatch 0/0, Phase valid, Running Loss valid 53.986576080322266, Time 0.4123551845550537, Overall 2.5586323738098145 \n",
+ "Fold 3/4, Epoch 11/19, Running Loss valid 17.995525360107422, Time 0.4123551845550537, Overall 2.5586323738098145 \n",
+ "Fold 3/4, Epoch 12/19, Minibatch 0/0, Phase train, Running Loss train 127.84117889404297, Time 0.44428563117980957, Overall 2.5905628204345703 \n",
+ "Fold 3/4, Epoch 12/19, Running Loss train 9.833936838003305, Time 0.44428563117980957, Overall 2.5905628204345703 \n",
+ "Fold 3/4, Epoch 12/19, Minibatch 0/0, Phase valid, Running Loss valid 25.29738998413086, Time 0.4542868137359619, Overall 2.6005640029907227 \n",
+ "Fold 3/4, Epoch 12/19, Running Loss valid 8.43246332804362, Time 0.4542868137359619, Overall 2.6005640029907227 \n",
+ "Fold 3/4, Epoch 13/19, Minibatch 0/0, Phase train, Running Loss train 128.30596923828125, Time 0.4860382080078125, Overall 2.6323153972625732 \n",
+ "Fold 3/4, Epoch 13/19, Running Loss train 9.86968994140625, Time 0.4860382080078125, Overall 2.6323153972625732 \n",
+ "Fold 3/4, Epoch 13/19, Minibatch 0/0, Phase valid, Running Loss valid 38.45119094848633, Time 0.5000584125518799, Overall 2.6463356018066406 \n",
+ "Fold 3/4, Epoch 13/19, Running Loss valid 12.817063649495443, Time 0.5010602474212646, Overall 2.6473374366760254 \n",
+ "Fold 3/4, Epoch 14/19, Minibatch 0/0, Phase train, Running Loss train 138.154541015625, Time 0.5335688591003418, Overall 2.6798460483551025 \n",
+ "Fold 3/4, Epoch 14/19, Running Loss train 10.627272385817308, Time 0.5335688591003418, Overall 2.6798460483551025 \n",
+ "Fold 3/4, Epoch 14/19, Minibatch 0/0, Phase valid, Running Loss valid 36.31549072265625, Time 0.5456972122192383, Overall 2.691974401473999 \n",
+ "Fold 3/4, Epoch 14/19, Running Loss valid 12.10516357421875, Time 0.5456972122192383, Overall 2.691974401473999 \n",
+ "Fold 3/4, Epoch 15/19, Minibatch 0/0, Phase train, Running Loss train 131.36422729492188, Time 0.578819751739502, Overall 2.7250969409942627 \n",
+ "Fold 3/4, Epoch 15/19, Running Loss train 10.104940561147837, Time 0.578819751739502, Overall 2.7250969409942627 \n",
+ "Fold 3/4, Epoch 15/19, Minibatch 0/0, Phase valid, Running Loss valid 26.20533561706543, Time 0.589928150177002, Overall 2.7362053394317627 \n",
+ "Fold 3/4, Epoch 15/19, Running Loss valid 8.735111872355143, Time 0.589928150177002, Overall 2.7362053394317627 \n",
+ "Fold 3/4, Epoch 16/19, Minibatch 0/0, Phase train, Running Loss train 102.89434814453125, Time 0.6234142780303955, Overall 2.7696914672851562 \n",
+ "Fold 3/4, Epoch 16/19, Running Loss train 7.914949857271635, Time 0.6234142780303955, Overall 2.7696914672851562 \n",
+ "Fold 3/4, Epoch 16/19, Minibatch 0/0, Phase valid, Running Loss valid 30.560850143432617, Time 0.6334211826324463, Overall 2.779698371887207 \n",
+ "Fold 3/4, Epoch 16/19, Running Loss valid 10.186950047810873, Time 0.634418249130249, Overall 2.7806954383850098 \n",
+ "Fold 3/4, Epoch 17/19, Minibatch 0/0, Phase train, Running Loss train 84.55962371826172, Time 0.6690185070037842, Overall 2.815295696258545 \n",
+ "Fold 3/4, Epoch 17/19, Running Loss train 6.504586439866286, Time 0.6690185070037842, Overall 2.815295696258545 \n",
+ "Fold 3/4, Epoch 17/19, Minibatch 0/0, Phase valid, Running Loss valid 38.085655212402344, Time 0.6800117492675781, Overall 2.826288938522339 \n",
+ "Fold 3/4, Epoch 17/19, Running Loss valid 12.695218404134115, Time 0.6800117492675781, Overall 2.826288938522339 \n",
+ "Fold 3/4, Epoch 18/19, Minibatch 0/0, Phase train, Running Loss train 72.6573257446289, Time 0.7180130481719971, Overall 2.864290237426758 \n",
+ "Fold 3/4, Epoch 18/19, Running Loss train 5.589025057279146, Time 0.7180130481719971, Overall 2.864290237426758 \n",
+ "Fold 3/4, Epoch 18/19, Minibatch 0/0, Phase valid, Running Loss valid 46.84284591674805, Time 0.7260105609893799, Overall 2.8722877502441406 \n",
+ "Fold 3/4, Epoch 18/19, Running Loss valid 15.61428197224935, Time 0.72701096534729, Overall 2.873288154602051 \n",
+ "Fold 3/4, Epoch 19/19, Minibatch 0/0, Phase train, Running Loss train 139.55865478515625, Time 0.7600202560424805, Overall 2.906297445297241 \n",
+ "Fold 3/4, Epoch 19/19, Running Loss train 10.735281137319712, Time 0.7600202560424805, Overall 2.906297445297241 \n",
+ "Fold 3/4, Epoch 19/19, Minibatch 0/0, Phase valid, Running Loss valid 20.921478271484375, Time 0.7690136432647705, Overall 2.9152908325195312 \n",
+ "Fold 3/4, Epoch 19/19, Running Loss valid 6.973826090494792, Time 0.7690136432647705, Overall 2.9152908325195312 \n",
+ "Fold 4/4, Epoch 0/19, Minibatch 0/0, Phase train, Running Loss train 164.28207397460938, Time 0.022998571395874023, Overall 2.948288679122925 \n",
+ "Fold 4/4, Epoch 0/19, Running Loss train 12.63708261343149, Time 0.022998571395874023, Overall 2.948288679122925 \n",
+ "Fold 4/4, Epoch 0/19, Minibatch 0/0, Phase valid, Running Loss valid 45.369056701660156, Time 0.03800058364868164, Overall 2.9632906913757324 \n",
+ "Fold 4/4, Epoch 0/19, Running Loss valid 15.123018900553385, Time 0.03800058364868164, Overall 2.9632906913757324 \n",
+ "Fold 4/4, Epoch 1/19, Minibatch 0/0, Phase train, Running Loss train 140.70533752441406, Time 0.0740060806274414, Overall 2.999296188354492 \n",
+ "Fold 4/4, Epoch 1/19, Running Loss train 10.823487501878004, Time 0.0740060806274414, Overall 2.999296188354492 \n",
+ "Fold 4/4, Epoch 1/19, Minibatch 0/0, Phase valid, Running Loss valid 23.810890197753906, Time 0.08252191543579102, Overall 3.007812023162842 \n",
+ "Fold 4/4, Epoch 1/19, Running Loss valid 7.936963399251302, Time 0.08252191543579102, Overall 3.007812023162842 \n",
+ "Fold 4/4, Epoch 2/19, Minibatch 0/0, Phase train, Running Loss train 176.64556884765625, Time 0.11450481414794922, Overall 3.039794921875 \n",
+ "Fold 4/4, Epoch 2/19, Running Loss train 13.588120680588942, Time 0.11450481414794922, Overall 3.039794921875 \n",
+ "Fold 4/4, Epoch 2/19, Minibatch 0/0, Phase valid, Running Loss valid 17.669809341430664, Time 0.12384343147277832, Overall 3.049133539199829 \n",
+ "Fold 4/4, Epoch 2/19, Running Loss valid 5.889936447143555, Time 0.12384343147277832, Overall 3.049133539199829 \n",
+ "Fold 4/4, Epoch 3/19, Minibatch 0/0, Phase train, Running Loss train 195.81195068359375, Time 0.14884638786315918, Overall 3.07413649559021 \n",
+ "Fold 4/4, Epoch 3/19, Running Loss train 15.062457744891827, Time 0.14884638786315918, Overall 3.07413649559021 \n",
+ "Fold 4/4, Epoch 3/19, Minibatch 0/0, Phase valid, Running Loss valid 39.49605178833008, Time 0.15784597396850586, Overall 3.0831360816955566 \n",
+ "Fold 4/4, Epoch 3/19, Running Loss valid 13.165350596110025, Time 0.15784597396850586, Overall 3.0831360816955566 \n",
+ "Fold 4/4, Epoch 4/19, Minibatch 0/0, Phase train, Running Loss train 136.81625366210938, Time 0.18484234809875488, Overall 3.1101324558258057 \n",
+ "Fold 4/4, Epoch 4/19, Running Loss train 10.524327204777645, Time 0.18484234809875488, Overall 3.1101324558258057 \n",
+ "Fold 4/4, Epoch 4/19, Minibatch 0/0, Phase valid, Running Loss valid 38.09334182739258, Time 0.19684553146362305, Overall 3.122135639190674 \n",
+ "Fold 4/4, Epoch 4/19, Running Loss valid 12.69778060913086, Time 0.19684553146362305, Overall 3.122135639190674 \n",
+ "Fold 4/4, Epoch 5/19, Minibatch 0/0, Phase train, Running Loss train 140.5540313720703, Time 0.22484254837036133, Overall 3.150132656097412 \n",
+ "Fold 4/4, Epoch 5/19, Running Loss train 10.811848567082333, Time 0.22484254837036133, Overall 3.150132656097412 \n",
+ "Fold 4/4, Epoch 5/19, Minibatch 0/0, Phase valid, Running Loss valid 24.40944480895996, Time 0.23284316062927246, Overall 3.1581332683563232 \n",
+ "Fold 4/4, Epoch 5/19, Running Loss valid 8.136481602986654, Time 0.233842134475708, Overall 3.159132242202759 \n",
+ "Fold 4/4, Epoch 6/19, Minibatch 0/0, Phase train, Running Loss train 210.15963745117188, Time 0.25884318351745605, Overall 3.184133291244507 \n",
+ "Fold 4/4, Epoch 6/19, Running Loss train 16.16612595778245, Time 0.25884318351745605, Overall 3.184133291244507 \n",
+ "Fold 4/4, Epoch 6/19, Minibatch 0/0, Phase valid, Running Loss valid 24.65675163269043, Time 0.26784229278564453, Overall 3.1931324005126953 \n",
+ "Fold 4/4, Epoch 6/19, Running Loss valid 8.21891721089681, Time 0.2688429355621338, Overall 3.1941330432891846 \n",
+ "Fold 4/4, Epoch 7/19, Minibatch 0/0, Phase train, Running Loss train 148.053955078125, Time 0.29694485664367676, Overall 3.2222349643707275 \n",
+ "Fold 4/4, Epoch 7/19, Running Loss train 11.388765775240385, Time 0.29694485664367676, Overall 3.2222349643707275 \n",
+ "Fold 4/4, Epoch 7/19, Minibatch 0/0, Phase valid, Running Loss valid 46.331695556640625, Time 0.30594873428344727, Overall 3.231238842010498 \n",
+ "Fold 4/4, Epoch 7/19, Running Loss valid 15.443898518880209, Time 0.30594873428344727, Overall 3.231238842010498 \n",
+ "Fold 4/4, Epoch 8/19, Minibatch 0/0, Phase train, Running Loss train 227.8346405029297, Time 0.3329458236694336, Overall 3.2582359313964844 \n",
+ "Fold 4/4, Epoch 8/19, Running Loss train 17.525741577148438, Time 0.3329458236694336, Overall 3.2582359313964844 \n",
+ "Fold 4/4, Epoch 8/19, Minibatch 0/0, Phase valid, Running Loss valid 43.96866989135742, Time 0.34183239936828613, Overall 3.267122507095337 \n",
+ "Fold 4/4, Epoch 8/19, Running Loss valid 14.65622329711914, Time 0.34183239936828613, Overall 3.267122507095337 \n",
+ "Fold 4/4, Epoch 9/19, Minibatch 0/0, Phase train, Running Loss train 218.80300903320312, Time 0.370100736618042, Overall 3.2953908443450928 \n",
+ "Fold 4/4, Epoch 9/19, Running Loss train 16.83100069486178, Time 0.370100736618042, Overall 3.2953908443450928 \n",
+ "Fold 4/4, Epoch 9/19, Minibatch 0/0, Phase valid, Running Loss valid 25.38977813720703, Time 0.3791024684906006, Overall 3.3043925762176514 \n",
+ "Fold 4/4, Epoch 9/19, Running Loss valid 8.46325937906901, Time 0.3801088333129883, Overall 3.305398941040039 \n",
+ "Fold 4/4, Epoch 10/19, Minibatch 0/0, Phase train, Running Loss train 132.7415008544922, Time 0.41510486602783203, Overall 3.340394973754883 \n",
+ "Fold 4/4, Epoch 10/19, Running Loss train 10.210884681114784, Time 0.41510486602783203, Overall 3.340394973754883 \n",
+ "Fold 4/4, Epoch 10/19, Minibatch 0/0, Phase valid, Running Loss valid 23.338838577270508, Time 0.4261000156402588, Overall 3.3513901233673096 \n",
+ "Fold 4/4, Epoch 10/19, Running Loss valid 7.779612859090169, Time 0.4261000156402588, Overall 3.3513901233673096 \n",
+ "Fold 4/4, Epoch 11/19, Minibatch 0/0, Phase train, Running Loss train 115.78433227539062, Time 0.4521005153656006, Overall 3.3773906230926514 \n",
+ "Fold 4/4, Epoch 11/19, Running Loss train 8.906487098106972, Time 0.4521005153656006, Overall 3.3773906230926514 \n",
+ "Fold 4/4, Epoch 11/19, Minibatch 0/0, Phase valid, Running Loss valid 40.697303771972656, Time 0.4611022472381592, Overall 3.38639235496521 \n",
+ "Fold 4/4, Epoch 11/19, Running Loss valid 13.565767923990885, Time 0.4611022472381592, Overall 3.38639235496521 \n",
+ "Fold 4/4, Epoch 12/19, Minibatch 0/0, Phase train, Running Loss train 201.9030303955078, Time 0.48734545707702637, Overall 3.412635564804077 \n",
+ "Fold 4/4, Epoch 12/19, Running Loss train 15.531002338115986, Time 0.48734545707702637, Overall 3.412635564804077 \n",
+ "Fold 4/4, Epoch 12/19, Minibatch 0/0, Phase valid, Running Loss valid 50.38485336303711, Time 0.4973471164703369, Overall 3.4226372241973877 \n",
+ "Fold 4/4, Epoch 12/19, Running Loss valid 16.79495112101237, Time 0.4973471164703369, Overall 3.4226372241973877 \n",
+ "Fold 4/4, Epoch 13/19, Minibatch 0/0, Phase train, Running Loss train 170.98562622070312, Time 0.5503287315368652, Overall 3.475618839263916 \n",
+ "Fold 4/4, Epoch 13/19, Running Loss train 13.152740478515625, Time 0.5503287315368652, Overall 3.475618839263916 \n",
+ "Fold 4/4, Epoch 13/19, Minibatch 0/0, Phase valid, Running Loss valid 21.86351776123047, Time 0.5635988712310791, Overall 3.48888897895813 \n",
+ "Fold 4/4, Epoch 13/19, Running Loss valid 7.287839253743489, Time 0.5635988712310791, Overall 3.48888897895813 \n",
+ "Fold 4/4, Epoch 14/19, Minibatch 0/0, Phase train, Running Loss train 76.59513092041016, Time 0.5939533710479736, Overall 3.5192434787750244 \n",
+ "Fold 4/4, Epoch 14/19, Running Loss train 5.891933147723858, Time 0.5949525833129883, Overall 3.520242691040039 \n",
+ "Fold 4/4, Epoch 14/19, Minibatch 0/0, Phase valid, Running Loss valid 52.189476013183594, Time 0.6061358451843262, Overall 3.531425952911377 \n",
+ "Fold 4/4, Epoch 14/19, Running Loss valid 17.39649200439453, Time 0.6061358451843262, Overall 3.531425952911377 \n",
+ "Fold 4/4, Epoch 15/19, Minibatch 0/0, Phase train, Running Loss train 100.11666870117188, Time 0.6322987079620361, Overall 3.557588815689087 \n",
+ "Fold 4/4, Epoch 15/19, Running Loss train 7.701282207782452, Time 0.6322987079620361, Overall 3.557588815689087 \n",
+ "Fold 4/4, Epoch 15/19, Minibatch 0/0, Phase valid, Running Loss valid 54.379581451416016, Time 0.6433010101318359, Overall 3.5685911178588867 \n",
+ "Fold 4/4, Epoch 15/19, Running Loss valid 18.126527150472004, Time 0.6433010101318359, Overall 3.5685911178588867 \n",
+ "Fold 4/4, Epoch 16/19, Minibatch 0/0, Phase train, Running Loss train 104.28076934814453, Time 0.6692976951599121, Overall 3.594587802886963 \n",
+ "Fold 4/4, Epoch 16/19, Running Loss train 8.021597642164965, Time 0.6692976951599121, Overall 3.594587802886963 \n",
+ "Fold 4/4, Epoch 16/19, Minibatch 0/0, Phase valid, Running Loss valid 45.9799919128418, Time 0.6782965660095215, Overall 3.6035866737365723 \n",
+ "Fold 4/4, Epoch 16/19, Running Loss valid 15.326663970947266, Time 0.6782965660095215, Overall 3.6035866737365723 \n",
+ "Fold 4/4, Epoch 17/19, Minibatch 0/0, Phase train, Running Loss train 200.76573181152344, Time 0.7022991180419922, Overall 3.627589225769043 \n",
+ "Fold 4/4, Epoch 17/19, Running Loss train 15.44351783165565, Time 0.7022991180419922, Overall 3.627589225769043 \n",
+ "Fold 4/4, Epoch 17/19, Minibatch 0/0, Phase valid, Running Loss valid 47.091033935546875, Time 0.711296796798706, Overall 3.636586904525757 \n",
+ "Fold 4/4, Epoch 17/19, Running Loss valid 15.697011311848959, Time 0.711296796798706, Overall 3.636586904525757 \n",
+ "Fold 4/4, Epoch 18/19, Minibatch 0/0, Phase train, Running Loss train 109.23597717285156, Time 0.737295389175415, Overall 3.662585496902466 \n",
+ "Fold 4/4, Epoch 18/19, Running Loss train 8.402767474834736, Time 0.737295389175415, Overall 3.662585496902466 \n",
+ "Fold 4/4, Epoch 18/19, Minibatch 0/0, Phase valid, Running Loss valid 55.27782440185547, Time 0.7452952861785889, Overall 3.6705853939056396 \n",
+ "Fold 4/4, Epoch 18/19, Running Loss valid 18.425941467285156, Time 0.7452952861785889, Overall 3.6705853939056396 \n",
+ "Fold 4/4, Epoch 19/19, Minibatch 0/0, Phase train, Running Loss train 108.97453308105469, Time 0.7704596519470215, Overall 3.6957497596740723 \n",
+ "Fold 4/4, Epoch 19/19, Running Loss train 8.38265639085036, Time 0.7704596519470215, Overall 3.6957497596740723 \n",
+ "Fold 4/4, Epoch 19/19, Minibatch 0/0, Phase valid, Running Loss valid 31.412357330322266, Time 0.7796492576599121, Overall 3.704939365386963 \n",
+ "Fold 4/4, Epoch 19/19, Running Loss valid 10.470785776774088, Time 0.7796492576599121, Overall 3.704939365386963 \n",
+ "It took 3.7111148834228516 to train the model.\n",
+ "Figure(640x480)\n",
+ "Figure(640x480)\n",
+ "Figure(640x480)\n",
+ "Figure(640x480)\n",
+ "Figure(640x480)\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "c:\\Users\\tea-n_\\anaconda3\\envs\\Temporal\\lib\\site-packages\\torch\\optim\\lr_scheduler.py:28: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate.\n",
+ " warnings.warn(\"The verbose parameter is deprecated. Please use get_last_lr() \"\n"
+ ]
+ }
+ ],
+ "source": [
+ "%cd src\n",
+ "!python -u main.py -data ../data/raw/dblp/toy.dblp.v12.json -domain dblp -model bnn tbnn"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Additional Resources\n",
+ "- [`OpeNTF` codebase](https://github.com/fani-lab/OpeNTF/tree/main)\n",
+ "- [`Adila` codebase](https://github.com/fani-lab/adila)\n",
+ "- [`vivaFemme` codebase](https://github.com/fani-lab/OpeNTF/tree/vivaFemme)\n",
+ "- [Streaming Training Strategy codebase](https://github.com/fani-lab/OpeNTF/tree/ecir24)\n",
+ "- [Tutorial Website and Materials](https://fani-lab.github.io/OpeNTF/tutorial/umap24/)\n",
+ " - [`OpeNTF` paper](https://doi.org/10.1145/3511808.3557526)\n",
+ " - [`Adila` paper](https://doi.org/10.1007/978-3-031-37249-0_9)\n",
+ " - [`vivaFemme` paper](https://hosseinfani.github.io/res/papers/2024_BIAS_SIGIR_vivaFemme_Mitigating_Gender_Bias_in_Neural_Team_Recommendation_via_Female-Advocate_Loss_Regularization.pdf)\n",
+ " - [Streaming Training Strategy paper](https://link.springer.com/chapter/10.1007/978-3-031-56027-9_20)\n",
+ "\n",
+ ""
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Temporal",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.19"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/docs/tutorial/umap24/jupyter/new-class-diagram.png b/docs/tutorial/umap24/jupyter/new-class-diagram.png
new file mode 100644
index 0000000000000000000000000000000000000000..28eefca8bef5e462025536e1cef428f56c065f98
GIT binary patch
literal 63234
zcmdqIgxJljTz@eWSpB^2euOvAb?h@HN+KF;6HIRN`;S6+!!%S_nH-+bJkAIo
z&|+Dj{qSOh2}vk1U!9`T$Ee1_%?xMSVa{TxBcUg`c=0$6{YUV!Ocdk(i}M1DUk1O_
ze!2W2(_;A@lJ+I7BrV#Y(jY2Lulz;1iem(lua?7ZU_>%r@W(5_bW;6iS@I@Iv(iqC
zri@;3Qrb1FLZtLc;!mQVCTppc