2208181.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 跟异类生物聊个天儿~ 【全平台首个微信语音聊天机器人】\n",
    "\n",
    "\n",
    "> **您想知道异类生物能懂你吗？**\n",
    "> \n",
    "> **您想知道跟异类生物聊天是个怎样的体验吗？**\n",
    "> \n",
    "> **又想知道异类生物说话是什么样子吗？**\n",
    "\n",
    "\n",
    "**本项目随意选了下面三个角色，给您一个逼真的体验！！！**\n",
    "<br/> \n",
    "<br/> \n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/b7880404329d48cd9275552e436451518faecb1e7c3f445aab3d124a3c540c82\" width='800' height=''></center>\n",
    "\n",
    "<br/> \n",
    "<br/> \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "\n",
    "**B站视频体验如下：**\n",
    "\n",
    "<iframe style=\"width:98%;height: 450px;\" src=\"//player.bilibili.com/player.html?aid=847123470&bvid=BV1m54y177sN&cid=379534777&page=1\" scrolling=\"no\" border=\"0\" frameborder=\"no\" framespacing=\"0\" allowfullscreen=\"true\"> </iframe>\n",
    "\n",
    "**b站视频链接：[https://www.bilibili.com/video/BV1m54y177sN](https://www.bilibili.com/video/BV1m54y177sN/)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 一、项目背景\n",
    "\n",
    "本项目是参加【**AI创造营 ：Metaverse启动机之重构现世**】的作品，PaddlePaddle 联合开源聊天机器人框架 Wechaty 和设计师社区 MixLab 带来炫酷科幻创意赛，并由未来事务管理局作为媒体支持，以期为AI算法工程师拓展Conversational AI 应用场景，为 Chatbot 开发者提供AI技术支持平台，为MixLab设计师探索创新构想落地途径。\n",
    "\n",
    "本项目主要功能是，选择一个虚拟角色作为聊天对象，通过音色克隆高度模仿虚拟对象的语音特征，基于Wechaty在微信端通过语音信息交互(语音文件)，也可以通过选择视频连线(生成视频文件)，带来更好的体验。\n",
    "\n",
    "本项目主要挑战是打通语音识别(DeepSpeech)，中文对话(Plato-mini), 语音音色克隆(Parakeet)， 语音动图驱动(wav2lip)整个流程， 流程集成响应速度慢, 模型需要更精细化的调优。\n",
    "\n",
    "关于Wechaty的安装和使用请参考我另一篇： [微信医聊自动问答 WeChaty + PaddleHub ](https://aistudio.baidu.com/aistudio/projectdetail/1868162)\n",
    "\n",
    "<br/>\n",
    "\n",
    "**参考项目**：\n",
    "\n",
    "[夜雨飘零大神的DeepSpeech版本超赞](http://github.com/yeyupiaoling/PaddlePaddle-DeepSpeech)\n",
    "\n",
    "[Parakeet：飞桨，你是个成熟的框架了，要学会自己读论文](https://aistudio.baidu.com/aistudio/projectdetail/676162)\n",
    "\n",
    "[Parakeet音色克隆：柯南的变声器成真啦](http://https://aistudio.baidu.com/aistudio/projectdetail/2206367) \n",
    "\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/7752d1d036e540868cea4923ae87b9cb984cf543263a41ccb94b785872a53266\" width='400' height=''></center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 二、关于WeChaty和PaddleHub\n",
    "\n",
    "wechaty（https://github.com/wechaty/wechaty）是一款开源的微信SDK，它基于微信公开的API，对接口进行了一系列的封装，提供一系列简单的接口，然后开发者可以在其之上进行微信机器人的开发。\n",
    "    \n",
    "这里使用docker脚本可方便快速部署。请将your_token处替换成你的WeChaty token (**puppet_padlocal_xxxxxxxxxxxxx)**\n",
    "\n",
    "```\n",
    "export WECHATY_LOG=\"verbose\"\n",
    "export WECHATY_PUPPET=\"wechaty-puppet-padlocal\"\n",
    "export WECHATY_PUPPET_PADLOCAL_TOKEN=\"your_token\"\n",
    "\n",
    "export WECHATY_PUPPET_SERVER_PORT=\"8080\"\n",
    "export WECHATY_TOKEN=\"your_token\"\n",
    "\n",
    "docker run -ti \\\n",
    "  --name wechaty_puppet_service_token_gateway \\\n",
    "  --rm \\\n",
    "  -e WECHATY_LOG \\\n",
    "  -e WECHATY_PUPPET \\\n",
    "  -e WECHATY_PUPPET_PADLOCAL_TOKEN \\\n",
    "  -e WECHATY_PUPPET_SERVER_PORT \\\n",
    "  -e WECHATY_TOKEN \\\n",
    "  -p \"$WECHATY_PUPPET_SERVER_PORT:$WECHATY_PUPPET_SERVER_PORT\" \\\n",
    "  wechaty/wechaty:latest\n",
    "~                        \n",
    "```\n",
    "\n",
    "运行成功后如下图:\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/73f123bb9b37410eb49f834dc512226e014f3f53a77d42888b201833e8ef6942\" width='800' height=''></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 三、安装环境以及python的依赖包\n",
    "\n",
    "\n",
    "`!pip install -r PaddleGAN/requirements.txt imageio-ffmpeg`\n",
    "\n",
    "`!pip install -U paddlehub`\n",
    "\n",
    "`!pip install ruamel  -U paddle-parakeet`\n",
    "\n",
    "`!pip install shapely pyclipper -i https://pypi.tuna.tsinghua.edu.cn/simple `\n",
    "\n",
    "为了将微信端slk格式的语音文件转为mp3, 需要用到silk-v3-decoder第三方库, 使用的命令如下:\n",
    "\n",
    "`sh converter.sh 33921FF3774A773BB193B6FD4AD7C33E.slk mp3`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 安装Parakeet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'silk-v3-decoder'...\n",
      "remote: Enumerating objects: 697, done.\u001b[K\n",
      "remote: Counting objects: 100% (697/697), done.\u001b[K\n",
      "remote: Compressing objects: 100% (306/306), done.\u001b[K\n",
      "remote: Total 697 (delta 388), reused 697 (delta 388), pack-reused 0\u001b[K\n",
      "Receiving objects: 100% (697/697), 75.21 MiB | 523.00 KiB/s, done.\n",
      "Resolving deltas: 100% (388/388), done.\n",
      "Checking connectivity... done.\n"
     ]
    }
   ],
   "source": [
    "#安装silk-v3-decoder\r\n",
    "!git clone https://gitee.com/xiangjilexiaochou/silk-v3-decoder.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 四、DeepSpeech模型介绍\n",
    "\n",
    "#### 安装DeepSpeech\n",
    "\n",
    "大神的安装教程已经非常详细，[在此不再赘述](http://github.com/yeyupiaoling/PaddlePaddle-DeepSpeech) \n",
    "\n",
    "本人是通过docker方式安装，方便隔离本机电脑环境， 如CUDA。\n",
    "\n",
    "本人下载的模型是自收集(超过1300小时)，花了4.9为大神训练的电费分担一点点一点点。 [下载地址](http://download.csdn.net/download/qq_33200967/16200230)\n",
    "\n",
    "DeepSpeech2是基于PaddlePaddle实现的端到端自动语音识别（ASR）引擎，其论文为[《Baidu's Deep Speech 2 paper》](http://proceedings.mlr.press/v48/amodei16.pdf) ，本项目同时还支持各种数据增强方法，以适应不同的使用场景。\n",
    "\n",
    "这篇论文是在2015年由Baidu AI Lab所发布的，依旧延续了上一篇论文的路线：抛弃复杂的传统框架，拥抱基于神经网络的端到端模型。这篇论文有三个亮点：\n",
    "* 1. 作者这次所训练的模型既识别英文语音，也识别中文（普通话）语音。 \n",
    "* 2. 作者利用 HPC 技术（High-performance Computing，即高性能计算），使得整个系统性能有了大幅的提高（得利于此，模型训练速度大幅提升，也引出了第三点）。\n",
    "* 3. 作者在 Deep Speech 的基础上做了大量修改与尝试：加深了网络深度，尝试了 (Bi-directional) Vanilla RNN 和 GRU，引进了1D/2D invariant convolution，引入 Batch Nomalization。\n",
    "\n",
    "**网络结构如下图:**\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/eb7d3b72779443adb7f4c1f54b6f11d84951dc01ac424ba885cb4f97ddd730f3\" width='800' height=''></center>\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### DeepSpeech部署效果\n",
    "\n",
    "**本机基于http server方式部署，方便各个模块解耦， 效果看起来并不太完美， 推理时间也比较长， 但也很不错了：**\n",
    "\n",
    "ps: 注意暂不支持mp3格式音频文件\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/e7b0d58c057d4168a2f7fbbe585a95f07f299132061741758b69c690718d9ed0\" width='800' height=''></center>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 五、Plato-mini模型介绍\n",
    "\n",
    "近年来，人机对话系统受到了学术界和产业界的广泛关注并取得了不错的发展。开放域对话系统旨在建立一个开放域的多轮对话系统，使得机器可以流畅自然地与人进行语言交互，既可以进行日常问候类的闲聊，又可以完成特定功能，以使得开放域对话系统具有实际应用价值。\n",
    "\n",
    "Plato-mini基于6层UnifiedTransformer预训练结构，结合海量中文对话语料数据预训练的轻量级中文闲聊对话模型。\n",
    "\n",
    "参考论文: [PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning](https://arxiv.org/pdf/2006.16779.pdf)\n",
    "\n",
    "**网络结构如下:**\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/a0a3f6b4e74c410aac78228970bb3a1871e8c4be568f4eb29fd3cb3ffd91aa2e\" width='800' height=''></center>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#安装paddleNLP\n",
    "!pip install --upgrade paddlenlp -i https://pypi.org/simple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2021-07-21 11:10:04,251] [    INFO] - Found /home/aistudio/.paddlenlp/models/plato-mini/plato-mini-vocab.txt\n",
      "[2021-07-21 11:10:04,325] [    INFO] - Found /home/aistudio/.paddlenlp/models/plato-mini/plato-mini-spm.model\n",
      "[2021-07-21 11:10:04,398] [    INFO] - Downloading https://paddlenlp.bj.bcebos.com/models/transformers/unified_transformer/plato-mini.pdparams and saved to /home/aistudio/.paddlenlp/models/plato-mini\n",
      "[2021-07-21 11:10:04,401] [    INFO] - Downloading plato-mini.pdparams from https://paddlenlp.bj.bcebos.com/models/transformers/unified_transformer/plato-mini.pdparams\n",
      "100%|██████████| 530157/530157 [00:07<00:00, 69805.22it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['我今年23岁,你多大了?']\n"
     ]
    }
   ],
   "source": [
    "# 加载模型\n",
    "from paddlenlp.transformers import UnifiedTransformerTokenizer\n",
    "from paddlenlp.transformers import UnifiedTransformerLMHeadModel\n",
    "from utils import select_response\n",
    "\n",
    "model_name = 'plato-mini'\n",
    "tokenizer = UnifiedTransformerTokenizer.from_pretrained(model_name)\n",
    "model = UnifiedTransformerLMHeadModel.from_pretrained(model_name)\n",
    "\n",
    "# 一键预测\n",
    "user_input = ['你好啊，你今年多大了']\n",
    "encoded_input = tokenizer.dialogue_encode(\n",
    "                    user_input,\n",
    "                    add_start_token_as_response=True,\n",
    "                    return_tensors=True,\n",
    "                    is_split_into_words=False)\n",
    "\n",
    "ids, scores = model.generate(\n",
    "                input_ids=encoded_input['input_ids'],\n",
    "                token_type_ids=encoded_input['token_type_ids'],\n",
    "                position_ids=encoded_input['position_ids'],\n",
    "                attention_mask=encoded_input['attention_mask'],\n",
    "                max_length=64,\n",
    "                min_length=1,\n",
    "                decode_strategy='sampling',\n",
    "                top_k=5,\n",
    "                num_return_sequences=20)\n",
    "\n",
    "result = select_response(ids, scores, tokenizer, keep_space=False, num_return_sequences=20)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 六、Parakeet 音色克隆模型介绍\n",
    "\n",
    "#### 实现原理\n",
    "\n",
    "在训练语音克隆模型时，目标音色作为**Speaker Encoder**的输入，模型会提取这段语音的说话人特征（音色）作为**Speaker Embedding**。\n",
    "\n",
    "接着，在训练模型重新合成此类音色的语音时，除了输入的目标文本外，说话人的特征也将成为额外条件加入模型的训练。\n",
    "\n",
    "在预测时，选取一段新的目标音色作为**Speaker Encoder**的输入，并提取其说话人特征，最终实现**输入为一段文本和一段目标音色，模型生成目标音色说出此段文本的语音片段，完美的实现音色克隆~**\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/982ab955b87244d3bae3b003aff8e28d9ec159ff0d6246a79757339076dfe7d4)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 安装 Parakeet\n",
    "# !git clone https://gitee.com/paddlepaddle/Parakeet.git -b release/v0.3 work/Parakeet\n",
    "!pip install -e work/Parakeet/  > install_parakeet.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 下载并解压预训练模型\n",
    "\n",
    "# %mkdir  /home/aistudio/work/pretrained\n",
    "# %cd /home/aistudio/work/pretrained\n",
    "\n",
    "# !wget https://paddlespeech.bj.bcebos.com/Parakeet/waveflow_ljspeech_ckpt_0.3.zip\n",
    "# !wget https://paddlespeech.bj.bcebos.com/Parakeet/tacotron2_aishell3_ckpt_0.3.zip\n",
    "# !wget https://paddlespeech.bj.bcebos.com/Parakeet/ge2e_ckpt_0.3.zip\n",
    "# !unzip waveflow_ljspeech_ckpt_0.3.zip\n",
    "# !unzip tacotron2_aishell3_ckpt_0.3.zip\n",
    "# !unzip ge2e_ckpt_0.3.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "embed shape:  [256]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/1000 [00:00<?, ?it/s] 34%|███▍      | 345/1000 [00:01<00:02, 306.38it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "content exhausted!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 34%|███▍      | 345/1000 [00:01<00:02, 286.34it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.6733529567718506s\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append(\"/home/aistudio/work/Parakeet\")\n",
    "sys.path.append(\"/home/aistudio/work/Parakeet/examples/tacotron2_aishell3\")\n",
    "\n",
    "import numpy as np\n",
    "import paddle\n",
    "from matplotlib import pyplot as plt\n",
    "from IPython import display as ipd\n",
    "import soundfile as sf\n",
    "import librosa.display\n",
    "from parakeet.utils import display\n",
    "from examples.ge2e.audio_processor import SpeakerVerificationPreprocessor\n",
    "from parakeet.models.lstm_speaker_encoder import LSTMSpeakerEncoder\n",
    "\n",
    "paddle.set_device(\"gpu:0\")\n",
    "\n",
    "# speaker encoder\n",
    "p = SpeakerVerificationPreprocessor(\n",
    "    sampling_rate=16000, \n",
    "    audio_norm_target_dBFS=-30, \n",
    "    vad_window_length=30, \n",
    "    vad_moving_average_width=8, \n",
    "    vad_max_silence_length=6, \n",
    "    mel_window_length=25, \n",
    "    mel_window_step=10, \n",
    "    n_mels=40, \n",
    "    partial_n_frames=160, \n",
    "    min_pad_coverage=0.75, \n",
    "    partial_overlap_ratio=0.5)\n",
    "speaker_encoder = LSTMSpeakerEncoder(n_mels=40, num_layers=3, hidden_size=256, output_size=256)\n",
    "speaker_encoder_params_path = \"/home/aistudio/work/pretrained/ge2e_ckpt_0.3/step-3000000.pdparams\"\n",
    "speaker_encoder.set_state_dict(paddle.load(speaker_encoder_params_path))\n",
    "speaker_encoder.eval()\n",
    "\n",
    "# synthesizer\n",
    "from parakeet.models.tacotron2 import Tacotron2\n",
    "from examples.tacotron2_aishell3.chinese_g2p import convert_sentence\n",
    "from examples.tacotron2_aishell3.aishell3 import voc_phones, voc_tones\n",
    "\n",
    "from yacs.config import CfgNode\n",
    "synthesizer = Tacotron2(\n",
    "    vocab_size=68,\n",
    "    n_tones=10,\n",
    "    d_mels= 80,\n",
    "    d_encoder= 512,\n",
    "    encoder_conv_layers = 3,\n",
    "    encoder_kernel_size= 5,\n",
    "    d_prenet= 256,\n",
    "    d_attention_rnn= 1024,\n",
    "    d_decoder_rnn = 1024,\n",
    "    attention_filters = 32,\n",
    "    attention_kernel_size = 31,\n",
    "    d_attention= 128,\n",
    "    d_postnet = 512,\n",
    "    postnet_kernel_size = 5,\n",
    "    postnet_conv_layers = 5,\n",
    "    reduction_factor = 1,\n",
    "    p_encoder_dropout = 0.5,\n",
    "    p_prenet_dropout= 0.5,\n",
    "    p_attention_dropout= 0.1,\n",
    "    p_decoder_dropout= 0.1,\n",
    "    p_postnet_dropout= 0.5,\n",
    "    d_global_condition=256,\n",
    "    use_stop_token=False\n",
    ")\n",
    "params_path = \"/home/aistudio/work/pretrained/tacotron2_aishell3_ckpt_0.3/step-450000.pdparams\"\n",
    "synthesizer.set_state_dict(paddle.load(params_path))\n",
    "synthesizer.eval()\n",
    "\n",
    "# vocoder\n",
    "from parakeet.models import ConditionalWaveFlow\n",
    "vocoder = ConditionalWaveFlow(upsample_factors=[16, 16], n_flows=8, n_layers=8, n_group=16, channels=128, n_mels=80, kernel_size=[3, 3])\n",
    "params_path = \"/home/aistudio/work/pretrained/waveflow_ljspeech_ckpt_0.3/step-2000000.pdparams\"\n",
    "vocoder.set_state_dict(paddle.load(params_path))\n",
    "vocoder.eval()\n",
    "\n",
    "# 提取参考音色特征\n",
    "mel_sequences = p.extract_mel_partials(p.preprocess_wav(\"/home/aistudio/ref.wav\"))\n",
    "with paddle.no_grad():\n",
    "    embed = speaker_encoder.embed_utterance(paddle.to_tensor(mel_sequences))\n",
    "print(\"embed shape: \", embed.shape)\n",
    "\n",
    "#合成频谱\n",
    "sentence = \"语音的表现形式%在未来%将变得越来越重要$\"\n",
    "phones, tones = convert_sentence(sentence)\n",
    "phones = np.array([voc_phones.lookup(item) for item in phones], dtype=np.int64)\n",
    "tones = np.array([voc_tones.lookup(item) for item in tones], dtype=np.int64)\n",
    "phones = paddle.to_tensor(phones).unsqueeze(0)\n",
    "tones = paddle.to_tensor(tones).unsqueeze(0)\n",
    "utterance_embeds = paddle.unsqueeze(embed, 0)\n",
    "with paddle.no_grad():\n",
    "    outputs = synthesizer.infer(phones, tones=tones, global_condition=utterance_embeds)\n",
    "mel_input = paddle.transpose(outputs[\"mel_outputs_postnet\"], [0, 2, 1])\n",
    "\n",
    "with paddle.no_grad():\n",
    "    wav = vocoder.infer(mel_input)\n",
    "wav = wav.numpy()[0]\n",
    "sf.write(f\"output.wav\", wav, samplerate=22050)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 七、Wav2Lip模型介绍\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/adb83d8b854246e0b713df86fa6fb6d4a714a01f01bb42d8a5ef50744ca7c13a\" width='800' height=''></center>\n",
    "<center><br>Wav2Lip唇形同步示意图</br></center>\n",
    "\n",
    "* 在训练阶段，生成器模型输入包含两部分（视频帧序列和音频），分别通过Face encoder和Audio encoder得到特征信息，并进行融合，再通过Face decoder获得唇形和音频同步的图像帧。把原始视频帧和生成图像帧输入到视觉质量判别器中，二分类结果表示是真实的图像、还是生成的图片，进而提高图像质量。把生成图像帧和音频输入到预先训练好的唇形同步判别器中，判断唇形是否生成的精准，在训练过程中，唇形同步判别器参数会一直被冻结，不参与训练、更新。\n",
    "\n",
    "* 在推理阶段，提供一段音频和视频(或图像、动画)即可合成唇形同步视频。\n",
    "\n",
    "<center><img src='https://ai-studio-static-online.cdn.bcebos.com/7e21a14b990745b7a5f4ce6eedbd70b99204d4e0010647f782f411ce7697b768' width='800' height=''></center>\n",
    "\n",
    "<center><br>Wav2Lip唇形同步设计方案</br></center>\n",
    "\n",
    "av2Lip实现唇形与语音精准同步突破的关键在于，它采用了唇形同步判别器，以强制生成器持续产生准确而逼真的唇部运动。此外，它通过在鉴别器中使用多个连续帧而不是单个帧，并使用视觉质量损失（而不仅仅是对比损失）来考虑时间相关性，从而改善了视觉质量。Wav2Lip适用于任何人脸、任何语言，对任意视频都能达到很高都准确率，可以无缝地与原始视频融合，还可以用于转换动画人脸。\n",
    "\n",
    "使用一个强有力的嘴唇同步鉴别器迫使发生器产生精确的嘴唇形状。然而，它有时会导致变形区域稍微模糊或包含轻微的伪像。为了减轻这种微小的质量损失，我们在 GAN 设置中训练了一个简单的视觉质量鉴别器和生成器。因此，我们有两个鉴别器，一个用于同步精度，另一个用于更好的视觉质量。一方面，唇同步鉴别器不需要进一步微调，即训练过程权重被冻结。另一方面，由于视觉质量鉴别器不对嘴唇同步进行任何检查，并且只惩罚不现实的人脸生成，所以它是在生成的人脸上进行训练的。\n",
    "\n",
    "Wav2Lip模型结构如下图所示：\n",
    "\n",
    "<center><img src='https://ai-studio-static-online.cdn.bcebos.com/3fbd93a0d6c04b66bfd1ac2f958635e80d0c7cc883e54a9c888e867d564407af' width='800' height=''></center>\n",
    "<center><br>图6：Wav2Lip结构图</br><center>  \n",
    "\n",
    "图6中下半部分被遮挡的目标人脸作为姿态先验输入，这是至关重要的，因为它允许生成的人脸区域无缝地粘回原始视频，不需要进一步的后期处理。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 八、基于wechaty部署全流程\n",
    "\n",
    "1. wav2lip代码无法直接读取slk格式文件，所以需要先借用工具转换一下。\n",
    "\n",
    "wechaty没有提供发送语音的接口，如果将slk或者mp3发回去，只能是以文件的形式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%cd silk-v3-decoder\r\n",
    "!sh converter.sh ../message-7116130031518557320-audio.slk mp3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#格式转换 mp3转wav\n",
    "# !pip install pydub\n",
    "\n",
    "from pydub import AudioSegment\n",
    "song = AudioSegment.from_mp3(\"/home/aistudio/test.mp3\")\n",
    "song.export(\"now.wav\", format=\"wav\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "2. 搜集科幻人物视频， 从中选择三个外星人角色的图片和语音， 本人从好看视频和bilibili上随意挑选了3段视频，使用ffmpeg对于目标片段进行裁剪拷贝音频，并使用脚本预提取声纹特征并保存在embed_3.npy中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# !ffmpeg -i work/tmp/3.mp4 -f wav -ss 00:01:11  -t  00:00:15  3.wav\n",
    "!python create_embed.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "3. 集成上面各个模块到wechaty脚本\n",
    "      * 接收mp3文件\n",
    "      * 语音识别\n",
    "      * 多轮对话\n",
    "      * 语音音色合成\n",
    "      * wav2lip视频生成\n",
    "      * 发送语音文件或视频动图\n",
    "4. wechaty在本机部署\n",
    "5. 在微信端实测\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 九、部署在wechaty上\n",
    "\n",
    "**实测效果还可以，只是集成模型太多，导致速度变得很慢，后期需要进一步优化。**\n",
    "<br/>\n",
    "<br/>\n",
    "<br/>\n",
    "\n",
    "\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/05d495cc5169410883e9bc1f6ed034110132cf6b8d7a4aec8a8ffdb9e88c8b0e)\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/c3e2ba31f5f04d05b3c56542a4717bd8f1619609758541f491e2ac2248f422f5)\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/93fe487eeae24845bc8e79d55a912fc25517c5c149e943e7b783f16b22cbdab8)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 十、总结\n",
    "\n",
    "尝试了Paddle的DeepSpeech， Parakeet， Plato-mini效果都还不错，学习了这几种模型的使用，能够很轻松的搭建一款语音聊天机器人，同时加上一点点的科幻元素，让chatbot变得更有趣味。\n",
    "\n",
    "项目中还有很多缺陷，要达到落地很难很难，响应速度非常慢，要达到实时聊天还有很长的距离。对话模型较小，聊天内容有些单调和重复。语音识别和音色克隆也有很大的提升空间。\n",
    "\n",
    "\n",
    "如果喜欢，请帮我star, fork, 一键三连！！！\n",
    "\n",
    "**关于作者**\n",
    "\n",
    "PaddlePaddle开发爱好者\n",
    "\n",
    "我在AI Studio上获得钻石等级，点亮10个徽章，来互关呀~ https://aistudio.baidu.com/aistudio/personalcenter/thirdview/89442"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 2.0.0b0 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}