Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

关于训练的问题 #4

Open
BrianWayland opened this issue Nov 30, 2022 · 0 comments
Open

关于训练的问题 #4

BrianWayland opened this issue Nov 30, 2022 · 0 comments

Comments

@BrianWayland
Copy link

BrianWayland commented Nov 30, 2022

您好,我训练时使用chinese_cleaners2来处理中文文本,然后在训练时输出了以下的异常信息:

Traceback (most recent call last):
  File "/home/featurize/data/vits-main/train_ms.py", line 119, in run
    train_and_evaluate(rank, epoch, hps, [net_g, net_d], [optim_g, optim_d], [scheduler_g, scheduler_d], scaler, [train_loader, eval_loader], logger, [writer, writer_eval])
  File "/home/featurize/data/vits-main/train_ms.py", line 147, in train_and_evaluate
    (z, z_p, m_p, logs_p, m_q, logs_q) = net_g(x, x_lengths, spec, spec_lengths, speakers)
  File "/home/featurize/data/vits-main/models.py", line 467, in forward
    z, m_q, logs_q, y_mask = self.enc_q(y, y_lengths, g=g)
  File "/home/featurize/data/vits-main/models.py", line 237, in forward
    x = self.enc(x, x_mask, g=g)
  File "/environment/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/featurize/data/vits-main/modules.py", line 166, in forward
    n_channels_tensor)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "/home/featurize/data/vits-main/commons.py", line 103, in fused_add_tanh_sigmoid_multiply
def fused_add_tanh_sigmoid_multiply(input_a, input_b, n_channels):
  n_channels_int = n_channels[0]
  in_act = input_a + input_b
           ~~~~~~~~~~~~~~~~~ <--- HERE

在打印了input_a和input_b的维度后,发现input_a的形状是[64, 384, 500],input_b的形状是[64, 384, 1]这两种形状的张量是可以通过广播进行相加的,不存在不能够相加的问题。此外,有时候输出异常信息时会提示后验编码器的卷积层计算错误。
请问您遇到了这种情况吗,又是如何解决的呢。我认为这种问题应该跟多卡训练有关,但问题是我使用的只有一张卡,也就是创建的只有一个训练进程,按理来说不会发生数据冲突之类的问题。期待您的回复。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant