RecursionError: maximum recursion depth exceeded while calling a Python object

[lgqfhwy]

when train on 18epoch, it would report error:

Traceback (most recent call last):
  File "train.py", line 396, in <module>
    train_model(args)
  File "train.py", line 213, in train_model
    lossTr, lr = train(args, trainLoader, model, criteria, optimizer, epoch)
  File "train.py", line 328, in train
    optimizer.step()
  File "/home/lgq/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 36, in wrapper
    return func(*args, **kwargs)
  File "/home/lgq/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 36, in wrapper
    return func(*args, **kwargs)
  File "/home/lgq/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 36, in wrapper
    return func(*args, **kwargs)
  [Previous line repeated 991 more times]
  File "/home/lgq/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/optim/adam.py", line 69, in step
    state = self.state[p]
  File "/home/lgq/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/tensor.py", line 393, in __hash__
    return id(self)
RecursionError: maximum recursion depth exceeded while calling a Python object

I found that you put WarmupPolyLR in every iters, it would make every iter call WarmupPolyLR, maybe it cause the error.

[xiaoyufenfei]

It's your personal question, you should specify the --warmup_iters, for me, there is no any problem

[lgqfhwy]

In my opinion, you could not put WarmupPolyLR in every iter.

1. There is no code on github like this, include pytorch document not do it. And it could bring burden to system. It would report errors due to the difference of pytorch version.
2. There is no need to do this. You put WarmupPolyLR in every iter because you want to use variable
   cur_iter. But cur_iter does not need to pass on. When you define WarmupPolyLR, you inherit torch.optim.lr_scheduler._LRScheduler, there is a variable self._step_count in _LRScheduler, it has a same purpose to cur_iter.So you could just write like:

    def get_lr(self):
        self.cur_iter = self._step_count - 2
        if self.cur_iter <= self.warmup_iters:
            alpha = self.cur_iter / self.warmup_iters
            warmup_factor = self.warmup_factor * (1 - alpha) + alpha
            # print(self.base_lrs[0]*warmup_factor)
            return [lr * warmup_factor for lr in self.base_lrs]
        else:
            return [self.eta_min + (base_lr - self.eta_min) *
                    math.pow(1 - (self.cur_iter - self.warmup_iters) / (self.T_max - self.warmup_iters),
                             self.power) for base_lr in self.base_lrs]

or you could just use (self._step_count - 2) to replace self.cur_iter. After do this, you only need to call WarmupPolyLR after train once, it could have same effect to your origin code, but it reduce many times call function WarmupPolyLR.
What's more, thank you for your code and happy new year :)

[xiaoyufenfei]

Happy New Year ! Thank you for you suggestion.

[dronefreak]

Hi @lgqfhwy
Were you able to resolve this issue because I am facing it as of now.
Please let me know your workaround.
I did try increasing the max recursion depth by using the sys library in python, but it still gets stuck at th 9th epoch for me.

[xiaoyufenfei]

Maybe this way can help you.

Efficient-Segmentation-Networks/train.py

Line 25 in 0f0c32e

sys.setrecursionlimit(1000000) # solve problem 'maximum recursion depth exceeded'