decoding issue

[synetkim]

Hello,
I have an issue when I tried to decode the model for TLE.

export THEANO_FLAGS=mode=FAST_RUN,device=gpu3,floatX=float32; $LVSR/exp/wsj/decode_tle.sh wsj_reward6 test 200

Here is the log:

• MODEL=wsj_reward6
• PART=test
• BEAM_SIZE=200
• LM=nolm
• LM_PATH=data/local/nist_lm/wsj_trigram_no_bos
• ls wsj_reward6/reports
  test_eval92_nolm_200
  test_nolm_200
• LM_CONF='monitoring.search.beam_size 200'
• '[' '!' nolm == nolm ']'
• /speech05-ssd/sykim/attention-lvcsr/bin/run.py search --part=test --report wsj_reward6/reports/test_nolm_200 wsj_reward6/annealing1_best_ll.zip /speech05-ssd/sykim/attention-lvcsr/exp/wsj/configs/wsj_reward6.yaml vocabulary data/local/nist_lm/wsj_trigram_no_bos/words.txt monitoring.search.beam_size 200
  Using gpu device 3: Tesla K20m (CNMeM is enabled with initial size: 90.0% of memory, cuDNN Version is too old. Update to v5, was 3007.)
  2016-06-07 14:49:00,862: pykwalify.core: INFO: validation.valid
  2016-06-07 14:49:00,867: pykwalify.core: INFO: validation.valid
  2016-06-07 14:49:00,872: pykwalify.core: INFO: validation.valid
  2016-06-07 14:49:00,877: pykwalify.core: INFO: validation.valid
  2016-06-07 14:49:00,884: main: INFO: Config:
  {'cmd_args': {'config_changes': <picklable_itertools.extras.equizip object at 0x7fb144a68050>,
  'config_path': '/speech05-ssd/sykim/attention-lvcsr/exp/wsj/configs/wsj_reward6.yaml',
  'decode_only': None,
  'decoded_save': None,
  'func': 'search',
  'load_path': 'wsj_reward6/annealing1_best_ll.zip',
  'nll_only': False,
  'params': None,
  'part': 'test',
  'report': 'wsj_reward6/reports/test_nolm_200',
  'seed': 1,
  'validate_config': True},
  'data': {'add_bos': 1,
  'add_eos': True,
  'batch_size': 20,
  'dataset_class': <class 'lvsr.datasets.h5py.H5PYAudioDataset'>,
  'dataset_filename': 'wsj.h5',
  'default_sources': ['recordings', 'labels'],
  'name_mapping': {'test': 'test_eval92', 'train': 'train_si284', 'valid': 'test_dev93'},
  'normalization': '',
  'sources_map': {'labels': 'characters', 'recordings': 'fbank_dd', 'uttids': 'uttids'}},
  'initialization': {'/recognizer': {'biases_init': Constant(_constant=0.0),
  'rec_weights_init': IsotropicGaussian(_mean=0,_std=0.1),
  'weights_init': Uniform(_mean=0.0,_width=0.1)},
  '/recognizer/generator/readout/post_merge/mlp': {'biases_init': Constant(_constant=-1.0)}},
  'monitoring': {'search': {'beam_size': 200, 'char_discount': 0.0, 'round_to_inf': 4.5, 'stop_on': 'patience'},
  'search_every_batches': 0,
  'search_every_epochs': 1,
  'validate_every_batches': 0,
  'validate_every_epochs': 1},
  'net': {'attention_type': 'content_and_conv',
  'bottom': {'activation': <blocks.bricks.simple.Rectifier object at 0x7fb118424510: name=rectifier>,
  'bottom_class': <class 'lvsr.bricks.recognizer.SpeechBottom'>,
  'dims': []},
  'conv_n': 100,
  'criterion': {'min_reward': -5, 'name': 'mse_gain'},
  'dec_transition': <class 'blocks.bricks.recurrent.GatedRecurrent'>,
  'dim_dec': 250,
  'dims_bidir': [250, 250, 250, 250],
  'enc_transition': <class 'blocks.bricks.recurrent.GatedRecurrent'>,
  'lm': {'normalize_am_weights': False},
  'max_decoded_length_scale': 3.0,
  'post_merge_activation': <blocks.bricks.simple.Rectifier object at 0x7fb1184336d0: name=rectifier>,
  'post_merge_dims': [250],
  'prior': {'after': 100,
  'before': 100,
  'initial_begin': 0,
  'initial_end': 80,
  'max_speed': 4.4,
  'min_speed': 2.4,
  'type': 'window_around_median'},
  'subsample': [1, 1, 2, 2],
  'use_states_for_readout': True},
  'parent': '$LVSR/exp/wsj/configs/wsj_reward4.yaml',
  'regularization': {'dropout': False, 'max_norm': 1.0},
  'stages': {'annealing1': {'net': {'lm': None},
  'training': {'epsilon': 1e-10, 'num_epochs': 3, 'restart_from': '_best_ll'}},
  'annealing2': None,
  'main': {'net': {'lm': None}, 'training': {'num_epochs': 15, 'restart_from': '_best_ll'}},
  'pretraining': {'net': {'criterion': {'min_reward': -1},
  'lm': None,
  'prior': {'initial_begin': 0,
  'initial_end': 40,
  'max_speed': 2.2,
  'min_speed': 1.2,
  'type': 'expanding'}},
  'training': {'num_epochs': 4}}},
  'training': {'decay_rate': 0.95,
  'epsilon': 1e-08,
  'exploration': 'greedy',
  'gradient_threshold': 100.0,
  'momentum': 0.0,
  'rules': ['momentum', 'adadelta'],
  'scale': 0.1},
  'vocabulary': 'data/local/nist_lm/wsj_trigram_no_bos/words.txt'}
  2016-06-07 14:49:00,996: lvsr.main: INFO: Recognizer initialization started
  2016-06-07 14:49:54,017: lvsr.main: INFO: Recognizer is initialized
  /speech05-ssd/sykim/attention-lvcsr/lvsr/bricks/recognizer.py:493: UserWarning: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 1 is not part of the computational graph needed to compute the outputs: groundtruth.
  To make this warning into an error, you can pass the parameter on_unused_input='raise' to theano.function. To disable it completely, use on_unused_input='ignore'.
  on_unused_input='warn')
  Traceback (most recent call last):
  File "/speech05-ssd/sykim/attention-lvcsr/bin/run.py", line 154, in
  getattr(lvsr.main, args.pop('func'))(config, **args)
  File "/speech05-ssd/sykim/attention-lvcsr/lvsr/main.py", line 781, in search
  prediction=raw_groundtruth)[:2]
  File "/speech05-ssd/sykim/attention-lvcsr/lvsr/bricks/recognizer.py", line 493, in analyze
  on_unused_input='warn')
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 322, in function
  output_keys=output_keys)
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 480, in pfunc
  output_keys=output_keys)
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1827, in orig_function
  output_keys=output_keys).create(
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1479, in init
  accept_inplace)
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 177, in std_fgraph
  update_mapping=update_mapping)
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 182, in init
  self.import_r(output, reason="init")
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 374, in import_r
  self.import(variable.owner, reason=reason)
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 416, in import
  variable=r)
  theano.gof.fg.MissingInputError: An input of the graph, used to compute DimShuffle{0,x}(labels), was not provided and not given a value.Use the Theano flag exception_verbosity='high',for more information on this error.

Backtrace when the variable is created:
File "/speech05-ssd/sykim/attention-lvcsr/bin/run.py", line 154, in
getattr(lvsr.main, args.pop('func'))(config, *_args)
File "/speech05-ssd/sykim/attention-lvcsr/lvsr/main.py", line 716, in search
recognizer = create_model(config, data, load_path)
File "/speech05-ssd/sykim/attention-lvcsr/lvsr/main.py", line 221, in create_model
*_net_config)
File "/speech05-ssd/sykim/attention-lvcsr/lvsr/bricks/recognizer.py", line 360, in init
self.single_labels = tensor.lvector('labels')

attention-lvcsr/lvsr/main.py", line 716, in search
recognizer = create_model(config, data, load_path)
File "/speech05-ssd/sykim/attention-lvcsr/lvsr/main.py", line 221, in create_model
**net_config)
File "/speech05-ssd/sykim/attention-lvcsr/lvsr/bricks/recognizer.py", line 360, in init
self.single_labels = tensor.lvector('labels')

[dmitriy-serdyuk]

Hmm, that's strange.

Can you run a postmortem debugger and check if input_variables contains the labels (this one)?

[synetkim]

Thanks for replying. Could you explain more detail how to use it?

[dmitriy-serdyuk]

Somehow, a function analyze is not compiled in your case. It just outputs the cost and visualizes the alignment, it is not needed for decoding, but it's very suspicious...

So, I propose to run a debugger like (assuming that you have ipdb; I'm taking the command from your log)

python -mipdb /speech05-ssd/sykim/attention-lvcsr/bin/run.py search --part=test --report wsj_reward6/reports/test_nolm_200 wsj_reward6/annealing1_best_ll.zip /speech05-ssd/sykim/attention-lvcsr/exp/wsj/configs/wsj_reward6.yaml vocabulary data/local/nist_lm/wsj_trigram_no_bos/words.txt monitoring.search.beam_size 200

You can use ... -mpdb ... if you don't want to install ipdb. If you are not familiar with pdb/ipdb, there are a lot of tutorials in internet how to use it.

You'll be able to go up/down the stack and print/inspect the variables.