Segmentation fault when using PyText 0.3.3 on Google Colab #1415

orestxherija · 2020-08-03T19:52:12Z

Steps to reproduce

Open a new Google Colab notebook and set the runtime type to GPU
Install latest version of PyText via !pip install --quiet git+https://github.com/facebookresearch/pytext
Import PyText with import pytext
Clone PyText repository via !git clone https://github.com/facebookresearch/pytext.git
Move pytext/tests/ directory to Google Colab's home directory /content/
Train demo CNN model via !pytext train < pytext/demo/configs/docnn.json

Observed Results

Below I am including the entire log.

Install apex from https://github.com/NVIDIA/apex/.
No config file specified, reading from stdin
Current Version: 8
WARNING - Applying old config adapter for version=8. Please consider migrating your old configs to the latest version.
Current Version: 9
WARNING - Applying old config adapter for version=9. Please consider migrating your old configs to the latest version.
Current Version: 10
WARNING - Applying old config adapter for version=10. Please consider migrating your old configs to the latest version.
Current Version: 11
WARNING - Applying old config adapter for version=11. Please consider migrating your old configs to the latest version.
Current Version: 12
WARNING - Applying old config adapter for version=12. Please consider migrating your old configs to the latest version.
Current Version: 13
WARNING - Applying old config adapter for version=13. Please consider migrating your old configs to the latest version.
Current Version: 14
WARNING - Applying old config adapter for version=14. Please consider migrating your old configs to the latest version.
Current Version: 15
WARNING - Applying old config adapter for version=15. Please consider migrating your old configs to the latest version.
Current Version: 16
WARNING - Applying old config adapter for version=16. Please consider migrating your old configs to the latest version.
Current Version: 17
WARNING - Applying old config adapter for version=17. Please consider migrating your old configs to the latest version.
Current Version: 18
WARNING - Applying old config adapter for version=18. Please consider migrating your old configs to the latest version.
Current Version: 19
WARNING - Applying old config adapter for version=19. Please consider migrating your old configs to the latest version.

===Starting training...
2020-08-03 19:39:44.136867: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1

Parameters: PyTextConfig:
    auto_resume_from_snapshot: False
    debug_path: /tmp/model.debug
    distributed_world_size: 1
    export_caffe2_path: /tmp/model.caffe2.predictor
    export_onnx_path: /tmp/model.onnx
    export_torchscript_path: /tmp/new_docnn.pt1
    gpu_streams_for_distributed_training: 1
    include_dirs: None
    load_snapshot_path: 
    modules_save_dir: 
    random_seed: None
    report_eval_results: False
    report_test_results: True
    save_all_checkpoints: False
    save_module_checkpoints: False
    save_snapshot_path: /tmp/model.pt
    task: DocumentClassificationTask.Config:
        data: Data.Config:
            batcher: PoolingBatcher.Config:
                eval_batch_size: 16
                num_shuffled_pools: 1
                pool_num_batches: 1000
                test_batch_size: 16
                train_batch_size: 16
            in_memory: True
            sort_key: None
            source: TSVDataSource.Config:
                column_mapping: {}
                delimiter: 	
                drop_incomplete_rows: False
                eval_filename: tests/data/test_data_tiny.tsv
                field_names: ['label', 'slots', 'text']
                quoted: False
                test_filename: tests/data/test_data_tiny.tsv
                train_filename: tests/data/train_data_tiny.tsv
        metric_reporter: ClassificationMetricReporter.Config:
            additional_column_names: []
            is_memory_efficient: False
            log_gradient: False
            model_select_metric: ComparableClassificationMetric.ACCURACY
            output_path: /tmp/test_out.txt
            pep_format: False
            recall_at_precision_thresholds: [0.2, 0.4, 0.6, 0.8, 0.9]
            student_column_names: []
            target_label: None
            text_column_names: ['text']
        model: DocModel.Config:
            decoder: MLPDecoder.Config:
                activation: Activation.RELU
                bias: True
                dropout: 0.0
                freeze: False
                hidden_dims: []
                layer_norm: False
                load_path: None
                out_dim: None
                save_path: None
                shared_module_key: None
            embedding: WordEmbedding.Config:
                cpu_only: False
                delimiter:  
                embed_dim: 100
                embeddding_init_std: 0.02
                embedding_init_range: None
                embedding_init_strategy: EmbedInitStrategy.RANDOM
                export_input_names: ['tokens_vals']
                freeze: False
                load_path: None
                lowercase_tokens: True
                min_freq: 1
                mlp_layer_dims: []
                padding_idx: None
                pretrained_embeddings_path: 
                save_path: None
                shared_module_key: None
                skip_header: True
                vocab_file: 
                vocab_from_all_data: False
                vocab_from_pretrained_embeddings: False
                vocab_from_train_data: True
                vocab_size: 0
            inputs: ModelInput:
                dense: None
                labels: LabelTensorizer.Config:
                    allow_unknown: False
                    column: label
                    is_input: False
                    label_vocab: None
                    label_vocab_file: None
                    pad_in_vocab: False
                tokens: TokenTensorizer.Config:
                    add_bos_token: False
                    add_eos_token: False
                    column: text
                    is_input: True
                    max_seq_len: None
                    tokenizer: Tokenizer.Config:
                        lowercase: True
                        split_regex: \s+
                    use_eos_token_for_bos: False
                    vocab: VocabConfig:
                        build_from_data: True
                        min_counts: 0
                        size_from_data: 0
                        vocab_files: []
                    vocab_file_delimiter:  
            output_layer: ClassificationOutputLayer.Config:
                freeze: False
                label_weights: None
                load_path: None
                loss: CrossEntropyLoss.Config:
                save_path: None
                shared_module_key: None
            representation: DocNNRepresentation.Config:
                cnn: CNNParams:
                    causal: False
                    dilated: False
                    kernel_num: 100
                    kernel_sizes: [3, 4]
                    weight_norm: False
                dropout: 0.4
                freeze: False
                load_path: None
                pooling: PoolingType.MAX
                save_path: None
                shared_module_key: None
        trainer: TaskTrainer.Config:
            do_eval: True
            early_stop_after: 0
            epochs: 10
            fp16_args: FP16OptimizerFairseq.Config:
                init_loss_scale: 128
                min_loss_scale: 0.0001
                scale_tolerance: 0.0
                scale_window: None
                threshold_loss_scale: None
            load_best_model_after_train: True
            max_clip_norm: None
            num_accumulated_batches: 1
            num_batches_per_epoch: None
            num_samples_to_log_progress: 1000
            optimizer: Adam.Config:
                eps: 1e-08
                lr: 0.001
                weight_decay: 1e-05
            privacy_engine: None
            report_train_metrics: True
            scheduler: None
            sparsifier: None
            target_time_limit_seconds: None
        use_elastic: None
    test_out_path: /tmp/test_out.txt
    torchscript_quantize: False
    use_config_from_snapshot: True
    use_cuda_for_testing: True
    use_cuda_if_available: True
    use_deterministic_cudnn: False
    use_fp16: False
    use_tensorboard: True
    version: 20


        # for debug of GPU
        use_cuda_if_available: True
        device_id: 0
        world_size: 1
        torch.cuda.is_available(): True
        cuda.CUDA_ENABLED: True
        cuda.DISTRIBUTED_WORLD_SIZE: 1
        
# for debug of FP16: fp16_enabled=False
Creating task: DocumentClassificationTask...
PyText data schema: {'text': <class 'str'>, 'label': <class 'str'>}.
Model :DocModel(
  (embedding): WordEmbedding(
    (word_embedding): Embedding(49, 100, padding_idx=1)
    (mlp): Sequential()
  )
  (representation): DocNNRepresentation(
    (convs): ModuleList(
      (0): Conv1d(100, 100, kernel_size=(3,), stride=(1,), padding=(3,))
      (1): Conv1d(100, 100, kernel_size=(4,), stride=(1,), padding=(4,))
    )
    (dropout): Dropout(p=0.4, inplace=False)
  )
  (decoder): MLPDecoder(
    (mlp): Sequential(
      (0): Linear(in_features=200, out_features=7, bias=True)
    )
  )
  (output_layer): MulticlassOutputLayer()
)
Num trainable parameters: 76507

Worker 0 starting epoch 1
Learning rate(s): 0.001
start training epoch 1


Stage.TRAIN
Epoch:1
loss: 1.950055
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             0.125 |   0.222 |
|          alarm/set_alarm |             0.306 |   0.381 |
|       alarm/snooze_alarm |             0.125 |   0.222 |
| alarm/time_left_on_alarm |             0.100 |   0.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             0.111 |   0.111 |
|             weather/find |             0.171 |   0.125 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.125 |   0.125 |   0.125 |   0.125 |   0.125 |
| alarm/set_alarm          |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| alarm/snooze_alarm       |   0.125 |   0.125 |   0.125 |   0.125 |   0.125 |
| alarm/time_left_on_alarm |   0.100 |   0.100 |   0.100 |   0.100 |   0.100 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.111 |   0.111 |   0.111 |   0.111 |   0.111 |
| weather/find             |   0.200 |   0.200 |   0.200 |   0.200 |   0.200 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  60.2ms |  60.2ms |  60.2ms |  60.2ms |  60.2ms |  60.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                | 268.4ms | 268.4ms | 268.4ms | 268.4ms | 268.4ms | 268.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |  12.5ns |  12.5ns |  12.5ns |  12.5ns |  12.5ns |  12.5ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 662.8ns | 662.8ns | 662.8ns | 662.8ns | 662.8ns | 662.8ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |  51.5ms |  51.5ms |  51.5ms |  51.5ms |  51.5ms |  51.5ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |  51.2ms |  51.2ms |  51.2ms |  51.2ms |  51.2ms |  51.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           | 207.1ms | 207.1ms | 207.1ms | 207.1ms | 207.1ms | 207.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   8.3ms |   8.3ms |   8.3ms |   8.3ms |   8.3ms |   8.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   8.1ms |   8.1ms |   8.1ms |   8.1ms |   8.1ms |   8.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   6.4ns |   6.4ns |   6.4ns |   6.4ns |   6.4ns |   6.4ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              |  16.2ns |  16.2ns |  16.2ns |  16.2ns |  16.2ns |  16.2ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 332.6ms |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 1
/usr/local/lib/python3.6/dist-packages/pytext/metrics/__init__.py:576: RuntimeWarning: invalid value encountered in long_scalars
  recall_diff = added_positives / total_positive


Stage.EVAL
Epoch:1
loss: 1.959144
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.500 |   0.667 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.500 |   0.500 |   0.500 |   0.500 |   0.500 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  20.4ms |  20.4ms |  20.4ms |  20.4ms |  20.4ms |  20.4ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   2.2ms |   2.2ms |   2.2ms |   2.2ms |   2.2ms |   2.2ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   6.4ns |   6.4ns |   6.4ns |   6.4ns |   6.4ns |   6.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 238.1ns | 238.1ns | 238.1ns | 238.1ns | 238.1ns | 238.1ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   6.3ns |   6.3ns |   6.3ns |   6.3ns |   6.3ns |   6.3ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   5.3ns |   5.3ns |   5.3ns |   5.3ns |   5.3ns |   5.3ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   4.2ns |   4.2ns |   4.2ns |   4.2ns |   4.2ns |   4.2ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   7.1ns |   7.1ns |   7.1ns |   7.1ns |   7.1ns |   7.1ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 26.1ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
Found a better model!

Worker 0 starting epoch 2
Learning rate(s): 0.001
start training epoch 2
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:2
loss: 1.928607
Accuracy: 30.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             0.333 |   0.778 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |             0.500 |   0.889 |
| alarm/time_left_on_alarm |             0.500 |   0.889 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             1.000 |   1.000 |
|             weather/find |             0.333 |   0.625 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.500 |   0.500 |   0.500 |   0.500 |   0.500 |
| alarm/time_left_on_alarm |   0.500 |   0.500 |   0.500 |   0.500 |   0.500 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  54.6ms |  54.6ms |  54.6ms |  54.6ms |  54.6ms |  54.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.7ms |   4.7ms |   4.7ms |   4.7ms |   4.7ms |   4.7ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   8.5ns |   8.5ns |   8.5ns |   8.5ns |   8.5ns |   8.5ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 253.1ns | 253.1ns | 253.1ns | 253.1ns | 253.1ns | 253.1ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 144.2ns | 144.2ns | 144.2ns | 144.2ns | 144.2ns | 144.2ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 62.2ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 2
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:2
loss: 1.956181
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.500 |   0.667 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.500 |   0.500 |   0.500 |   0.500 |   0.500 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  20.1ms |  20.1ms |  20.1ms |  20.1ms |  20.1ms |  20.1ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   1.9ms |   1.9ms |   1.9ms |   1.9ms |   1.9ms |   1.9ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   5.5ns |   5.5ns |   5.5ns |   5.5ns |   5.5ns |   5.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 242.4ns | 242.4ns | 242.4ns | 242.4ns | 242.4ns | 242.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.5ns |   4.5ns |   4.5ns |   4.5ns |   4.5ns |   4.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.4ns |   3.4ns |   3.4ns |   3.4ns |   3.4ns |   3.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   6.6ns |   6.6ns |   6.6ns |   6.6ns |   6.6ns |   6.6ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 24.9ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 3
Learning rate(s): 0.001
start training epoch 3
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:3
loss: 1.935113
Accuracy: 20.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             0.125 |   0.222 |
|          alarm/set_alarm |             0.792 |   0.762 |
|       alarm/snooze_alarm |             0.111 |   0.111 |
| alarm/time_left_on_alarm |             0.200 |   0.556 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             0.125 |   0.222 |
|             weather/find |             0.367 |   0.688 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   0.667 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.125 |   0.125 |   0.125 |   0.125 |   0.125 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.375 |   0.375 |
| alarm/snooze_alarm       |   0.111 |   0.111 |   0.111 |   0.111 |   0.111 |
| alarm/time_left_on_alarm |   0.200 |   0.200 |   0.200 |   0.200 |   0.200 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.125 |   0.125 |   0.125 |   0.125 |   0.125 |
| weather/find             |   0.400 |   0.400 |   0.400 |   0.400 |   0.400 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  53.0ms |  53.0ms |  53.0ms |  53.0ms |  53.0ms |  53.0ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.5ms |   4.5ms |   4.5ms |   4.5ms |   4.5ms |   4.5ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   8.7ns |   8.7ns |   8.7ns |   8.7ns |   8.7ns |   8.7ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 229.1ns | 229.1ns | 229.1ns | 229.1ns | 229.1ns | 229.1ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 134.1ns | 134.1ns | 134.1ns | 134.1ns | 134.1ns | 134.1ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 60.5ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 3
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:3
loss: 1.953246
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             0.833 |   0.750 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  20.8ms |  20.8ms |  20.8ms |  20.8ms |  20.8ms |  20.8ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   5.3ns |   5.3ns |   5.3ns |   5.3ns |   5.3ns |   5.3ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 208.2ns | 208.2ns | 208.2ns | 208.2ns | 208.2ns | 208.2ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.5ns |   4.5ns |   4.5ns |   4.5ns |   4.5ns |   4.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.5ns |   3.5ns |   3.5ns |   3.5ns |   3.5ns |   3.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   5.6ns |   5.6ns |   5.6ns |   5.6ns |   5.6ns |   5.6ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 25.3ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 4
Learning rate(s): 0.001
start training epoch 4
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:4
loss: 1.919790
Accuracy: 20.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |             0.200 |   0.556 |
| alarm/time_left_on_alarm |             0.500 |   0.889 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             0.200 |   0.556 |
|             weather/find |             1.000 |   1.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.200 |   0.200 |   0.200 |   0.200 |   0.200 |
| alarm/time_left_on_alarm |   0.500 |   0.500 |   0.500 |   0.500 |   0.500 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.200 |   0.200 |   0.200 |   0.200 |   0.200 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  53.4ms |  53.4ms |  53.4ms |  53.4ms |  53.4ms |  53.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.9ms |   4.9ms |   4.9ms |   4.9ms |   4.9ms |   4.9ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |  22.3ns |  22.3ns |  22.3ns |  22.3ns |  22.3ns |  22.3ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 255.4ns | 255.4ns | 255.4ns | 255.4ns | 255.4ns | 255.4ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   4.1ns |   4.1ns |   4.1ns |   4.1ns |   4.1ns |   4.1ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 152.8ns | 152.8ns | 152.8ns | 152.8ns | 152.8ns | 152.8ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 61.3ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 4
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:4
loss: 1.950457
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             0.833 |   0.750 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  19.3ms |  19.3ms |  19.3ms |  19.3ms |  19.3ms |  19.3ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   6.1ns |   6.1ns |   6.1ns |   6.1ns |   6.1ns |   6.1ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 229.9ns | 229.9ns | 229.9ns | 229.9ns | 229.9ns | 229.9ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   5.7ns |   5.7ns |   5.7ns |   5.7ns |   5.7ns |   5.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |  20.4ns |  20.4ns |  20.4ns |  20.4ns |  20.4ns |  20.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.6ns |   3.6ns |   3.6ns |   3.6ns |   3.6ns |   3.6ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 23.8ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 5
Learning rate(s): 0.001
start training epoch 5
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:5
loss: 1.918015
Accuracy: 20.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             0.698 |   0.762 |
|       alarm/snooze_alarm |             1.000 |   1.000 |
| alarm/time_left_on_alarm |             0.100 |   0.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             0.333 |   0.778 |
|             weather/find |             0.583 |   0.875 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.333 |   0.333 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   0.667 |   0.667 |   0.429 |   0.429 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   0.100 |   0.100 |   0.100 |   0.100 |   0.100 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| weather/find             |   0.667 |   0.667 |   0.667 |   0.667 |   0.667 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  76.5ms |  76.5ms |  76.5ms |  76.5ms |  76.5ms |  76.5ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.6ms |   4.6ms |   4.6ms |   4.6ms |   4.6ms |   4.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |  23.4ns |  23.4ns |  23.4ns |  23.4ns |  23.4ns |  23.4ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 223.2ns | 223.2ns | 223.2ns | 223.2ns | 223.2ns | 223.2ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   4.0ns |   4.0ns |   4.0ns |   4.0ns |   4.0ns |   4.0ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 147.9ns | 147.9ns | 147.9ns | 147.9ns | 147.9ns | 147.9ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 84.0ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 5
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:5
loss: 1.947547
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             0.833 |   0.750 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  20.7ms |  20.7ms |  20.7ms |  20.7ms |  20.7ms |  20.7ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   2.2ms |   2.2ms |   2.2ms |   2.2ms |   2.2ms |   2.2ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   6.5ns |   6.5ns |   6.5ns |   6.5ns |   6.5ns |   6.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 265.8ns | 265.8ns | 265.8ns | 265.8ns | 265.8ns | 265.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   6.4ns |   6.4ns |   6.4ns |   6.4ns |   6.4ns |   6.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   5.0ns |   5.0ns |   5.0ns |   5.0ns |   5.0ns |   5.0ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   4.4ns |   4.4ns |   4.4ns |   4.4ns |   4.4ns |   4.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   7.3ns |   7.3ns |   7.3ns |   7.3ns |   7.3ns |   7.3ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 26.0ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 6
Learning rate(s): 0.001
start training epoch 6
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:6
loss: 1.891813
Accuracy: 40.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |             1.000 |   1.000 |
| alarm/time_left_on_alarm |             1.000 |   1.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             0.500 |   0.889 |
|             weather/find |             1.000 |   1.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.500 |   0.500 |   0.500 |   0.500 |   0.500 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  54.4ms |  54.4ms |  54.4ms |  54.4ms |  54.4ms |  54.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.3ms |   4.3ms |   4.3ms |   4.3ms |   4.3ms |   4.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   8.8ns |   8.8ns |   8.8ns |   8.8ns |   8.8ns |   8.8ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 248.9ns | 248.9ns | 248.9ns | 248.9ns | 248.9ns | 248.9ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 131.8ns | 131.8ns | 131.8ns | 131.8ns | 131.8ns | 131.8ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 61.7ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 6
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:6
loss: 1.944635
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             0.833 |   0.750 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  19.3ms |  19.3ms |  19.3ms |  19.3ms |  19.3ms |  19.3ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 221.0ns | 221.0ns | 221.0ns | 221.0ns | 221.0ns | 221.0ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   5.9ns |   5.9ns |   5.9ns |   5.9ns |   5.9ns |   5.9ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch | 997.5ns | 997.5ns | 997.5ns | 997.5ns | 997.5ns | 997.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.5ns |   4.5ns |   4.5ns |   4.5ns |   4.5ns |   4.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.6ns |   3.6ns |   3.6ns |   3.6ns |   3.6ns |   3.6ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   6.5ns |   6.5ns |   6.5ns |   6.5ns |   6.5ns |   6.5ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 23.9ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 7
Learning rate(s): 0.001
start training epoch 7
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:7
loss: 1.895764
Accuracy: 30.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |             1.000 |   1.000 |
| alarm/time_left_on_alarm |             1.000 |   1.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             1.000 |   1.000 |
|             weather/find |             0.833 |   0.938 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  53.1ms |  53.1ms |  53.1ms |  53.1ms |  53.1ms |  53.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   6.5ms |   6.5ms |   6.5ms |   6.5ms |   6.5ms |   6.5ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   9.3ns |   9.3ns |   9.3ns |   9.3ns |   9.3ns |   9.3ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 274.2ns | 274.2ns | 274.2ns | 274.2ns | 274.2ns | 274.2ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   2.4ms |   2.4ms |   2.4ms |   2.4ms |   2.4ms |   2.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.7ms |   1.7ms |   1.7ms |   1.7ms |   1.7ms |   1.7ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   3.9ns |   3.9ns |   3.9ns |   3.9ns |   3.9ns |   3.9ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 146.6ns | 146.6ns | 146.6ns | 146.6ns | 146.6ns | 146.6ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 62.6ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 7
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:7
loss: 1.941633
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             0.833 |   0.750 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  19.6ms |  19.6ms |  19.6ms |  19.6ms |  19.6ms |  19.6ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   2.0ms |   2.0ms |   2.0ms |   2.0ms |   2.0ms |   2.0ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   6.0ns |   6.0ns |   6.0ns |   6.0ns |   6.0ns |   6.0ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 209.7ns | 209.7ns | 209.7ns | 209.7ns | 209.7ns | 209.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.4ns |   4.4ns |   4.4ns |   4.4ns |   4.4ns |   4.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.8ns |   3.8ns |   3.8ns |   3.8ns |   3.8ns |   3.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   6.3ns |   6.3ns |   6.3ns |   6.3ns |   6.3ns |   6.3ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 24.5ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 8
Learning rate(s): 0.001
start training epoch 8
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:8
loss: 1.885373
Accuracy: 40.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |             0.333 |   0.778 |
| alarm/time_left_on_alarm |             1.000 |   1.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             1.000 |   1.000 |
|             weather/find |             1.000 |   1.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  61.6ms |  61.6ms |  61.6ms |  61.6ms |  61.6ms |  61.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   5.6ms |   5.6ms |   5.6ms |   5.6ms |   5.6ms |   5.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   9.1ns |   9.1ns |   9.1ns |   9.1ns |   9.1ns |   9.1ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 266.3ns | 266.3ns | 266.3ns | 266.3ns | 266.3ns | 266.3ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.6ms |   1.6ms |   1.6ms |   1.6ms |   1.6ms |   1.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.6ms |   1.6ms |   1.6ms |   1.6ms |   1.6ms |   1.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   4.0ns |   4.0ns |   4.0ns |   4.0ns |   4.0ns |   4.0ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 190.7ns | 190.7ns | 190.7ns | 190.7ns | 190.7ns | 190.7ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 70.1ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 8
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:8
loss: 1.938505
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             0.833 |   0.750 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  19.5ms |  19.5ms |  19.5ms |  19.5ms |  19.5ms |  19.5ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   2.5ms |   2.5ms |   2.5ms |   2.5ms |   2.5ms |   2.5ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 234.4ns | 234.4ns | 234.4ns | 234.4ns | 234.4ns | 234.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.7ms |   1.7ms |   1.7ms |   1.7ms |   1.7ms |   1.7ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.7ns |   4.7ns |   4.7ns |   4.7ns |   4.7ns |   4.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 25.5ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 9
Learning rate(s): 0.001
start training epoch 9
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:9
loss: 1.872775
Accuracy: 60.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |             1.000 |   1.000 |
| alarm/time_left_on_alarm |             1.000 |   1.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             1.000 |   1.000 |
|             weather/find |             1.000 |   1.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  62.0ms |  62.0ms |  62.0ms |  62.0ms |  62.0ms |  62.0ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.6ms |   4.6ms |   4.6ms |   4.6ms |   4.6ms |   4.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   9.1ns |   9.1ns |   9.1ns |   9.1ns |   9.1ns |   9.1ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 250.3ns | 250.3ns | 250.3ns | 250.3ns | 250.3ns | 250.3ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   3.8ns |   3.8ns |   3.8ns |   3.8ns |   3.8ns |   3.8ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 156.4ns | 156.4ns | 156.4ns | 156.4ns | 156.4ns | 156.4ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 69.5ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 9
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:9
loss: 1.935212
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  19.6ms |  19.6ms |  19.6ms |  19.6ms |  19.6ms |  19.6ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   5.7ns |   5.7ns |   5.7ns |   5.7ns |   5.7ns |   5.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 206.6ns | 206.6ns | 206.6ns | 206.6ns | 206.6ns | 206.6ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   5.7ns |   5.7ns |   5.7ns |   5.7ns |   5.7ns |   5.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.6ns |   4.6ns |   4.6ns |   4.6ns |   4.6ns |   4.6ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   6.1ns |   6.1ns |   6.1ns |   6.1ns |   6.1ns |   6.1ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 24.1ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 10
Learning rate(s): 0.001
start training epoch 10
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:10
loss: 1.872801
Accuracy: 60.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             0.792 |   0.762 |
|       alarm/snooze_alarm |             1.000 |   1.000 |
| alarm/time_left_on_alarm |             1.000 |   1.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             1.000 |   1.000 |
|             weather/find |             1.000 |   1.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   0.667 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.375 |   0.375 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  58.4ms |  58.4ms |  58.4ms |  58.4ms |  58.4ms |  58.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.6ms |   4.6ms |   4.6ms |   4.6ms |   4.6ms |   4.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   8.6ns |   8.6ns |   8.6ns |   8.6ns |   8.6ns |   8.6ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 246.9ns | 246.9ns | 246.9ns | 246.9ns | 246.9ns | 246.9ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   3.8ns |   3.8ns |   3.8ns |   3.8ns |   3.8ns |   3.8ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 130.3ns | 130.3ns | 130.3ns | 130.3ns | 130.3ns | 130.3ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 65.9ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 10
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:10
loss: 1.931939
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  26.1ms |  26.1ms |  26.1ms |  26.1ms |  26.1ms |  26.1ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 217.9ns | 217.9ns | 217.9ns | 217.9ns | 217.9ns | 217.9ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |  20.8ns |  20.8ns |  20.8ns |  20.8ns |  20.8ns |  20.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.3ns |   4.3ns |   4.3ns |   4.3ns |   4.3ns |   4.3ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 30.6ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

=== Saving model to: /tmp/model.pt
Saving pytorch model to: /tmp/model.pt
Exporting metrics
WARNING: Unable to visualize embedding space in TensorBoard.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pytext/metric_reporters/channel.py", line 376, in export
    model.embedding.visualize(self.summary_writer)
  File "/usr/local/lib/python3.6/dist-packages/pytext/models/embeddings/word_embedding.py", line 168, in visualize
    self.word_embedding.weight, metadata=self.vocab
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/tensorboard/writer.py", line 788, in add_embedding
    fs = tf.io.gfile.get_filesystem(save_path)
AttributeError: module 'tensorflow._api.v2.io.gfile' has no attribute 'get_filesystem'
Saving caffe2 model to: /tmp/model.caffe2.predictor
Saving onnx model to: /tmp/model.onnx
/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py:243: UserWarning: `add_node_names' can be set to True only when 'operator_export_type' is `ONNX`. Since 'operator_export_type' is not set to 'ONNX', `add_node_names` argument will be ignored.
  "`{}` argument will be ignored.".format(arg_name, arg_name))
/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py:243: UserWarning: `do_constant_folding' can be set to True only when 'operator_export_type' is `ONNX`. Since 'operator_export_type' is not set to 'ONNX', `do_constant_folding` argument will be ignored.
  "`{}` argument will be ignored.".format(arg_name, arg_name))
/bin/bash: line 1:   357 Segmentation fault      (core dumped) pytext train < pytext/demo/configs/docnn.json

Expected Results

The model snapshot should have been saved without a segmentation fault.

Relevant Code

Set up environment:

!pip install --quiet git+https://github.com/facebookresearch/pytext
!git clone https://github.com/facebookresearch/pytext.git
!mv pytext/tests/ .

Import modules:

import pytext

Train:

!pytext train < pytext/demo/configs/docnn.json

The text was updated successfully, but these errors were encountered:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segmentation fault when using PyText 0.3.3 on Google Colab #1415

Segmentation fault when using PyText 0.3.3 on Google Colab #1415

orestxherija commented Aug 3, 2020

Segmentation fault when using PyText 0.3.3 on Google Colab #1415

Segmentation fault when using PyText 0.3.3 on Google Colab #1415

Comments

orestxherija commented Aug 3, 2020

Steps to reproduce

Observed Results

Expected Results

Relevant Code