[rank2]: AttributeError: 'IndexedDataset' object has no attribute 'sizes'

[wccccp]

File "/gpfs01/unifiedcsi/gpfs/csi-dfs-ti-platform-fs/wcp/vllm_test/torchtune/Pai-Megatron-Patch/examples/llama2/pretrain_megatron_llama.py", line 110, in train_valid_test_datasets_provider
[rank2]: build_pretrain_dataset_from_idxmap(
[rank2]: File "/gpfs01/unifiedcsi/gpfs/csi-dfs-ti-platform-fs/wcp/vllm_test/torchtune/Pai-Megatron-Patch/megatron_patch/data/init.py", line 139, in build_pretrain_dataset_from_idxmap
[rank2]: return _build_train_valid_test_datasets(data_prefix[0],max_padding_length,
[rank2]: File "/gpfs01/unifiedcsi/gpfs/csi-dfs-ti-platform-fs/wcp/vllm_test/torchtune/Pai-Megatron-Patch/megatron_patch/data/init.py", line 224, in _build_train_valid_test_datasets
[rank2]: train_dataset = build_dataset(0, 'train')
[rank2]: File "/gpfs01/unifiedcsi/gpfs/csi-dfs-ti-platform-fs/wcp/vllm_test/torchtune/Pai-Megatron-Patch/megatron_patch/data/init.py", line 215, in build_dataset
[rank2]: dataset = LLamaIdxMapDataset(
[rank2]: File "/gpfs01/unifiedcsi/gpfs/csi-dfs-ti-platform-fs/wcp/vllm_test/torchtune/Pai-Megatron-Patch/megatron_patch/data/llama.py", line 248, in init
[rank2]: assert np.max(documents) < indexed_dataset.sizes.shape[0]
[rank2]: AttributeError: 'IndexedDataset' object has no attribute 'sizes'

[bao-xiaoyi]

同样的问题，蹲一下