Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash When Generating Multiple Rooms #333

Open
GavinZhengOI opened this issue Sep 22, 2024 · 1 comment
Open

Crash When Generating Multiple Rooms #333

GavinZhengOI opened this issue Sep 22, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@GavinZhengOI
Copy link

Describe the bug

As I mentioned in #326 , Infinigen crashed while generating an indoor scene with multiple rooms containing objects.

Steps to Reproduce

python -m infinigen.datagen.manage_jobs --output_folder outputs/huge_dataset_nooverride \
--num_scenes 5 --pipeline_configs local_256GB.gin monocular.gin blender_gt.gin indoor_background_configs.gin \ 
--pipeline_overrides get_cmd.driver_script='infinigen_examples.generate_indoors' manage_datagen_jobs.num_concurrent=16

This will run tens of hours and then crash.

What version of the code were you using?

v1.8.1

commit 88fb49cde0bbca401601d05d672b31d28e9b45cb (HEAD -> main, origin/main, origin/HEAD)
Merge: 126a41eb ef50641f
Author: Alex Raistrick <[email protected]>
Date:   Fri Aug 23 13:01:23 2024 -0400

    Merge pull request #252 from princeton-vl/develop
    
    v1.8.1

What command did you run?

python -m infinigen.datagen.manage_jobs --output_folder outputs/huge_dataset_nooverride \
--num_scenes 5 --pipeline_configs local_256GB.gin monocular.gin blender_gt.gin indoor_background_configs.gin \ 
--pipeline_overrides get_cmd.driver_script='infinigen_examples.generate_indoors' manage_datagen_jobs.num_concurrent=16

What are your FULL output logs?

crash_summaries.txt

First Sample

92139964677_0_log.out.txt
92139964677_0_log.err.txt

Second Sample

80530831448_0_log.out.txt
80530831448_0_log.err.txt

If this is your first time running Infinigen, what are the full install logs?**

Not the first time running. If turn on --configs singleroom.gin and --overrides compose_indoors.restrict_single_supported_roomtype=True, it does not lead to a crash.

Platform

  • OS & OS Version: Ubuntu 24.04.1 LTS
  • GPU: 4090
  • GPU Driver Version: 535
  • RAM (GB): 64GB

Additional context

I suspect this might be a memory leak issue. I noticed that swap usage was over 85%, while memory utilization was only around 50%. I think it's possible that memory is leaking during execution and eventually consuming all available memory. I've just installed Prometheus on my system and started another execution with the same configuration to monitor memory usage. I'll update you once I have the results.

@GavinZhengOI GavinZhengOI added the bug Something isn't working label Sep 22, 2024
@GavinZhengOI
Copy link
Author

CleanShot 2024-09-23 at 09 25 42@2x
It seems like there are some kind of memory leak. I'm using python -m infinigen.datagen.manage_jobs --output_folder outputs/single_data_test --num_scenes 1 --pipeline_configs local_64GB.gin monocular.gin blender_gt.gin indoor_background_configs.gin --pipeline_overrides get_cmd.driver_script='infinigen_examples.generate_indoors' manage_datagen_jobs.num_concurrent=16 running single job, and it takes more than 55G of ram gradually then crashed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant