[BugFix,Refactor] Dreamer refactor #1918

BY571 · 2024-02-16T10:45:29Z

Description

Updates the dreamer example to be aligned with other example scripts and prove paper performance.
Adds the option to run dreamer with non-image obs environments and fixes small issues in the objective and other elements.

Motivation and Context

Why is this change required? What problem does it solve?
If it fixes an open issue, please link to the issue here.
You can use the syntax close #15213 if this solves the issue #15213

I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds core functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (update in the documentation)
Example (update in the folder of examples)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

I have read the CONTRIBUTION guide (required)
My change requires a change to the documentation.
I have updated the tests accordingly (required for a bug fix or a new feature).
I have updated the documentation accordingly.

pytorch-bot · 2024-02-16T10:45:33Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1918

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures, 11 Unrelated Failures

As of commit 7733c37 with merge base 0ea236d ():

NEW FAILURES - The following jobs have failed:

Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
Workflow failed! Resource not accessible by integration
Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
Workflow failed! Resource not accessible by integration
Examples Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
RuntimeError: Command docker exec -t 602165d007b7aad281233917ae77cde2b00d4369a13ce063d66acbda1ad68175 /exec failed with exit code 1
Unit-tests on MacOS CPU / tests (3.11) / macos-job (gh)
The process '/usr/local/bin/git' failed with exit code 128
Unit-tests on MacOS CPU / tests (3.8) / macos-job (gh)
The process '/usr/local/bin/git' failed with exit code 128
Unit-tests on Windows / unittests-cpu / windows-job (gh)
The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

Habitat Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Libs Tests on Linux / unittests-gym (3.9, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Libs Tests on Linux / unittests-sklearn (3.9, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-cpu (3.8) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-gpu (3.10, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-optdeps (3.10, 12.1) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128
Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh)
The process '/usr/bin/git' failed with exit code 128

This comment was automatically generated by Dr. CI and updates every 15 minutes.

BY571 · 2024-02-16T10:47:46Z

Current performance for cheetah run dm_control task. Not yet paper performance.

BY571 · 2024-02-16T10:52:34Z

torchrl/envs/common.py

@@ -2421,6 +2421,10 @@ def _rollout_stop_early(
                    tensordict = tensordict.to(policy_device, non_blocking=True)
                else:
                    tensordict.clear_device_()
+


This is not ideal and just a current solution, Id be interested to hear what you think @vmoens. We need to detach those values similar to this maybe via a detach transform?

do we need to backprop anything in this loop?

I added a transform to the mb env,
can you confirm that we always want to detach these values in mbenv.step?

# Conflicts: # sota-implementations/dreamer/dreamer_utils.py # torchrl/objectives/dreamer.py

vmoens · 2024-04-08T12:46:43Z

sota-implementations/dreamer/dreamer_utils.py

+        frames_per_batch=cfg.collector.frames_per_batch,
+        total_frames=cfg.collector.total_frames,
+        device=cfg.collector.device,
+        reset_at_each_iter=True,


We should set the horizon instead

vmoens · 2024-04-18T12:46:27Z

torchrl/objectives/dreamer.py

+        return (
+            -0.5 * ((x.to(mean.dtype) - mean) / std).pow(2) - std.log()
+        )  # - 0.5 * math.log(2 * math.pi)
+


here's the equation of the normal log prob if you're interested. It's a distance loss as you can see

torchrl/objectives/dreamer.py

# Conflicts: # torchrl/data/replay_buffers/storages.py

torchrl/objectives/dreamer.py

BY571 added 17 commits January 25, 2024 11:54

update config

a7b6d33

update dreamer utils

171b15c

fixes

2f273a8

fix

6856587

flake

578150e

update and add dense networks

82fc9d8

updates loss

0e48d9a

update losses

e839dee

fixes

b555c9b

test changes

46e7234

add eval env

12a594d

use independent normal + cleanup + dense encoder/decoder

0f21038

cleanup

fe65b95

fixes

8adef8a

Merge branch 'main' into dreamer_v1_refactor

cc61cea

Merge branch 'main' into dreamer_v1_refactor

fb47a66

update naming

1faacc9

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 16, 2024

BY571 commented Feb 16, 2024

View reviewed changes

vmoens changed the title ~~[WIP] Dreamer refactor~~ [BugFix,Refactor] Dreamer refactor Apr 8, 2024

vmoens added 5 commits April 8, 2024 11:15

Merge branch 'main' into dreamer_v1_refactor

6a79e10

# Conflicts: # sota-implementations/dreamer/dreamer_utils.py # torchrl/objectives/dreamer.py

amend

99e9c3c

amend

fbb09fa

amend

a7554c9

amend

ac5f2fa

vmoens reviewed Apr 8, 2024

View reviewed changes

vmoens added 2 commits April 8, 2024 14:56

amend

a912f7c

amend

74cf3f8

vmoens added 5 commits April 17, 2024 20:54

amend

e9b6ebc

amend

d881613

amend

449f962

amend

b2473aa

amend

6d8e006

vmoens reviewed Apr 18, 2024

View reviewed changes

torchrl/objectives/dreamer.py Outdated Show resolved Hide resolved

vmoens and others added 4 commits April 18, 2024 13:46

Update torchrl/objectives/dreamer.py

0a86244

lint

dac6a36

Merge remote-tracking branch 'origin/main' into dreamer_v1_refactor

1315a4e

Merge remote-tracking branch 'origin/main' into dreamer_v1_refactor

7d0a158

vmoens marked this pull request as ready for review April 18, 2024 16:08

vmoens added 4 commits April 18, 2024 18:41

Merge remote-tracking branch 'origin/main' into dreamer_v1_refactor

4451f63

# Conflicts: # torchrl/data/replay_buffers/storages.py

lint

7441f79

lint

4f374d9

amend

2dfa7ae

vmoens reviewed Apr 22, 2024

View reviewed changes

torchrl/objectives/dreamer.py Outdated Show resolved Hide resolved

torchrl/objectives/dreamer.py Outdated Show resolved Hide resolved

vmoens and others added 12 commits April 22, 2024 16:11

Update torchrl/objectives/dreamer.py

4e74969

Update torchrl/objectives/dreamer.py

b36f86b

Merge remote-tracking branch 'origin/main' into dreamer_v1_refactor

63f7580

fix examples

46e8ac0

amend

e43aee4

amend

fd23a54

Merge remote-tracking branch 'origin/main' into dreamer_v1_refactor

dbc4954

init

98d4020

amend

a9e1cb0

amend

12db41f

amend

81ec41c

amend

7733c37

vmoens merged commit bfadce9 into pytorch:main Apr 23, 2024
21 of 27 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix,Refactor] Dreamer refactor #1918

[BugFix,Refactor] Dreamer refactor #1918

BY571 commented Feb 16, 2024

pytorch-bot bot commented Feb 16, 2024 •

edited

Loading

BY571 commented Feb 16, 2024

BY571 Feb 16, 2024

vmoens Apr 8, 2024

vmoens Apr 8, 2024

vmoens Apr 8, 2024

vmoens Apr 18, 2024

[BugFix,Refactor] Dreamer refactor #1918

[BugFix,Refactor] Dreamer refactor #1918

Conversation

BY571 commented Feb 16, 2024

Description

Motivation and Context

Types of changes

Checklist

pytorch-bot bot commented Feb 16, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1918

❌ 6 New Failures, 11 Unrelated Failures

BY571 commented Feb 16, 2024

BY571 Feb 16, 2024

Choose a reason for hiding this comment

vmoens Apr 8, 2024

Choose a reason for hiding this comment

vmoens Apr 8, 2024

Choose a reason for hiding this comment

vmoens Apr 8, 2024

Choose a reason for hiding this comment

vmoens Apr 18, 2024

Choose a reason for hiding this comment

pytorch-bot bot commented Feb 16, 2024 •

edited

Loading