Enabled DETR (Object Detection) model #1046

cfgfung · 2024-06-05T21:40:58Z

What does this PR do?

This PR contains the patch, example, and test codes for DETR models.

A100 CUDA BF16(Autocast) benchmarks:
n_iterations: 10
Total latency (ms): 241.29199981689453
Average latency (ms): 24.129199981689453

Gaudi2 BF16(Autocast and Graph mode) benchmarks:
n_iterations: 10
Total latency (ms): 65.3073787689209
Average latency (ms): 6.53073787689209

Before submitting

[N.A.] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[Yes] Did you make sure to update the documentation with your changes?
[Yes] Did you write any new necessary tests?

libinta · 2024-06-25T01:32:24Z

@cfgfung can you rebase the PR?

cfgfung · 2024-06-25T18:41:03Z

@cfgfung can you rebase the PR?

Hi,
I have applied rebase and here is the test result

imangohari1

Hi @cfgfung
thank you for the work here.
Below are my suggestions for this PR. Let's work on these and do a final review.

I've made some changes regarding some minor clean ups and ci tests. Please review and implement them via git am < 0001* (file attached).
Few minor comments are give in comment.

0001-fea-Minor-CI-updates-and-clean-ups.patch

G2 results (after applying the patch)

---------------------------: System Configuration :---------------------------
Num CPU Cores : 160
CPU RAM       : 1056375224 KB
------------------------------------------------------------------------------
Detected cat with confidence 1.0 at location [344.0, 25.25, 640.0, 376.0]
Detected remote with confidence 0.996 at location [328.0, 75.5, 372.0, 188.0]
Detected remote with confidence 0.996 at location [39.0, 70.5, 176.0, 118.0]
Detected cat with confidence 1.0 at location [15.62, 52.5, 316.0, 472.0]
Detected couch with confidence 0.996 at location [-1.25, 0.94, 640.0, 472.0]

Stats:
------------------------------------------------------------
Total latency (ms): 58.38346481323242 (for n_iterations=10)
Average latency (ms): 5.838346481323242 (per iteration)
------------------------------------------------------------

imangohari1

minor comments.

imangohari1 · 2024-06-27T16:08:32Z

Makefile

+# Run unit and integration tests related to Image segmentation
+fast_tests_object_detection:
+	python -m pip install .[tests]
+	python -m pip install timm


if no_timm is used for the test, is this needed?

Yes, it is still needed. The model has called some API/modules from timm library

then let's move this to here: https://github.com/huggingface/optimum-habana/blob/main/setup.py#L41-L51
doesn't need to be a separate pip install.

imangohari1 · 2024-06-27T16:09:11Z

examples/object-detection/run_example.py

+
+    adapt_transformers_to_gaudi()
+
+    # you can specify the revision tag if you don't want the timm dependency


not sure how to approach this here, but would be nice to able to pass/toggle between no_timm and main revisions.

Hi,
The DETR is composed of different modules from timm, transformer and other popular libraries.
Calling this line will apply any applicable optimizations to this DETR model, especially the transformer submodules.

imangohari1 · 2024-07-13T00:09:24Z

@cfgfung
Can we move this pr along? Pls. apply the changes I suggested. Tnx.

cfgfung · 2024-07-15T20:27:21Z

Hi @cfgfung thank you for the work here. Below are my suggestions for this PR. Let's work on these and do a final review.

I've made some changes regarding some minor clean ups and ci tests. Please review and implement them via git am < 0001* (file attached).

Few minor comments are give in comment.

0001-fea-Minor-CI-updates-and-clean-ups.patch

G2 results (after applying the patch)
---------------------------: System Configuration :---------------------------
Num CPU Cores : 160
CPU RAM       : 1056375224 KB
------------------------------------------------------------------------------
Detected cat with confidence 1.0 at location [344.0, 25.25, 640.0, 376.0]
Detected remote with confidence 0.996 at location [328.0, 75.5, 372.0, 188.0]
Detected remote with confidence 0.996 at location [39.0, 70.5, 176.0, 118.0]
Detected cat with confidence 1.0 at location [15.62, 52.5, 316.0, 472.0]
Detected couch with confidence 0.996 at location [-1.25, 0.94, 640.0, 472.0]

Stats:
------------------------------------------------------------
Total latency (ms): 58.38346481323242 (for n_iterations=10)
Average latency (ms): 5.838346481323242 (per iteration)
------------------------------------------------------------

I have applied the patch.

cfgfung · 2024-07-15T20:43:35Z

@imangohari1 Hi, I have applied rebase and addressed the changes mentioned above.

imangohari1

minor change.
Also could we test this with multiple input images?

Please make sure to run make style and rebase/sync with the head of OH at main.

imangohari1 · 2024-07-15T21:39:58Z

Makefile

+# Run unit and integration tests related to Image segmentation
+fast_tests_object_detection:
+	python -m pip install .[tests]
+	python -m pip install timm


then let's move this to here: https://github.com/huggingface/optimum-habana/blob/main/setup.py#L41-L51
doesn't need to be a separate pip install.

cfgfung · 2024-07-15T23:18:48Z

minor change. Also could we test this with multiple input images?

Please make sure to run make style and rebase/sync with the head of OH at main.

Moved the timm installation command to setup.py. This is the result for the make style

A single image should be enough to safeguard the precision/numerical issues and throughput. The following merged PRs are also using a single image as a test.

#801
#814
#826

imangohari1

Looks good.

@regisss
Could you take a final look here please?
Thanks

HuggingFaceDocBuilderDev · 2024-07-22T08:41:46Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

setup.py

yeonsily · 2024-07-25T18:23:26Z

@cfgfung can you please fix code style?

cfgfung · 2024-07-25T23:07:38Z

@cfgfung can you please fix code style?

Hi @yeonsily,

On my side, I cannot see anything changed using make style

Anyhow, I have rebased with the latest main and pushed the updates.

regisss

LGTM!

cfgfung requested a review from regisss as a code owner June 5, 2024 21:40

cfgfung force-pushed the examples/detr_resnet branch from b26d202 to 5dddd9e Compare June 25, 2024 18:28

imangohari1 suggested changes Jun 27, 2024

View reviewed changes

imangohari1 reviewed Jun 27, 2024

View reviewed changes

cfgfung force-pushed the examples/detr_resnet branch from 6e0f7e9 to 1d2eb59 Compare July 15, 2024 20:42

imangohari1 suggested changes Jul 15, 2024

View reviewed changes

imangohari1 approved these changes Jul 16, 2024

View reviewed changes

libinta added the run-test Run CI for PRs from external contributors label Jul 16, 2024

regisss reviewed Jul 22, 2024

View reviewed changes

setup.py Show resolved Hide resolved

splotnikv mentioned this pull request Jul 24, 2024

Enable eager mode training for DETR #1155

Draft

libinta added the synapse1.17 PR that should be available along with Synapse 1.17 but have no dependency on Synapse 1.17 content. label Jul 24, 2024

cfgfung and others added 6 commits July 25, 2024 23:08

Enabled DETR on Gaudi.

8dfac42

Added test case.

2f4639b

Amend the naming of the function.

edc4d91

Improved the test to adapt Synapse AI 1.16

f49af2f

fea(): Minor CI updates and clean ups

0a02cd3

Move installation command of timm library to setup.py

8e088af

cfgfung force-pushed the examples/detr_resnet branch from 475e63a to 8e088af Compare July 25, 2024 23:16

regisss approved these changes Jul 28, 2024

View reviewed changes

regisss merged commit a246660 into huggingface:main Jul 28, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enabled DETR (Object Detection) model #1046

Enabled DETR (Object Detection) model #1046

cfgfung commented Jun 5, 2024

libinta commented Jun 25, 2024

cfgfung commented Jun 25, 2024

imangohari1 left a comment •

edited

Loading

imangohari1 left a comment

imangohari1 Jun 27, 2024

cfgfung Jul 15, 2024

imangohari1 Jul 15, 2024

imangohari1 Jun 27, 2024

cfgfung Jul 15, 2024

imangohari1 commented Jul 13, 2024

cfgfung commented Jul 15, 2024

cfgfung commented Jul 15, 2024

imangohari1 left a comment •

edited

Loading

imangohari1 Jul 15, 2024

cfgfung commented Jul 15, 2024

imangohari1 left a comment

HuggingFaceDocBuilderDev commented Jul 22, 2024

yeonsily commented Jul 25, 2024

cfgfung commented Jul 25, 2024 •

edited

Loading

regisss left a comment


		adapt_transformers_to_gaudi()

		# you can specify the revision tag if you don't want the timm dependency

Enabled DETR (Object Detection) model #1046

Enabled DETR (Object Detection) model #1046

Conversation

cfgfung commented Jun 5, 2024

What does this PR do?

Before submitting

libinta commented Jun 25, 2024

cfgfung commented Jun 25, 2024

imangohari1 left a comment • edited Loading

Choose a reason for hiding this comment

imangohari1 left a comment

Choose a reason for hiding this comment

imangohari1 Jun 27, 2024

Choose a reason for hiding this comment

cfgfung Jul 15, 2024

Choose a reason for hiding this comment

imangohari1 Jul 15, 2024

Choose a reason for hiding this comment

imangohari1 Jun 27, 2024

Choose a reason for hiding this comment

cfgfung Jul 15, 2024

Choose a reason for hiding this comment

imangohari1 commented Jul 13, 2024

cfgfung commented Jul 15, 2024

cfgfung commented Jul 15, 2024

imangohari1 left a comment • edited Loading

Choose a reason for hiding this comment

imangohari1 Jul 15, 2024

Choose a reason for hiding this comment

cfgfung commented Jul 15, 2024

imangohari1 left a comment

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Jul 22, 2024

yeonsily commented Jul 25, 2024

cfgfung commented Jul 25, 2024 • edited Loading

regisss left a comment

Choose a reason for hiding this comment

imangohari1 left a comment •

edited

Loading

imangohari1 left a comment •

edited

Loading

cfgfung commented Jul 25, 2024 •

edited

Loading