update results for brightness

ika-rwth-aachen · Mar 28, 2024 · 1d2ada7 · 1d2ada7
1 parent d9eba27
commit 1d2ada7
Show file tree

Hide file tree

Showing 14 changed files with 278 additions and 290 deletions.
diff --git a/README.md b/README.md
@@ -112,28 +112,30 @@ Preprint: [https://arxiv.org/abs/2402.11677](https://arxiv.org/abs/2402.11677)
 *Note: Right click and click on `Open Image in new tab` to enlarge an animation*
 
 ## News
-- [01.02.2024] Initial Release with **10 corruption types** and **5 evaluated models**
+- [28.03.2024] **v0.0.3** Changed severity configuration for *Brightness*, reevaluated all models and metrics
+- [17.02.2024] **v0.0.2** Changed severity configuration for *Pointsreducing*, reevaluated all models and metrics
+- [01.02.2024] **v0.0.1** Initial Release with **10 corruption types** and **5 evaluated models**
 
 ## Benchmark Results
 ### Resistance Ability (RA) computed with NDS metric
 
 | Model                                                           | Clean     | Beams Red.   | Brightness   | Darkness   | Fog       | Missing Cam.   | Motion Blur   | Points Red.   | Snow      | Spatial Mis.   | Temporal Mis.   | mRA       |
 |:----------------------------------------------------------------|:----------|:-------------|:-------------|:-----------|:----------|:---------------|:--------------|:--------------|:----------|:---------------|:----------------|:----------|
-| [CMT](https://github.com/junjie18/CMT)                          | 0.729     | **0.786**    | 0.985        | 0.948      | **0.806** | 0.974          | 0.841         | **0.925**     | **0.833** | **0.809**      | **0.788**       | **0.870** |
-| [Sparsefusion](https://github.com/yichen928/SparseFusion)       | **0.732** | 0.689        | 0.992        | 0.963      | 0.767     | 0.954          | 0.848         | 0.879         | 0.770     | 0.714          | 0.777           | 0.835     |
-| [BEVfusion](https://github.com/mit-han-lab/bevfusion)           | 0.714     | 0.676        | 0.987        | 0.969      | 0.752     | 0.974          | **0.866**     | 0.872         | 0.774     | 0.705          | 0.742           | 0.832     |
-| [TransFusion](https://github.com/XuyangBai/TransFusion)         | 0.708     | 0.633        | **0.998**    | **0.988**  | 0.754     | **0.985**      | 0.826         | 0.851         | 0.748     | 0.685          | 0.777           | 0.824     |
-| [DeepInteraction](https://github.com/fudan-zvg/DeepInteraction) | 0.691     | 0.655        | 0.984        | 0.929      | 0.583     | 0.842          | 0.832         | 0.882         | 0.759     | 0.731          | 0.768           | 0.796     |
+| [CMT](https://github.com/junjie18/CMT)                          | 0.729     | **0.786**    | 0.937        | 0.948      | **0.806** | 0.974          | 0.841         | **0.925**     | **0.833** | **0.809**      | **0.788**       | **0.865** |
+| [Sparsefusion](https://github.com/yichen928/SparseFusion)       | **0.732** | 0.689        | 0.975        | 0.963      | 0.767     | 0.954          | 0.848         | 0.879         | 0.770     | 0.714          | 0.777           | 0.834     |
+| [BEVfusion](https://github.com/mit-han-lab/bevfusion)           | 0.714     | 0.676        | 0.967        | 0.969      | 0.752     | 0.974          | **0.866**     | 0.872         | 0.774     | 0.705          | 0.742           | 0.830     |
+| [TransFusion](https://github.com/XuyangBai/TransFusion)         | 0.708     | 0.633        | **0.993**    | **0.988**  | 0.754     | **0.985**      | 0.826         | 0.851         | 0.748     | 0.685          | 0.777           | 0.824     |
+| [DeepInteraction](https://github.com/fudan-zvg/DeepInteraction) | 0.691     | 0.655        | 0.969        | 0.929      | 0.583     | 0.842          | 0.832         | 0.882         | 0.759     | 0.731          | 0.768           | 0.795     |
 
 
 ### Relative Resistance Ability (RRA) computed with NDS metric and baseline BEVfusion
 | Model                                                           | Clean     | Beams Red.   | Brightness   | Darkness   | Fog       | Missing Cam.   | Motion Blur   | Points Red.   | Snow      | Spatial Mis.   | Temporal Mis.   | mRRA      |
 |:----------------------------------------------------------------|:----------|:-------------|:-------------|:-----------|:----------|:---------------|:--------------|:--------------|:----------|:---------------|:----------------|:----------|
-| [BEVfusion](https://github.com/mit-han-lab/bevfusion)           | 0.714     | -----        | -----        | -----      | -----     | -----          | -----         | -----         | -----     | -----          | -----           | -----     |
-| [CMT](https://github.com/junjie18/CMT)                          | 0.729     | **18.642**   | 1.821        | -0.096     | **9.398** | **2.041**      | -0.841        | **8.213**     | **9.887** | **17.053**     | **8.448**       | **7.457** |
-| [Sparsefusion](https://github.com/yichen928/SparseFusion)       | **0.732** | 4.264        | **2.913**    | **1.821**  | 4.429     | 0.297          | **0.280**     | 3.242         | 1.887     | 3.699          | 7.228           | 3.006     |
-| [TransFusion](https://github.com/XuyangBai/TransFusion)         | 0.708     | -7.210       | 0.270        | 1.146      | -0.552    | 0.340          | -5.412        | -3.296        | -4.220    | -3.626         | 3.850           | -1.871    |
-| [DeepInteraction](https://github.com/fudan-zvg/DeepInteraction) | 0.691     | -6.361       | -3.551       | -7.215     | -25.037   | -16.386        | -7.077        | -2.188        | -5.149    | 0.212          | 0.145           | -7.261    |
+| [BEVfusion](https://github.com/mit-han-lab/bevfusion)           | 0.714     | ---        | ---        | ---      | ---     | ---          | ---         | ---         | ---     | ---          | ---           | ---     |
+| [CMT](https://github.com/junjie18/CMT)                          | 0.729     | **18.642**   | -1.138       | -0.096     | **9.398** | **2.041**      | -0.841        | **8.213**     | **9.887** | **17.053**     | **8.448**       | **7.161** |
+| [Sparsefusion](https://github.com/yichen928/SparseFusion)       | **0.732** | 4.264        | **3.179**    | **1.821**  | 4.429     | 0.297          | **0.280**     | 3.242         | 1.887     | 3.699          | 7.228           | 3.033     |
+| [TransFusion](https://github.com/XuyangBai/TransFusion)         | 0.708     | -7.210       | 1.799        | 1.146      | -0.552    | 0.340          | -5.412        | -3.296        | -4.220    | -3.626         | 3.850           | -1.718    |
+| [DeepInteraction](https://github.com/fudan-zvg/DeepInteraction) | 0.691     | -6.361       | -3.150       | -7.215     | -25.037   | -16.386        | -7.077        | -2.188        | -5.149    | 0.212          | 0.145           | -7.221    |
 
 ## Metrics
 We adhere to the official nuScenes metric definition for computing the NDS and mAP metrics on the MultiCorrupt dataset. To quantitatively compare the performance between the corrupted dataset and the clean nuScenes datasets, we use a metric called the *Resistance Ability* (RA). This metric is calculated across the different severity levels with 

diff --git a/assets/brightness_1_front_camera_scene-0097_scene_animation.gif b/assets/brightness_1_front_camera_scene-0097_scene_animation.gif
diff --git a/assets/brightness_1_multi_view_camera_scene-0097_scene_animation.gif b/assets/brightness_1_multi_view_camera_scene-0097_scene_animation.gif
diff --git a/assets/brightness_2_front_camera_scene-0097_scene_animation.gif b/assets/brightness_2_front_camera_scene-0097_scene_animation.gif
diff --git a/assets/brightness_2_multi_view_camera_scene-0097_scene_animation.gif b/assets/brightness_2_multi_view_camera_scene-0097_scene_animation.gif
diff --git a/assets/brightness_3_front_camera_scene-0097_scene_animation.gif b/assets/brightness_3_front_camera_scene-0097_scene_animation.gif
diff --git a/assets/brightness_3_multi_view_camera_scene-0097_scene_animation.gif b/assets/brightness_3_multi_view_camera_scene-0097_scene_animation.gif
diff --git a/evaluation/evaluations_nds.csv b/evaluation/evaluations_nds.csv
@@ -1,7 +1,7 @@
 model,clean,temporalmisalignment_s1,temporalmisalignment_s2,temporalmisalignment_s3,spatialmisalignment_s1,spatialmisalignment_s2,spatialmisalignment_s3,pointsreducing_s1,pointsreducing_s2,pointsreducing_s3,motionblur_s1,motionblur_s2,motionblur_s3,missingcamera_s1,missingcamera_s2,missingcamera_s3,fog_s1,fog_s2,fog_s3,dark_s1,dark_s2,dark_s3,brightness_s1,brightness_s2,brightness_s3,beamsreducing_s1,beamsreducing_s2,beamsreducing_s3,snow_s1,snow_s2,snow_s3
-CMT,0.729,0.6572,0.568,0.4988,0.6462,0.586,0.5367,0.7012,0.6834,0.638,0.6886,0.6154,0.5357,0.718,0.7109,0.7009,0.6974,0.6527,0.4134,0.7051,0.6904,0.6786,0.7284,0.7212,0.7037,0.6826,0.5873,0.4497,0.6483,0.619,0.5554
-DeepInteraction,0.6909,0.6139,0.5237,0.4544,0.5863,0.5014,0.4267,0.6522,0.6242,0.5518,0.655,0.5751,0.4939,0.6253,0.5816,0.5383,0.5788,0.4844,0.1452,0.6594,0.6415,0.6254,0.6845,0.6767,0.6785,0.6162,0.4571,0.2839,0.5526,0.5311,0.4896
-TransFusion,0.7084,0.6306,0.5437,0.4766,0.5708,0.4761,0.4095,0.6554,0.619,0.5331,0.6727,0.5878,0.4944,0.7017,0.698,0.6946,0.6506,0.5996,0.3529,0.7025,0.6996,0.6978,0.708,0.7069,0.7056,0.6222,0.4591,0.2636,0.5447,0.5353,0.5087
-Sparsefusion,0.7315,0.6523,0.5622,0.4901,0.6061,0.5149,0.4461,0.6881,0.6578,0.5838,0.6997,0.6234,0.5374,0.7117,0.6981,0.6836,0.6762,0.6261,0.3811,0.7122,0.7039,0.6978,0.7302,0.7257,0.7205,0.6593,0.5139,0.338,0.5859,0.5704,0.5337
-BEVfusion,0.7144,0.6341,0.5471,0.4085,0.5903,0.4943,0.4266,0.6684,0.6379,0.5628,0.6861,0.6217,0.5475,0.7038,0.6961,0.6873,0.6555,0.5981,0.3584,0.699,0.691,0.6861,0.7126,0.7053,0.6969,0.6396,0.4899,0.3199,0.5782,0.5572,0.5233
+CMT,0.729,0.6572,0.568,0.4988,0.6462,0.586,0.5367,0.7012,0.6834,0.638,0.6886,0.6154,0.5357,0.718,0.7109,0.7009,0.6974,0.6527,0.4134,0.7051,0.6904,0.6786,0.7038,0.6852,0.6605,0.6826,0.5873,0.4497,0.6483,0.619,0.5554
+DeepInteraction,0.6909,0.6139,0.5237,0.4544,0.5863,0.5014,0.4267,0.6522,0.6242,0.5518,0.655,0.5751,0.4939,0.6253,0.5816,0.5383,0.5788,0.4844,0.1452,0.6594,0.6415,0.6254,0.6785,0.6719,0.6574,0.6162,0.4571,0.2839,0.5526,0.5311,0.4896
+TransFusion,0.7084,0.6306,0.5437,0.4766,0.5708,0.4761,0.4095,0.6554,0.619,0.5331,0.6727,0.5878,0.4944,0.7017,0.698,0.6946,0.6506,0.5996,0.3529,0.7025,0.6996,0.6978,0.7056,0.7041,0.7007,0.6222,0.4591,0.2636,0.5447,0.5353,0.5087
+Sparsefusion,0.7315,0.6523,0.5622,0.4901,0.6061,0.5149,0.4461,0.6881,0.6578,0.5838,0.6997,0.6234,0.5374,0.7117,0.6981,0.6836,0.6762,0.6261,0.3811,0.7122,0.7039,0.6978,0.7205,0.7146,0.7039,0.6593,0.5139,0.338,0.5859,0.5704,0.5337
+BEVfusion,0.7144,0.6341,0.5471,0.4085,0.5903,0.4943,0.4266,0.6684,0.6379,0.5628,0.6861,0.6217,0.5475,0.7038,0.6961,0.6873,0.6555,0.5981,0.3584,0.699,0.691,0.6861,0.6967,0.6913,0.6851,0.6396,0.4899,0.3199,0.5782,0.5572,0.5233
 Our,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
diff --git a/evaluation/ra_results.csv b/evaluation/ra_results.csv
@@ -1,6 +1,6 @@
 model,RA_beamsreducing_s1,RA_brightness_s1,RA_dark_s1,RA_fog_s1,RA_missingcamera_s1,RA_motionblur_s1,RA_pointsreducing_s1,RA_snow_s1,RA_spatialmisalignment_s1,RA_temporalmisalignment_s1,RA_beamsreducing_s2,RA_brightness_s2,RA_dark_s2,RA_fog_s2,RA_missingcamera_s2,RA_motionblur_s2,RA_pointsreducing_s2,RA_snow_s2,RA_spatialmisalignment_s2,RA_temporalmisalignment_s2,RA_beamsreducing_s3,RA_brightness_s3,RA_dark_s3,RA_fog_s3,RA_missingcamera_s3,RA_motionblur_s3,RA_pointsreducing_s3,RA_snow_s3,RA_spatialmisalignment_s3,RA_temporalmisalignment_s3,mRA
-CMT,0.9363511659807956,0.9991769547325103,0.9672153635116597,0.9566529492455419,0.9849108367626886,0.9445816186556928,0.9618655692729767,0.8893004115226337,0.8864197530864197,0.9015089163237312,0.8056241426611798,0.9893004115226337,0.9470507544581619,0.8953360768175582,0.9751714677640604,0.8441700960219478,0.9374485596707819,0.8491083676268861,0.803840877914952,0.7791495198902606,0.6168724279835391,0.9652949245541839,0.9308641975308642,0.5670781893004115,0.9614540466392318,0.7348422496570645,0.8751714677640604,0.7618655692729767,0.7362139917695473,0.6842249657064472,0.8696021947873801
-DeepInteraction,0.8918801563178463,0.9907367202200029,0.9544072948328268,0.8377478651034882,0.9050513822550297,0.9480387899840789,0.9439861050803301,0.799826313504125,0.8486032710956725,0.8885511651469099,0.6616008105369808,0.9794470979881315,0.9284990591981473,0.701114488348531,0.841800550007237,0.8323925314806774,0.9034592560428427,0.7687074829931974,0.7257200752641482,0.7579968157475757,0.41091330149080907,0.9820523954262557,0.9051961210015921,0.21016066000868433,0.7791286727456941,0.7148646692719641,0.7986684035316254,0.7086409031697786,0.6176002315819946,0.6576928643797946,0.7964828484585323
-TransFusion,0.8783173348390739,0.9994353472614341,0.9916713721061547,0.9184076792772444,0.9905420666290231,0.9496047430830039,0.9251835121400338,0.7689158667419536,0.8057594579333709,0.8901750423489554,0.6480801806888763,0.9978825522303783,0.9875776397515528,0.8464144551101073,0.9853190287972896,0.8297571993224166,0.8738001129305477,0.7556465273856577,0.672077922077922,0.7675042348955392,0.37210615471485037,0.9960474308300394,0.9850367024280067,0.4981648785996612,0.9805194805194805,0.6979107848673066,0.752540937323546,0.7180971202710333,0.5780632411067194,0.6727837380011293,0.8244447581404103
-Sparsefusion,0.9012987012987013,0.998222829801777,0.9736158578263842,0.9244019138755981,0.9729323308270676,0.9565276828434722,0.9406698564593301,0.800956937799043,0.8285714285714285,0.8917293233082706,0.7025290498974709,0.9920710868079289,0.9622693096377306,0.8559125085440874,0.9543403964456596,0.8522214627477784,0.8992481203007519,0.7797676008202323,0.7038961038961039,0.7685577580314422,0.46206425153793573,0.9849624060150376,0.9539302802460696,0.5209842788790157,0.9345181134654819,0.7346548188653451,0.7980861244019138,0.7295967190704031,0.6098427887901572,0.6699931647300068,0.8352791068580542
-BEVfusion,0.8952967525195967,0.9974804031354982,0.9784434490481522,0.9175531914893617,0.9851623740201567,0.9603863381858903,0.9356103023516237,0.8093505039193729,0.8262877939529676,0.8875979843225084,0.6857502799552071,0.9872620380739082,0.967245240761478,0.8372060470324747,0.9743840985442329,0.8702407614781635,0.8929171332586786,0.7799552071668533,0.691909294512878,0.7658174692049272,0.44778835386338184,0.9755039193729003,0.9603863381858903,0.5016797312430011,0.9620660694288914,0.7663773796192609,0.7877939529675251,0.7325027995520716,0.597144456886898,0.5718085106382979,0.831630272489735
+CMT,0.9363511659807956,0.9654320987654321,0.9672153635116597,0.9566529492455419,0.9849108367626886,0.9445816186556928,0.9618655692729767,0.8893004115226337,0.8864197530864197,0.9015089163237312,0.8056241426611798,0.9399176954732511,0.9470507544581619,0.8953360768175582,0.9751714677640604,0.8441700960219478,0.9374485596707819,0.8491083676268861,0.803840877914952,0.7791495198902606,0.6168724279835391,0.9060356652949245,0.9308641975308642,0.5670781893004115,0.9614540466392318,0.7348422496570645,0.8751714677640604,0.7618655692729767,0.7362139917695473,0.6842249657064472,0.8648559670781892
+DeepInteraction,0.8918801563178463,0.9820523954262557,0.9544072948328268,0.8377478651034882,0.9050513822550297,0.9480387899840789,0.9439861050803301,0.799826313504125,0.8486032710956725,0.8885511651469099,0.6616008105369808,0.9724996381531338,0.9284990591981473,0.701114488348531,0.841800550007237,0.8323925314806774,0.9034592560428427,0.7687074829931974,0.7257200752641482,0.7579968157475757,0.41091330149080907,0.9515125199015777,0.9051961210015921,0.21016066000868433,0.7791286727456941,0.7148646692719641,0.7986684035316254,0.7086409031697786,0.6176002315819946,0.6576928643797946,0.7949437931200849
+TransFusion,0.8783173348390739,0.9960474308300394,0.9916713721061547,0.9184076792772444,0.9905420666290231,0.9496047430830039,0.9251835121400338,0.7689158667419536,0.8057594579333709,0.8901750423489554,0.6480801806888763,0.9939299830604177,0.9875776397515528,0.8464144551101073,0.9853190287972896,0.8297571993224166,0.8738001129305477,0.7556465273856577,0.672077922077922,0.7675042348955392,0.37210615471485037,0.9891304347826086,0.9850367024280067,0.4981648785996612,0.9805194805194805,0.6979107848673066,0.752540937323546,0.7180971202710333,0.5780632411067194,0.6727837380011293,0.8239695087521175
+Sparsefusion,0.9012987012987013,0.9849624060150376,0.9736158578263842,0.9244019138755981,0.9729323308270676,0.9565276828434722,0.9406698564593301,0.800956937799043,0.8285714285714285,0.8917293233082706,0.7025290498974709,0.9768967874231032,0.9622693096377306,0.8559125085440874,0.9543403964456596,0.8522214627477784,0.8992481203007519,0.7797676008202323,0.7038961038961039,0.7685577580314422,0.46206425153793573,0.9622693096377306,0.9539302802460696,0.5209842788790157,0.9345181134654819,0.7346548188653451,0.7980861244019138,0.7295967190704031,0.6098427887901572,0.6699931647300068,0.8335748462064251
+BEVfusion,0.8952967525195967,0.9752239641657334,0.9784434490481522,0.9175531914893617,0.9851623740201567,0.9603863381858903,0.9356103023516237,0.8093505039193729,0.8262877939529676,0.8875979843225084,0.6857502799552071,0.9676651735722285,0.967245240761478,0.8372060470324747,0.9743840985442329,0.8702407614781635,0.8929171332586786,0.7799552071668533,0.691909294512878,0.7658174692049272,0.44778835386338184,0.958986562150056,0.9603863381858903,0.5016797312430011,0.9620660694288914,0.7663773796192609,0.7877939529675251,0.7325027995520716,0.597144456886898,0.5718085106382979,0.8296845837999254
diff --git a/evaluation/ra_results_brightness.csv b/evaluation/ra_results_brightness.csv
@@ -1,4 +1,4 @@
 CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion
-0.9991769547325103,0.9907367202200029,0.9994353472614341,0.998222829801777,0.9974804031354982
-0.9893004115226337,0.9794470979881315,0.9978825522303783,0.9920710868079289,0.9872620380739082
-0.9652949245541839,0.9820523954262557,0.9960474308300394,0.9849624060150376,0.9755039193729003
+0.9654320987654321,0.9820523954262557,0.9960474308300394,0.9849624060150376,0.9752239641657334
+0.9399176954732511,0.9724996381531338,0.9939299830604177,0.9768967874231032,0.9676651735722285
+0.9060356652949245,0.9515125199015777,0.9891304347826086,0.9622693096377306,0.958986562150056
diff --git a/evaluation/rra_nds_results.csv b/evaluation/rra_nds_results.csv
@@ -1,7 +1,7 @@
 model,CMT,DeepInteraction,TransFusion,Sparsefusion,BEVfusion
 clean,0.729,0.6909,0.7084,0.7315,0.7144
 RRA_beamsreducing,18.64219677107768,-6.3612529322478295,-7.209879950324272,4.263833310335308,0.0
-RRA_brightness,1.8205031208624822,-3.551163230565535,0.269529033478344,2.9128049933800026,0.0
+RRA_brightness,-1.1383917804254517,-3.1498721721094025,1.7992378563503664,3.1788143360185206,0.0
 RRA_dark,-0.09633447329125966,-7.215452049515914,1.1463802321660888,1.8207215452049619,0.0
 RRA_fog,9.398263027295272,-25.03722084367247,-0.5521091811414469,4.429280397022328,0.0
 RRA_missingcamera,2.041011881947097,-16.38558834802607,0.3401686469911791,0.2970486776542547,0.0