-
Notifications
You must be signed in to change notification settings - Fork 120
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extend benchmarks with computation of scores for NR and DB metrics #270
Conversation
Signed-off-by: Sergey Kastryulin <[email protected]>
Codecov Report
@@ Coverage Diff @@
## master #270 +/- ##
=======================================
Coverage 94.04% 94.04%
=======================================
Files 34 34
Lines 2485 2485
=======================================
Hits 2337 2337
Misses 148 148
Flags with carried forward coverage won't be shown. Click here to find out more. |
Ready for review. |
Kudos, SonarCloud Quality Gate passed! 0 Bugs No Coverage information |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the implementation, thanks @snk4tr
Let's add benchmarks results into README table and also some explanation on how values for distribution-based metrics are computed.
Signed-off-by: Sergey Kastryulin <[email protected]>
@zakajd about including results on PIPAL to the table: the paper on the PIPAL dataset has the following table: The problem with this table is that there is no single column for evaluation of metrics on all types of artefacts. Hence, we need to either
At this point in time I prefer the first option but the the question of which distortion type to choose arises. Personally, I think that so called "Traditional distortions" (Gaussian noise, blur etc.) is the most logical choice but I would like to know what you think. |
I also think that we should not add values for distribution-based metrics in the table even though the benchmarking tool lets us to compute them. The reason is that the main purpose of the table in the README file is to provide a reference on how close our implementations of metrics are to the original ones. At this point in time, there are no reference values for distribution-based metrics, which means that there is nothing to compare with. |
@snk4tr PIPAL paper is a bit confusing. They actually do evaluate metrics on all distortions (see Fig. 4), but don’t publish numbers. In their next work with SWDN metric (https://arxiv.org/pdf/2011.15002.pdf) they post some results in Table 5. So we can
|
@zakajd I sent a request to provide the data. I see that values from the SWDN metric paper are already there. I will add our values for all metrics in this PR. Hopefully the authors will kindly provide us the data soon. |
Signed-off-by: Sergey Kastryulin <[email protected]>
Kudos, SonarCloud Quality Gate passed! 0 Bugs No Coverage information |
Kudos, SonarCloud Quality Gate passed! 0 Bugs No Coverage information |
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work. Just some minor changes.
Line 200: I would change it to "Feature Based"
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
f8c461d
to
740a040
Compare
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
Signed-off-by: Sergey Kastryulin <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, just a minor change to make documentation consistent
Signed-off-by: Sergey Kastryulin <[email protected]>
Kudos, SonarCloud Quality Gate passed! 0 Bugs No Coverage information |
Closes #265
Proposed Changes