Reviewer #2

[ncclementi]

Reviewer 2 comments

It was a pleasure to review the manuscript “Reproducible Validation and Replication Studies in Nanoscale Physics” by Clementi and Barba. This paper presents validation and replication studies in area of nanostructure responses to electromagnetic waves. The authors use their previously developed open-source software PyGBe to replicate results from previous publications. The authors make their reproducibility packages openly accessible, including the computational software, input files, execution scripts, analysis code, data and figures. This is essential for open science and for the feasibility of replication study. The manuscript addresses the important topic of reproducibility and replication. Its well-written background section makes the paper not only of interest to this specific field, but also to a general audience. I recommend to publish the manuscript after some minor revisions:

Although the main purpose of the paper is replication, the other closely related topics should also be addressed. This includes:

• Although the authors released their reproducibility packages, they don’t show the reproducibility of their calculations: can consistent results be generated when the calculations are repeated, how sensitive the results are to the input data, etc.

• What are the uncertainties of the calculations? Can error bars be added to the results as displayed in the figures and tables?

• In the “replication of results from Rockstuhl, et al., 2005”, it appears that the main reason for the differences is the 2D and 3D representation between the previous and current studies. The authors show the impact of the y dimensions to the calculated results, but only use two lengths. Are there any limitations to to stop the authors to lengthen y larger than 2688 nm? Will some peaks disappear if the y dimension approaches to infinity?

Typos:

• Page 5, line 30: PyGBE --> PyGBe
• Page 18, line 18, “… he relationship …” --> “ … the relationship …”

[ncclementi]

PyGBe typo fixed in commit ccab590
We also included different writings of PyGBe across the manuscript to always use /pygbe
The other typo was also pointed by reviewer number one, changes were already made.

[ncclementi]

Although the authors released their reproducibility packages, they don’t show the reproducibility of their calculations: can consistent results be generated when the calculations are repeated, how sensitive the results are to the input data, etc.

I'm a bit confused, is the reviewer asking to reproduce our own results?

• can consistent results be generated when the calculations are repeated? Yes.
• how sensitive the results are to the input data? We always get the same results.

What are the uncertainties of the calculations? Can error bars be added to the results as displayed in the figures and tables?

I'm not sure how to answer this, if we run the simulations multiple times, we always get the same results. I don't think we can add error bars.

In the “replication of results from Rockstuhl, et al., 2005”, it appears that the main reason for the differences is the 2D and 3D representation between the previous and current studies. The authors show the impact of the y dimensions to the calculated results, but only use two lengths. Are there any limitations to to stop the authors to lengthen y larger than 2688 nm? Will some peaks disappear if the y dimension approaches to infinity?

By using 2688 nm we are close to the limit were we can use quasistatic limit (wavelength > size of nanoparticle). Our wavelengths go from 10-12 microns (10000 - 12000 nm) if we make y longer this can be a problem since the model might not be valid anymore. Also we need to add the effect of absorption domination.

Added a sentence to explain the reason why we do not use longer y, included in commit 975ee26

[labarba]

Quantifiable Uncertainties / Errors:

• discretization error
• algebraic iterative errors

Unknown uncertainties:

• dielectric constant of the material

Running the same calculation with the same mesh on the same machine gives bit-by-bit the same output.

Error bars would not be appropriate to represent these kinds of uncertainties, which are not statistical variations in the results.

[labarba]

Small fix: be9e36b

[labarba]

Response

Although the authors released their reproducibility packages, they don’t show the reproducibility of their calculations: can consistent results be generated when the calculations are repeated, how sensitive the results are to the input data, etc.
What are the uncertainties of the calculations? Can error bars be added to the results as displayed in the figures and tables?

Running the execution scripts with the same input file on the same machine will produce the same results with bit-by-bit reproducibility. If running on a different machine, the results could show a difference due to floating-point error. We did confirm this. In fact, we have manually triggered correctness tests on the software that we run each time a change is applied in the code, which compare to previously generated results; we set these tests to pass when the difference in the results match to 10^{-10}.

The uncertainties in the calculations are: model uncertainties (quasi static approximation), spatial discretization errors (quantified via grid-convergence analysis in a previous paper that we cite in the manuscript), algebraic errors (from the iterative linear solver, set via exit tolerance), tree code approximation (quantifiable and controllable via expansion order and tree level). Additionally, the experimentally obtained value of the dielectric constant of the material is also subject to uncertainties (unknown to us). Since none of these sources of uncertainty is of statistical nature, error bars are not appropriate.