Reviewer 3 comments

[labarba]

Recommendation: Accept If Certain Minor Revisions Are Made

Comments:
... below is my review of the paper "Reproducible and
replicable CFD: It's harder than you think". This is a nice paper, and
will be a good contribution to the literature. I have the following
remarks for revising the manuscript:

• 1. A little more introduction to the physics in nature, setup of the
  simulation methodology, and findings of the main problem (lift of a
  flying snake) should be discussed, instead of just referencing the
  paper of the study

• 2. There should be some discussion about other efforts to compare
  different codes on the same problem. Some of the ones that come to
  mind are:

  • The Santa Barbara Cluster Comparison Project:
    https://arxiv.org/abs/astro-ph/9906160
  • The Alpha Group Collaboration:
    http://scitation.aip.org/content/aip/journal/pof2/16/5/10.1063/1.1688328
  • Disk-planet interaction comparisons:
    http://www.tat.physik.uni-tuebingen.de/~kley/publ/paper/valborro.pdf

  Although many of these are compressible (but not all, e.g. for the
  alpha group), they share the level of differences and in-depth
  knowledge of the code needed as reported in this paper.

• 3. In "Story 1", the authors use the term "blow up" -- this should be
  clarified

• 4. In "Story 1", the authors devote a few paragraphs discussing mesh
  quality -- figures showing these meshes need to be included in the
  paper -- it will really help the discussion.

• 5. "advective" boundary conditions should be defined -- what is the
  mathematical definition of this?

• 6. In this discussion of the hardware changes from 3 years ago to
  present, it was noted that the GPUs changed -- were they all doing
  double precision floating point? A note about this should be
  added.

• 7. In the final paragraph, the authors advocate for continued release
  of CFD codes, and list some reason for not sharing. An important
  one that readers should be aware of is export control (especially
  for radiation hydrodynamics codes).

• 8. The authors discuss issues with the the software stack changing
  from three years ago to present. There are some efforts to help
  capture the state of the machine (OS, compilers, libraries, etc.)
  using Docker containers. The authors should comment on these efforts
  and whether they would have helped with their workflow.

Additional Questions:

• How relevant is this manuscript to the readers of this periodical? Please explain your rating in the Detailed Comments section. : Very Relevant
• To what extent is this manuscript relevant to readers around the world? : The manuscript is of interest to readers throughout the world

1) Please summarize what you view as the key point(s) of the manuscript and the importance of the content to the readers of this periodical. : This provides a nice case study on how challenging reproducible research is to investigators, and will be common to all the readers.

2) Is the manuscript technically sound? Please explain your answer in the Detailed Comments section. : Yes

3) What do you see as this manuscript's contribution to the literature in this field?: This is a broad description of the problems one encounters with reproducibility.

4) What do you see as the strongest aspect of this manuscript?: The lengthy personal experience that the authors convey.

5) What do you see as the weakest aspect of this manuscript?: It would benefit from discussion on other attempts on reproducibility in CFD reported in the field.

1. Does the manuscript contain title, abstract, and/or keywords?: Yes
2. Are the title, abstract, and keywords appropriate? Please elaborate in the Detailed Comments section.: Yes
3. Does the manuscript contain sufficient and appropriate references (maximum 12-unless the article is a survey or tutorial in scope)? Please elaborate in the Detailed Comments section.: References are sufficient and appropriate
4. Does the introduction clearly state a valid thesis? Please explain your answer in the Detailed Comments section.: Yes
5. How would you rate the organization of the manuscript? Please elaborate in the Detailed Comments section.: Could be improved
6. Is the manuscript focused? Please elaborate in the Detailed Comments section.: Satisfactory
7. Is the length of the manuscript appropriate for the topic? Please elaborate in the Detailed Comments section.: Satisfactory
8. Please rate and comment on the readability of this manuscript in the Detailed Comments section.: Easy to read
9. Please rate and comment on the timeliness and long term interest of this manuscript to CiSE readers in the Detailed Comments section. Select all that apply.: Topic and content are of immediate and continuing interest to CiSE readers

Please rate the manuscript. Explain your choice in the Detailed Comments section.: Excellent

[mesnardo]

Addressing reviewer comment # 2:

There should be some discussion about other efforts to compare
different codes on the same problem.

In the late 90's, a project initiated at the Institute for Theoretical Physics of the University of California, Santa Barbara addressed the reliability of current methods (at that time) used in cosmological gas dynamics to assess the reproducibility of numerical studies (Frenk et al., 1999). The group of researchers compared the results from twelve different codes of the formation of a cluster in cold dark matter universe.
They achieved a successful full-replication of the study:
"Although there is, of course, no guarantee that any of the calculations gives the correct solution to the problem, the agreement among the various simulations was better than a pessimist might have predicted."

In the field of compressible flows, Dimonte et al. (2004) performed a comparative study of the turbulent Rayleigh-Taylor instability using seven different codes to justify the use of a computationally-intensive approach (interface reconstruction) but believed to reduce the overestimation a certain quantity.
"Finally, if anyone is interested in repeating these calculations with their own numerical methods, please contact [email protected] or any of the other authors for an electronic form of the initial conditions."

In the field of astrophysical fluids, de Val-Borro et al. (2006) tested the reliability of current astrophysical hydrodynamic codes using a disc-planet interaction problem. They compared the solution from various grid-based and smooth particle hydrodynamics codes (a total of 17 codes was used). The good agreement between the different codes permitted to provide a reliable reference for future calculations.
"In closing, we would like to reiterate that computational work might be regarded as an experiment, rather than a simulation. We have shown that different codes can give slightly different results for the same physical problem. Reproducibility of experimental results is fundamental to the scientific process, and this standard must be applied to those performed with computers. Before a computational result can be regarded as reliable, it must be confirmed by an independent test with a different code."

References:

• Frenk, C. S., White, S. D. M., Bode, P., Bond, J. R., Bryan, G. L., Cen, R., ... & Khokhlov, A. M. (1999). The Santa Barbara cluster comparison project: a comparison of cosmological hydrodynamics solutions. The Astrophysical Journal, 525(2), 554.
• Dimonte, G., Youngs, D. L., Dimits, A., Weber, S., Marinak, M., Wunsch, S., ... & Calder, A. C. (2004). A comparative study of the turbulent Rayleigh–Taylor instability using high-resolution three-dimensional numerical simulations: the Alpha-Group collaboration. Physics of Fluids (1994-present), 16(5), 1668-1693.
• De Val-Borro, M., Edgar, R. G., Artymowicz, P., Ciecielag, P., Cresswell, P., D'Angelo, G., ... & Klahr, H. (2006). A comparative study of disc–planet interaction. Monthly Notices of the Royal Astronomical Society, 370(2), 529-558.

---

[UPDATES]

• Added comments of previous efforts to compare multiple CFD codes: see change history: commit 1, commit 2, and commit 3.

[mesnardo]

Addressing reviewer comment # 3:

In "Story 1", the authors use the term "blow up" -- this should be
clarified

The term "blow up" used in Story 1 of the manuscript means a degeneration of the numerical solution leading to a crash (sudden halt) of the simulation.

---

[UPDATES]

• Clarify the term "blow up". See change history.

[mesnardo]

Addressing reviewer comment # 4:

In "Story 1", the authors devote a few paragraphs discussing mesh
quality -- figures showing these meshes need to be included in the
paper -- it will really help the discussion.

The OpenFOAM package provides the utility checkMesh to check the validity of a mesh (non-orthogonality, maximum skewness, maximum aspect-ration, ...) before running the solver.

Figure 1 of the manuscript shows the vorticity field on a triangular mesh generated with the software GMSH.
The two-dimensional mesh contains about 700k cells.
We observed the degeneration of the numerical solution at random spots (characterized by the apparition of vorticity in Figure 1) in the domain where a freestream flow was expected.
We attributed this phenomenon to the poor quality of the mesh.
As a workaround, we switched to another meshing tool, snappyHexMesh, provided by OpenFOAM.

The two figures below show the meshes in the vicinity of the leading edge of the snake cross-section, on which the numerical solution, reported in the manuscript, has been computed.

• Mesh generated with GMSH:
  Figure added to the Reproducibility Package for results with GMSH and Re=2000 and AoA=35deg.

• Mesh generated with snappyHexMesh:
  Figure added to the Reproducibility Package for results with snappyHexMesh Re=2000 and AoA=35deg

The figure below shows the vorticity field as well as the shape of the cells at the location of the non-physical vorticity spot reported in Figure 1 of the manuscript.

I propose to include the outputs of the utility checkMesh in the corresponding folders of the reproducibility packages.

---

[UPDATES]

• Add note about mesh plots added to supplementary materials. See change history.

[mesnardo]

Addressing reviewer comment # 5:

"advective" boundary conditions should be defined -- what is the
mathematical definition of this?

The advective boundary condition implemented in OpenFOAM provides an advective outflow condition, solving an advection equation at the boundary.
The default implementation in OpenFOAM uses the value of the local flux as the advection speed.

---

[UPDATES]

• Explain the definition of an advective boundary condition. See change history.

[mesnardo]

Addressing reviewer comment # 6:

In this discussion of the hardware changes from 3 years ago to
present, it was noted that the GPUs changed -- were they all doing
double precision floating point? A note about this should be
added.

Krishnan et al. (2014) used a NVidia Tesla C2070 GPU to compute the numerical solution with cuIBM.
The Tesla C2070 is based on the CUDA architecture Fermi.
We used a NVidia Tesla K20 to run cuIBM.
Both GPUs were used with double precision floating point.

---

[UPDATES]

• Add note about double precision. See change history.

[mesnardo]

Addressing reviewer comment # 8:

The authors discuss issues with the the software stack changing
from three years ago to present. There are some efforts to help
capture the state of the machine (OS, compilers, libraries, etc.)
using Docker containers. The authors should comment on these efforts
and whether they would have helped with their workflow.

Our in-house software, cuIBM and PetIBM, use GitHub's wiki or README markdown files to guide the user during the building stage.
Software documentation can be imprecise or incomplete, making it difficult for the users to build and use an application and the courageous ones to reproduce a computational study.
Docker appears as an attractive open-source tool to adopt a reproducible workflow (see Boettiger, 2015, and Chamberlain and Schommer, 2014).
Docker allows a developer to deploy an application (and all its dependencies) in a container, a lightweight virtual machine--a container does not create a whole virtual operating system but used the Linux kernel on the system it is running on.
Thus, Docker ensures that a software will run the same way on any Linux machine.
It would have been easier for us to reproduce the results from our previous study if a Docker container had been created in first instance, ensuring that the computational environment would be the identical to the one used during the previous study. But this tool was not available to us at that time.

References:

• Boettiger, C. (2015). An introduction to Docker for reproducible research. ACM SIGOPS Operating Systems Review, 49(1), 71-79.
• Chamberlain, R., & Schommer, J. (2014). Using Docker to support reproducible research. DOI: http://dx. doi. org/10.6084/m9. figshare, 1101910.

---

[UPDATES]

• Add mention of Docker containers. See change history. Note that Docker containers were not available when the first study was in progress.

[labarba]

Addressing Comment # 1:

• Addition to the introduction partially addresses this referee comment. See change history.
• Findings of the previous study (Krishnan et al. 2014) are discussed briefly in the introduction. (Sentence beginning "The main finding of our study on wakes of flying snakes was ... "). We added a citation to the experimental study that revealed the enhanced lift (see change history) and a sentence about the mechanism behind it (see change history).
• The manuscript is already past the maximum word limit, and so we must count on the interested reader consulting the original study for details of the simulation methodology and the explanation of the lift-enhancement mechanism. We think that the main points of this paper can be followed without these details about the previous study.