subdirectory of examples that emphasize published results

[carltape]

At the Toronto meeting, we discussed the multiple uses for examples in specfem. Most examples attempt to demonstrate one more more capabilities, and they do it in a minimal way (small mesh, few time steps), so that the example can be automatically tested. There is also a need for a second set of examples that reproduce findings that are published in papers. A recent inclusion was the Liu and Tromp (2006) kernels, which @aakash10gupta added. This is NOT the kind of example you'd want to test every time a change is made to the code. Nevertheless, we want to encourage examples along these lines, especially if there are minimal-sized input files needed for the examples.

To address this, I was thinking we should make a subdirectory in EXAMPLES called something like PUBLISHED (or the like) that would include sensitivity_kernels_liutromp2006, for starters. (Are there others to move there? What about some of the original specfem benchmarks?)

How does this all sound? And is there any other category of examples that should be established at this time?

A bigger topic is whether each of the examples serves a useful purpose, or whether we need others or could cut others. Let's deal with that separately.

[gassmoeller]

I agree that not all examples are the same and you want to let users know about their purpose/history by putting them in different folders. That being said, I am not sure examples vs published captures the essence of the difference you want to point out. It seems like you want to separate "educational examples" on the one hand from "computational benchmarks" or "research applications" on the other, with the additional distinction of models that are quick to run as tests, and models that require significant computational resources.
Here is how some other CIG codes are handling this:
geodynamics/Rayleigh separates into:

• examples: shows how to use features, including documentation
• input_examples: benchmark setups, potentially large, including documentation
• tests: small, regularly run as part of CI, no documentation

geodynamics/citcom doesnt really distinguish, and has 10 documented cookbooks, and a few undocumented input files as examples.

geodynamics/pylith separates into:

• examples: educational examples for teaching, moderately small
• tests / fullscale: integration tests (benchmarks)
• tests / other: smaller tests without documentation

geodynamics/aspect separates into:

• benchmarks (dozens): models showing correctness of something; often published in some paper, but not all; small or large; requires small documentation, but not much; most run before release
• cookbooks (20 or so): educational examples of features, mostly small enough to be run on a workstation (not all), heavily documented, not necessarily published; most run before release
• tests (1000): small enough to be run in CI, contains some smaller versions of cookbooks and benchmarks, almost no documentation (1 sentence about the purpose), but sometimes useful to see how things work; all run constantly

I like the distinction into benchmarks, cookbooks, and tests, but I am of course biased :-). Different ways can work for specfem and the decision is of course up to you.

[carltape]

Thanks, @gassmoeller, this is great. The use of benchmarks, cookbooks, and tests sounds great to me. @danielpeter, do you know the examples well enough to assign them to these three (or guess)? The sensitivity_kernels_liutromp2006 would be benchmark. Presumably also BENCHMARK_CLAERBOUT_ADJOINT. For the rest, someone could guess the subfolder assignment. Not sure if we have many tests or if they are mostly cookbooks. Also, some renaming of folders is probably also needed. These changes would set us up better for future examples.

[williameaton]

Hi @carltape , @gassmoeller,

We had a discussion today about next steps for the Training section of the website.

It has been raised that in addition to the raw documentation for each code in the SPECFEM family, we need the website to link to a large suite of examples and tutorials that help new users. Splitting into benchmarks and cookbooks sounds like a good idea for this.

Can we compile a list of desired cookbooks each with an aim for what they teach the user? We can then designate/ask the community to contribute a cookbook/example. If we do this, what do you think about creating a broad outline/template that each cookbook should follow?

I think there needs to be some consistency across examples...for example what is the minimum expected details included in the cookbook, and what is the format of the tutorial (individual files with a README / a Jupyter notebook / a notebook with an accompanying recorded video of the author talking through the notebook etc)?

Any suggestions/thoughts would be great!

[carltape]

Hi @williameaton, what you propose sounds great to me. We started a bit toward this with the users workshop, which included this notebook for the homogeneous halfspace example (@aakash10gupta). It would be a significant effort to overhaul all the examples (and add notebooks), but it is needed. (Will, anyone: please step in) In the short term (and for the next release?), it'd be good to set up some subfolders and maybe standardize the folder names. But these directory names are presumably used in the buildbot testing, so I'm not touching it. :)

[danielpeter]

yes, that's a good idea to clean up the EXAMPLES/ folder! - but it's a bit complicated at the moment...

I think it would be good to define first what the different types and purposes of the examples are and how we want to bundle it together. in principle, we would distinguish between software testing and user examples & tutorials.

unfortunately, software testing is a vast field with many different approaches and terminology. software testing is typically separated into different levels. relevant for us would be:

• unit tests: individual components of the code are tested. in our case for example, if a mesh reading routine works correctly.
• integration tests: tests different components together. for example, if the solver and mesher together work fine, or if combining acoustic/elastic domains are handled correctly.
• system tests: for example, if the whole software stack works. in our case, that could be a test to see if SPECFEM and the auxiliary tools work properly. we also have the adjTomo tools and shakemovie, which interface with SPECFEM. that would be an even larger system test.

then, more specific test types and techniques for us would be:

• verification tests: that would verify if the code is written in a correct and coherent way, which we mostly do by code review at the moment.
• validation tests: those would validate the output, i.e., compare against analytical solutions or other numerical methods (normal modes,..).
• benchmarks: benchmark tests would in principle test the software performance.
• regression tests: that would check if the performance has slowed down or the accuracy of the solution degraded.
• stress/load tests: those would check the limits of the code, e.g., if too many receivers are breaking the code, etc.
• scenario tests: that would take some complex testing scenario, like an earthquake example for the SF fault to test end-to-end functionality.
• and there's probably more terminology that I forgot now.

so, regarding software testing, we do unit tests in the tests/ folder. those are not much documented. they run when you type: make tests. the integration tests and system tests are handled for us mostly by doing some scenario testing and validation testing. for those, a few of the examples in EXAMPLES/ are used by running the run_this_example.sh script provided in each of those examples, and then we compare the solutions against the solutions provided in the REF_SEIS/ subfolder in those examples.

and here, we don't separate between software testing purpose or user example & tutorial. those examples are also provided to help new users learn how to run a (simple) simulation. at the moment, we don't separate the examples in EXAMPLES/ between software testing purpose and user example & tutorial.

that leaves us with the question on how we want to deal with more specific user examples & tutorials.
all examples in EXAMPLES/ come with a small readme to explain what it is about. there is a mix between benchmarks, analytic solutions, reproducibility, and simple user examples, as well as small examples more for software testing. we probably want to move the benchmarks into a new separate folder, together with the analytic solution examples, as those are mostly about software testing.

The rest of the examples in the EXAMPLES/ folder are mostly ordered by simulation functionality, e.g., if they use the dynamic fault rupture module, or if they use CPMLs, or the in-house mesher meshfem3D. and then there are some scenario & reproducibility examples, Aakash sensitivity_kernels_liutromp2006/ example would be such a case.

I have the impression, that we still want to have EXAMPLES/ as a repo place to gather all these different user examples, scenarios and feature examples. the hierarchy is mostly flat in that folder, so it is more of a question how to provide an overview page to it. At the moment, this is the README in EXAMPLES/. this one we could certainly make a bit nicer to read as markdown page and better link into the manual / wiki / website.

The main missing part to me is to have even simpler cookbooks. from my side, these would ideally be jupyter notebooks, as they integrate text explanations with command-line code. also, they can be displayed on a website. so these tutorials could be all shown on the specfem.org website in our Training/ page - we just need to decide where we want them to be stored, and figure out how users can contribute new ones in a simple way. given that these cookbooks are specific to a certain version, SPECFEM2D, SPECFEM3D, etc., they probably belong into those repos with a new folder. or we create a completely new github repo in the SPECFEM/ organization to hold all tutorials, slide material etc., and then need to figure out a way how to display it nicely - any suggestions are welcome!

[carltape]

Exciting to see the latest updates with this pull request. Thank you!