regression test and examples cleanup

[jeinstei]

Work in support of cleaning up regression testing pathing and test output (e.g. subtests vs TestRunners)

• Remove bad symlink paths for examples
• Modify documentation with examples and make sure READMEs are consistent and have example commands
• Include all the examples as needed with proper options as part of regression testing
• Test regression test in prod and dev environments currently used

[cchall]

One note: I think the readmes should not be removed entirely. There should probably be at least a brief description and the run command accompanying each example.

[jeinstei]

libensemble 0.9.1 installed over a default sirepo jupyter is seeing a failure in the test_executor_timeout test that I'm looking in to. This has something to do with logging within libensemble not seeing the ensemble.log file in /tmp, even though it does get created. There's some sort of race condition somewhere.

======================================================================
ERROR: test_executor_timeout (test_setup.TestSetupFeatures)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/jupyter/rsoft-git/rsopt/tests/test_setup.py", line 35, in test_executor_timeout
    H, persis_info, _ = runner.run()
  File "/home/vagrant/jupyter/rsoft-git/rsopt/rsopt/libe_tools/optimizer.py", line 74, in run
    self.alloc_specs, self.libE_specs, H0=self.H0)
  File "/home/vagrant/.pyenv/versions/py3/lib/python3.7/site-packages/libensemble/libE.py", line 232, in libE
    return libE_funcs[comms_type](sim_specs, gen_specs, exit_criteria, persis_info, alloc_specs, libE_specs, H0)
  File "/home/vagrant/.pyenv/versions/py3/lib/python3.7/site-packages/libensemble/libE.py", line 464, in libE_local
    wcomms, sim_specs, gen_specs, exit_criteria, persis_info, alloc_specs, libE_specs, hist, on_cleanup=cleanup
  File "/home/vagrant/.pyenv/versions/py3/lib/python3.7/site-packages/libensemble/libE.py", line 249, in manager
    logger.info("Logger initializing: [workerID] precedes each line. [0] = Manager")
...
  File "/home/vagrant/.pyenv/versions/3.7.2/lib/python3.7/logging/__init__.py", line 1121, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp9tzl2hl6/ensemble.log'

----------------------------------------------------------------------
Ran 34 tests in 9.011s

FAILED (errors=1)

The documentation updates work pretty well, but I need to figure out how to include raw readme files so the markup works correctly. Also looking at getting in some example commands.

[cchall]

I should have asked if this was test_executor_timeout
This is a known issue:

rsopt/tests/test_setup.py

Line 16 in f98def6

# Tests all pass when run individually but if the whole test is executed then test_executor_timeout fails

Running test_executor_timeout alone should pass. If you can figure out the root problem that's great though.

[jeinstei]

Boy, that's interesting. I'll take a look a bit more as it doesn't seem to make sense.