Improve test situation #199

brettviren · 2023-03-06T21:40:23Z

This PR begins addressing https://github.com/WireCell/wire-cell-toolkit/projects/16

…test/bats subrepo: subdir: "test/bats" merged: "a7106392" upstream: origin: "https://github.com/bats-core/bats-core.git" branch: "master" commit: "a7106392" git-subrepo: version: "0.4.5" origin: "???" commit: "???"

…tory

brettviren · 2023-03-17T12:40:19Z

The PR #194 is gaining some BATS tests that will need porting once this PR is merged. I list them here to remind me

test-uboone-img.bats
[ ]

- Invert default of when to run tests. Default is that they are OFF. - Simplify test framework to remove "variant" tests from scope. - Simplify by no longer bake "fast" vs "slow" or "unit" vs "integ" into test source file name. - Bring `waf_unit_test` into `waft/wcb_unit_test` and move test argument handling there. - Test related arguments will set during `configure` are saved to `build`. - Add new feature to measure, store and report the run time for each test. - Add new feature to exclude re-running slow tests based on a maximum duration limit. - Update `framework.org` to match.

- Make the repo "live" only and entirely under build/tests and remove complexity of searching a "test data path". - Add fillrepo and packrepo "wcb" commands to automate some management of test data repository files. - Remove wct-bats.sh functions related to the old way of locting test data files, add/improve ones for the new way. - Further purge the concept of "variant" tests and likewise downplay their opposites of "atomic" tests. - Sync documentation and tests with these code/concept shifts.

Downgrade issue202 bats to a check as it is too long running

brettviren · 2023-04-10T13:33:39Z

Hi @HaiwangYu

I feel this PR is now ready for merging.

There are a lot of files changed/added but actually not much library code has been touched. The build system is revamped. Here is a summary of what this PR brings:

The building and running of tests is off by default. To run tests:

./wcb distclean 
./wcb configure [...]
./wcb install
./wcb --tests

Except for a needed distclean changes should be backward-compatible with old scripts. In particular --notests is still honored. You can give --tests to configure to avoid having to remember to run it during build.
wcb is now simply a copy of plain waf and the new waft/ sub package holds the waf-tools that we used to dump into wcb plus some new ones.
Changes to define test groups to run "history" and "report" test types after the usual "atomic" tests.
A new Waf tool is added to support the new "test data repository" feature to provide files for "history" tests. Until we find something more appropriate, the repo simply lives in my BNL web area.
Some documentation on this build system refresh is here:

https://github.com/WireCell/wire-cell-toolkit/blob/check-test/waft/README.org

Many new and changed code in */test/ and the test/ sub package.
Introduction of BATS tests and a copy of the bats-core source to run them.
Lots of new and cleaned up documentation including fairly comprehensive description of the revamped testing.
Documentation on the revamped testing is here:

https://github.com/WireCell/wire-cell-toolkit/blob/check-test/test/README.org

A waft tool to build docs from org files.
Building docs is also off by default and activated with ./wcb --docs.

Please give ./wcb --tests a try and also run your usual external tests. After this is merged I would like to capture your external tests into internal ones.

HaiwangYu · 2023-04-13T15:34:10Z

Hi @brettviren, trying to build gives me this error both with singularity and on a dunegpvm. Any suggestions how to fix it?

Waf: Entering directory `/home/yuhw/wc/larsoft925/src/pr-199/build'
[1/6] Creating build/tests/input.url
[2/6] Creating build/tests/history-0.20.0.url
[3/6] Creating build/tests/history-0.21.0.url
[4/6] Creating build/tests/history-0.22.0.url
[5/6] Creating build/tests/history-0.23.0.url
[6/6] Creating build/tests/history-0.24.1.url
Waf: Leaving directory `/home/yuhw/wc/larsoft925/src/pr-199/build'
Build failed
Traceback (most recent call last):
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1350, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1240, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1286, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1235, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1006, in _send_output
    self.send(msg)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 946, in send
    self.connect()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1409, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yuhw/wc/larsoft925/src/pr-199/.waf3-2.0.25-767522112be77f8585812fcfaa08e805/waflib/Task.py", line 180, in process
    ret=self.run()
  File "/home/yuhw/wc/larsoft925/src/pr-199/waft/datarepo.py", line 111, in run
    web = urlopen(url)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1353, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>

Traceback (most recent call last):
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1350, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1240, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1286, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1235, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1006, in _send_output
    self.send(msg)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 946, in send
    self.connect()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1409, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yuhw/wc/larsoft925/src/pr-199/.waf3-2.0.25-767522112be77f8585812fcfaa08e805/waflib/Task.py", line 180, in process
    ret=self.run()
  File "/home/yuhw/wc/larsoft925/src/pr-199/waft/datarepo.py", line 111, in run
    web = urlopen(url)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1353, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>

Traceback (most recent call last):
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1350, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1240, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1286, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1235, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1006, in _send_output
    self.send(msg)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 946, in send
    self.connect()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1409, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yuhw/wc/larsoft925/src/pr-199/.waf3-2.0.25-767522112be77f8585812fcfaa08e805/waflib/Task.py", line 180, in process
    ret=self.run()
  File "/home/yuhw/wc/larsoft925/src/pr-199/waft/datarepo.py", line 111, in run
    web = urlopen(url)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1353, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>

Traceback (most recent call last):
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1350, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1240, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1286, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1235, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1006, in _send_output
    self.send(msg)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 946, in send
    self.connect()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1409, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yuhw/wc/larsoft925/src/pr-199/.waf3-2.0.25-767522112be77f8585812fcfaa08e805/waflib/Task.py", line 180, in process
    ret=self.run()
  File "/home/yuhw/wc/larsoft925/src/pr-199/waft/datarepo.py", line 111, in run
    web = urlopen(url)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1353, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>

Traceback (most recent call last):
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1350, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1240, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1286, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1235, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1006, in _send_output
    self.send(msg)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 946, in send
    self.connect()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1409, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yuhw/wc/larsoft925/src/pr-199/.waf3-2.0.25-767522112be77f8585812fcfaa08e805/waflib/Task.py", line 180, in process
    ret=self.run()
  File "/home/yuhw/wc/larsoft925/src/pr-199/waft/datarepo.py", line 111, in run
    web = urlopen(url)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1353, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>

Traceback (most recent call last):
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1350, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1240, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1286, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1235, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1006, in _send_output
    self.send(msg)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 946, in send
    self.connect()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/http/client.py", line 1409, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yuhw/wc/larsoft925/src/pr-199/.waf3-2.0.25-767522112be77f8585812fcfaa08e805/waflib/Task.py", line 180, in process
    ret=self.run()
  File "/home/yuhw/wc/larsoft925/src/pr-199/waft/datarepo.py", line 111, in run
    web = urlopen(url)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_8_3b/Linux64bit+3.10-2.17/lib/python3.8/urllib/request.py", line 1353, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>

brettviren · 2023-04-14T14:26:40Z

@HaiwangYu interesting.

The failing code is what should install the "test data repo" files into build/tests/. For now, the "repo" is just a bunch of tar files in my web area under:

https://www.phy.bnl.gov/~bviren/tmp/wcttest/data_repo/

I can not reproduce the certificate verify failure on dunegpvm01:

-bash-4.2$ source /cvmfs/larsoft.opensciencegrid.org/products/setup
-bash-4.2$ setup python v3_8_3b
-bash-4.2$ cat t.py

from urllib import request
u = request.urlopen("https://www.phy.bnl.gov/~bviren/tmp/wcttest/data_repo/")
print(u.getcode())

-bash-4.2$ python t.py 
200

Likewise, wget and curl work.

It looks like something is "special" in your particular FNAL environment. Maybe you can think what may be different. In my case, I just log in with very minimal default setup (as usual with FNAL's broken auth, I can't even read my home directory). I cd /tmp and then do the above setup.

Searching for solutions gives one lead. Can you please try to run the following in your environment:

openssl s_client -connect some-domain.com:443
Ctrl-d

vs:

openssl s_client -CApath /etc/ssl/certs/ -connect some-domain.com:443
Ctrl-d

If the latter succeeds then perhaps we can fix FNAL's broken environment by setting:

export SSL_CERT_DIR=/etc/ssl/certs/

HaiwangYu · 2023-04-14T15:05:10Z

Hi @brettviren, thanks a lot!

I can confirm your minimum setup works for me too. And I can further confirm this line is the culprit in my setup.

source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh

And further this line in that setup_dune.sh file:

# For ifbeam
export SSL_CERT_FILE=""

However, for me setup SSL_CERT_DIR doesn't seem to work. But export SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt seems to work on a dunegpvm. I will keep testing this PR then.

brettviren · 2023-04-14T15:46:00Z

Great! Perhaps we should make creating the "test data repo" be contingent on using --tests.

HaiwangYu · 2023-04-14T15:51:01Z

Great! Perhaps we should make creating the "test data repo" be contingent on using --tests.

Yes, I agree that is a better option.

HaiwangYu · 2023-04-14T20:38:52Z

@brettviren, thanks for building this amazing testing system. I think a lot of moving parts have been tightened up. I gave it a try and think it should work well. I made a PR to this dev branch with my practice: #213. Here are some test notes: pr199.pdf

I think I am still a little bit confused by how to use the test categories (atomic, history, report...). But I think I will learn more from your future introduction talk on this.

One thing I noticed is that sigproc results changed while gen stayed identical with 0.24.1. While this could be explained by RNG and I can not find any source code change related to this. Still puzzles me a bit:
https://www.phy.bnl.gov/~yuhw/larsoft925/src/pr-199/build/tests/reports/0.24.1-82-g74968af/history-addnoise/comp1d-spec-history.png
https://www.phy.bnl.gov/~yuhw/larsoft925/src/pr-199/build/tests/reports/0.24.1-82-g74968af/history_pdsp_sim_sp/comp1d-wave-history.png

For now, if you are OK, I will merge my practice PR and this PR to the master branch in one pass

brettviren · 2023-04-14T21:45:50Z

Hi @HaiwangYu

The shift in time of the that one SP peak near tick 2000 is really weird. I guess if it is due to noise then maybe it can be explained by different random stream.

Maybe looking at more channel waveforms would help show what is happening. Some new - old "frame difference" plots at both ADC and SP data tiers might be informative.

Yes, I'm okay with you merging your #213. Note also #214 can go in soon after as well.

My first try

HaiwangYu · 2023-04-17T14:25:42Z

@brettviren, I think the changes at tick ~2000 is not a shift but fluctuations of yes/no ROI found for those two peaks. I saw this change several times for previous sigproc across version checks. So there is a high chance that it is due to fluctuations. But worth more investigations to confirm as you said. For the time being, I would like to merge this PR first. Because no charges in this PR should affect sigproc directly as far as I can see.

brettviren added 2 commits March 2, 2023 08:49

Import waf-tools to waft/

6835a99

Add support for 'variant' tests, examples, docs

abda320

brettviren marked this pull request as draft March 6, 2023 21:47

brettviren added 2 commits March 8, 2023 11:15

Prepare to merge master

baa586a

Merge master

bd90b96

brettviren mentioned this pull request Mar 8, 2023

#include "config.h" in WireCellUtil/Exceptions.h #200

Closed

brettviren added 7 commits March 9, 2023 10:24

Use provided bats

c4aded4

More test infra, examples and docs

2bb1f6f

Add testing docs, adding building org to pdf with pandoc

5e979eb

Merge remote-tracking branch 'origin/master'

694fe32

Start to integrate a data repo

a6c41f7

Add a way to save out files generated in BATS test to the build direc…

f7cc775

…tory

brettviren added 16 commits March 20, 2023 08:06

Merge remote-tracking branch 'origin/master'

edf89b0

From tiling-nudge, missed due to .gitignore rules

dedfc03

Finally stop wire-cell from giving error on -h/-v or no args

eb93e48

Upgrade old bats to use wct-bats.sh

581776d

More testing related improvements

4713e26

More test cleanup including purging obsolete and broken tests

503ed9f

Add bats test result archive mechanism

7326ac3

More testing related improvements

f2d6fdd

Merge remote-tracking branch 'origin/master'

5cef597

Reflect command change in wire-cell-python

e5696d7

Checkpoint before merge latest from issue #202 fixes

ede5dc8

Merge

4b74442

Merge remote-tracking branch 'origin/master'

4902731

Fix test

961cccf

Split off issue 202 test and make roundtrip test thinner

bc5a4f6

Simplify saveout, more thoughts on how to do historical testing

3c259c1

brettviren added 10 commits March 31, 2023 14:48

Add first historical test.

bddd0cd

More on historical test, plus changes missed by last commit

c9eba11

Somewhat integrate the local package docs

4daf65d

More yak shaving of docs

f9d92c7

Waft tools for datarepo and org export working

ce9649e

Make READMEs a bit more uniform, flesh out a couple

5c8bfef

Merge

7ce7b82

Make finding wct bats lib more portable.

74968af

Downgrade issue202 bats to a check as it is too long running

brettviren marked this pull request as ready for review April 10, 2023 13:04

brettviren requested a review from HaiwangYu April 10, 2023 13:33

HaiwangYu added 2 commits April 14, 2023 16:23

pdsp sim sp check

77ead3c

update comments

ebd8e3e

brettviren mentioned this pull request Apr 14, 2023

Fix issue #211 #214

Merged

Merge pull request #213 from HaiwangYu/check-test

dc2e717

My first try

HaiwangYu merged commit c16e2d5 into master Apr 17, 2023

This was referenced Apr 19, 2023

A better test data repository is needed. #215

Open

Check test2 #216

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve test situation #199

Improve test situation #199

brettviren commented Mar 6, 2023

brettviren commented Mar 17, 2023

brettviren commented Apr 10, 2023

HaiwangYu commented Apr 13, 2023 •

edited

Loading

brettviren commented Apr 14, 2023

HaiwangYu commented Apr 14, 2023 •

edited

Loading

brettviren commented Apr 14, 2023

HaiwangYu commented Apr 14, 2023

HaiwangYu commented Apr 14, 2023 •

edited

Loading

brettviren commented Apr 14, 2023

HaiwangYu commented Apr 17, 2023

Improve test situation #199

Improve test situation #199

Conversation

brettviren commented Mar 6, 2023

brettviren commented Mar 17, 2023

brettviren commented Apr 10, 2023

HaiwangYu commented Apr 13, 2023 • edited Loading

brettviren commented Apr 14, 2023

HaiwangYu commented Apr 14, 2023 • edited Loading

brettviren commented Apr 14, 2023

HaiwangYu commented Apr 14, 2023

HaiwangYu commented Apr 14, 2023 • edited Loading

brettviren commented Apr 14, 2023

HaiwangYu commented Apr 17, 2023

HaiwangYu commented Apr 13, 2023 •

edited

Loading

HaiwangYu commented Apr 14, 2023 •

edited

Loading

HaiwangYu commented Apr 14, 2023 •

edited

Loading