add page about "How to test notebooks"

[haraldschilly]

There are various ways how to test jupyter notebooks, especially for python and R. Document the various ways how this works on CoCalc, what's available, and at the same time make sure this actually works on CoCalc :-)

• pytest: %%ipytest magic
• Support pytest-cov with ipytest chmp/ipytest#88
• nbval
• many others ... https://github.com/ml-tooling/best-of-jupyter
• and orthogonal are ways to add tests to student notebooks, nbgrader and others like https://otter-grader.readthedocs.io/en/latest/

[westurner]

• https://github.com/chmp/ipytest is one way to run pytest within jupyter notebooks.
• pytest functions conventionally start with def test_.
• There are many ways to make assertions of types and values in pytest test functions:
  • x=None; if x is None: raise AssertionError(("x is None!", dict(x=x)))
  • assert (the assert keyword)
  • self.assert_(expr, msg) (or unittest.TestCase().assert_())
• It's possible to call unittest assertion functions without a unittest.TestCase() in every Jupyter notebook input cell:

import unittest
Test = unittest.TestCase()

def test_testcase_methods_as_function_calls_that_raise_AssertionErrors():
  Test.assertTrue(True)
  Test.assertEqual("123", "123")
  
  with Test.assertRaises(ZeroDivisionError):
      1/0
  with Test.assertRaises(ValueError):
      int("0x0")

# a simple test runner that runs callables that start with `test_` (like pytest and iirc nose)
testscope = {**globals(), **locals()}
for name, testfunc in ((k,v) for k,v in testscope.items() if k.startswith('test_') and callable(v)):
    objstate = dict(name=name, func=testfunc, state='running')
    print(('running', objstate))
    try:
        objstate['output'] = testfunc()
        print(('completed', objstate))
    except KeyboardInterrupt:
        raise
    except Exception as exc:
        traceback.print_tb(exc.__traceback__)
        pass #

• unittest has more helpful AssertionError messages:

  • >>> assert "123" == "1234"
    AssertionError:
    >>> unittest.TestCase().assertEqual("123", "1234")
    AssertionError: '123' != '1234'
    - 123
    + 1234
    ?    +

• .. note:: Because assert statements are optimized out with python -O and python -OO, test assertions that use the assert keyword will not run if the -O or -OO flags are passed to the python process that runs the tests.
  • https://docs.python.org/3/using/cmdline.html#cmdoption-O
• Shorter def test_()-style functions are more succint than class Test0(unittest.TestCase)-style functions
• Shorter functions are easier to scroll through in Jupyter notebooks than TestCase subclasses (for simple assertions, at least)
• Porting functions between TestCase subclasses with self. references to refactor is more work than copy/pasting def test_() functions, in the average case
• Nose and pytest support functions named test_
  • unittest can, too
• pytest
  • Source: https://github.com/pytest-dev/pytest
  • Docs: https://docs.pytest.org/en/latest/
  • Docs: https://docs.pytest.org/en/latest/contents.html
  • Docs: https://docs.pytest.org/en/latest/reference/plugin_list.html
  • Docs: https://docs.pytest.org/en/latest/how-to/unittest.html
    • Pytest does run unittest.TestCase tests
  • Docs: https://docs.pytest.org/en/latest/reference/fixtures.html
  • Docs: https://docs.pytest.org/en/latest/how-to/fixtures.html
    • Pytest has fixtures instead of unittest.TestCase's setUp and tearDown methods