How to setup and develop SuperDuperDB

How to develop in SuperDuperDB

This development workflow is known to work on MacOS and Linux.

You will need to globally install the following programs outside your Python environment using pyenv, Brew, apt or your favorite package manager.

• Python 3.8 and 3.10, our minimum and maximum supported versions
• Docker, a virtualization system
• And Poetry, a Python package manager (this needs to be at least 1.5.0)

(MongoDB Community edition, our primary database integration, is now installed through Docker and doesn’t need to be installed locally, at least for running the test-suite.)

Virtual environments

We do all our development in Python virtual environments or Docker containers so as not to pollute our system Python installations. Poetry can create and manage a virtual environment if you don’t have one. This involves installing and then activating the project Poetry environment (see below for more details).

Suggestion: making things break if there’s no virtual environment

pip3 might point to any executable so should always be avoided.

python -m pip or poetry will use whatever your current Python is, which might accidentally be the wrong one.

If you develop from the command line, a useful idea to avoid problems is to make sure that python does not resolve to anything if no virtual environment is loaded, so your commands just break rather than silently installing things in the wrong place.

Don’t worry, you can and should have python3.8, python3.10 and any other Python versions you have installed in your PATH.

To test, deactivate or otherwise leave your virtual environment, and then type -a python should say bash: type: python: not found

To fix this, change your PATH or if there’s something in there that can’t go, let me know!

If you use an IDE, this probably won’t be necessary.

Getting the code

The code is located at https://github.com/SuperDuperDB/superduperdb.

We each work on personal forks of this repository, so create your own fork on your personal GitHub account here.

From that landing page, click on the green Code button, then Clone, SSH, and then Copy, to get a URL like this: [email protected]:<your-name-here>/superduperdb.git

Go to the parent of the directory you want to work in, type git clone and paste that URL, to get something like:

git clone [email protected]:<your-name-here>/superduperdb.git

Now type cd superduperdb. All operations are done in this directory.

Either create a new Python 3.8 virtual environment and make sure it’s active, or make sure that python is not in your PATH, so poetry will create a new virtual environment for you in a cache.

Then type poetry install and wait about 15(!) minutes.

This step downloads different software depending on your hardware and software configuration and might cause issues on previously unseen configurations, let us know!

Make sure that Docker is running.

Finally, type make test and if all has gone well you should see a couple of dozen lines of success messages, and about fifty lines of warnings from other packages, see https://github.com/SuperDuperDB/superduperdb/issues/219.

make test will be quite slow on the first run, as it loads a Docker and compiles all the Python, but subsequent runs will be quite fast. If progress stops for more than a few seconds, then it is likely a problem communicating with a Docker container.

Development workflow

Please take a look at Git workflow and[How to use poetry for the basics and some notes and tips.

Starting out:

For each new feature or bug fix, create a new branch on your own fork

The work loop:

1. Make changes to the Python code
2. make fix-and-test which fixes style errors and runs style tests and unit tests.
3. Commit often
4. Update from mainoften
   1. git pull --rebase upstream/main and git push -f
   2. or git update if you have it
   3. Finally poetry install --sync will install any missing dependencies, and remove unknown ones.

When ready to submit:

1. Distill the work into one or more minimal, self-consistent commits
2. Run make test which runs style tests and unit tests but fixes nothing
3. Open a pull request

During reviews:

1. Fix code in response to comments in one of two ways:
   1. Use --fixup commits (read here)
   2. Rebase changes into existing commits and git push -f
2. Update from main
3. Run make test

Finally:

1. Rebase any --fixup commits then git push -f
2. Update from main
3. Run make test
4. Go to the pull request
5. Wait until it has passed the Continuous Integration tests
6. Pull it if you have pull rights on the main repo, or ping an administrator.

Documentation rebuild (not needed at this instant):

1. cd docs
2. make clean
3. make html

Note:

In the early phases where no one sees the repo, we can even rewrite history to erase egregious mistakes, so don’t get too stressed.