Pipeline/Build automation tool

[aucampia]

As discussed in the 2022-04-22 maintainers meeting, it would be good to write our pipelines that run on CI platforms in a platform agnostic way so it is easier to switch CI platforms.

We discussed Makefiles and Taskfiles as possibilities. Some factors to consider:

1. It should ideally work on most platforms we support.
2. It should support conditional/parameterized steps - we want to only install some dependencies on some operating systems, and installing dependencies on different operating systems require different commands, we may also only want to run some commands, like black when explicitly requested.

Some options:

• GNU Make / Makefiles:

  Makefiles are quite widely used for pipelines, however they are quite dependent on the userland, and consequently only really work well on Linux.

• Taskfile:

  Taskfiles are much simpler than Makefiles in some ways but they also rely quite heavily on go text templates to get conditional things done and there simplicitly kind of breaks down quickly.

• pyinvoke [pypi]:

  pyinvoke is python based, it has some quirks but being python based it is also likely a lot more familiar to those working on the RDFLib code base than both Makefiles and go's text templates are. One major downside of pyinvoke is however that while it is not dead, development of it is not nearly as active as Taskfile.

I know GNU Make fairly well, but I don't really like it, it is entirely turing complete but it is also incredibly convoluted and doing anything complex in it kind of makes it very difficult to read or maintain for anyone that is not quite familiar with it.

I'm fine with going for Taskfiles and it may even be the best choice, but coding in a YAML DSL interleaved with another text based DSL (Go text templates) with some shell DSL in the mix is also not really the best experience, one mitigating factor is that we don't need that much logic. While pyinvoke is not as well maintained as Taskfile, it is a lot easier to write logic in.

I'm interested to know what others think though, there may be other options also (list).

I'm somewhat open to writing an example in Taskfile and pyinvoke to compare and see what others think, but if there is clear consensus on one or the other then it is best to just go for that.

@white-gecko @nicholascar @edmondchuc

[aucampia]

Okay I gave pyinvoke a try, it will work but it is very annoying in that it does not support type annotations (pyinvoke/invoke#357), so I'm going for Taskfile for now. My main gripe with Taskfile is that it is still incredibly OS dependent for doing things like cleaning a directory, but this can be mitigated a bit. Still would like to know of more options.

[white-gecko]

I use Taskfiles in other situations, have not that much experience with it so far but I like it more then Makefiles and I am happy with it so far. So I would vote for Taskfiles.

[aucampia]

Busy working on it here: #1867 - I think it is mostly fine, just need to touch up the docs a bit and I also want to document the devcontainers.json and devtools image usage.

[aucampia]

Okay it is ready for review now:

• build: add Taskfile with development tasks #1867

I also included a devcontainer, please have a look when you can.