Add Guix buildpack #1048

hulecomte · 2021-06-21T14:46:55Z

Hi! This patch adds a backend that uses the GNU Guix package manager. Guix provides reproducible software environments, transparency and provenance tracking. Coupled with a channels.scm file, it allows users to pin to a specific Guix revision from which the set of packages listed in their manifest.scm is deployed, thereby supporting very fine-grain reproducibility.

Let me know what you think

welcome · 2021-06-21T14:46:56Z

Thanks for submitting your first pull request! You are awesome! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please make sure you followed the pull request template, as this will help us review your contribution more quickly.

You can meet the other Jovyans by joining our Discourse forum. There is also a intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

yuvipanda

Thank you so much for adding this.

We already support nix here - can you tell me just a little bit about how they relate to each other?

Also, the current maintainers of repo2docker don't have a lot of guix experience. Do you think there will be enough interest from the guix community itself to keep this up to date?

I've left general comments but haven't managed to test it.

I look forward to working on getting this merged!

repo2docker/buildpacks/guix/__init__.py

civodul · 2021-07-05T12:27:11Z

Hi @yuvipanda. I can reply on the Nix vs. Guix bit (I'm a Guix co-maintainer). :-)

Guix and Nix are independent projects with different code bases. The tools are both "functional package managers", which sets them apart from other tools out there; you'll find some of the core concepts are the same, and Guix borrowed them from Nix (Guix's build daemon is based on that of Nix).

Apart from that, the user interfaces are different (in Guix we try hard to make the CLI accessible to non-experts, to offer customization options, and to provide good integration for a variety of software deployment tools), the packages they provide are different (e.g., Guix insists on free software built from source as a way to improve transparency and security), and so on.

There's a growing scientific computing community around Guix with a focus on high-performance computing (HPC) and reproducible science. I'm confident there'll be interest in keeping Guix support afloat in repo2docker.

Let me know if you need more info!

yuvipanda · 2021-07-05T14:43:54Z

@civodul Thank you for your explanations - that sounds good enough to me. Let's working on getting stuff outside of $HOME and work towards getting this deployed! \o/

If you rebase your PR, it should also fix the failing docker build and build docs as well.

hulecomte · 2021-07-05T16:47:59Z

Thanks for your feedback @yuvipanda, what do you think now ?

yuvipanda

Thanks, @hulecomte! I've left some more comments. I haven't tested it locally yet, though.

Another question I have is - if a user sets up a repo to use guix, then they will need to make sure that jupyter is installed inside the container. the Nix buildbpack has a nice note, can you add something similar for Guix too?

I'm not sure why, but it looks like the lint failure must be fixed first before the integration tests will run (not sure why?!). Can you take a shot at it?

Thank you for all your work! \o/

repo2docker/buildpacks/guix/guix-install.bash

repo2docker/buildpacks/guix/__init__.py

yuvipanda · 2021-07-05T18:59:54Z

repo2docker/buildpacks/guix/__init__.py

+        assemble_script ="""
+                 /var/guix/profiles/per-user/root/current-guix/bin/guix-daemon \
+                 --build-users-group=guixbuild --disable-chroot & \
+                 mv /usr/bin/python /usr/bin/python.debian && \


Can you add some more comments on why this is necessary? Removing system python might have unintended side effects, no? Wouldn't the path precedence deal with it?

I added some comments about this steps in the header of the function, let me now what you think.

The point of using Guix is to avoid any confusions about the version or the provenance of the software you want to use and therefore guarantee and allow reproducibility of a software environment. Therefore having a python that we now nothing about could jeopardize that. Also counting on the path precedence to deal with that give us no guarantee of software environment reproducibility and could influence our results by making things work when they shouldn't for example.

While testing the Guix buildpack, I tried installing and using numpy, it take me a long time to realize that I where trying to do so but with the python already installed on the docker image, wich could not work with numpy (versions conflict). Also it can be confusing when the software use environment variables and you are not aware of all the other software present on the system.

repo2docker/buildpacks/guix/__init__.py

repo2docker/app.py

yuvipanda

Thanks for addressing these, @hulecomte! I still haven't managed to test this locally, but hope this is all still useful.

To respond to #1048 (comment), IMO we shouldn't hand-move things installed with another package manager (in this case, apt). If you try to use apt to remove it, I'm sure it'll take a bunch of other packages with it. IMO, this means we shouldn't try to do that, since it will just silently break other things. We already do this with nix, conda, etc - IMO that's what we should do here too.

When the container runs and the user tries to do pip install numpy, it should automatically use the 'correct' python installed - in this case, the guix one. For example, in the conda buildpack (

repo2docker/repo2docker/buildpacks/conda/__init__.py

Line 54 in f07111c

path.insert(0, "${CONDA_DIR}/bin")

), just the PATH is enough to 'do the right thing'. And /usr/bin/python3 is never used.

Let's try do the same here?

Thanks for working with me through this!

yuvipanda · 2021-07-06T13:12:44Z

repo2docker/buildpacks/guix/guix-install.bash

@@ -0,0 +1,13 @@
+# This downloads and installs a pinned version of Guix using a pinned version of the installation script.


Suggested change

# This downloads and installs a pinned version of Guix using a pinned version of the installation script.

#!/bin/bash

# This downloads and installs a pinned version of Guix using a pinned version of the installation script.

Hey thanks for that, I will add it in the next rebase!

yuvipanda · 2021-07-06T13:22:05Z

repo2docker/buildpacks/guix/__init__.py

+        """
+        assemble_script ="""
+                 /var/guix/profiles/per-user/root/current-guix/bin/guix-daemon \
+                 --build-users-group=guixbuild --disable-chroot & \


Is this single & intentional? Does this daemon continue to run in the background after this step is done?

Indeed, you need the guix-daemon running to be able to use guix!

yuvipanda · 2021-07-06T13:47:09Z

repo2docker/buildpacks/guix/__init__.py

+                 /var/guix/profiles/per-user/root/current-guix/bin/guix-daemon \
+                 --build-users-group=guixbuild --disable-chroot & \
+                 mv /usr/bin/python /usr/bin/python.debian && \
+                 su - $NB_USER -c '{}' && \


You can return multiple scripts from here, some to be run as root and some as NB_USER. That's probably a bit cleaner than using su here

Hey, so Im struggling a bit with this part. It seems I can't keep the guix-daemon running between distinct steps. In some cases I have some issues when the docker images is not completely rebuilt, steps by steps but reusing some steps already executed before. That's why I did that here, I launch the guix-daemon just before running the Guix command I need, therefore I'm absolutely sure guix-daemon is running. Maybe you have some clues or ideas about a better way to do so ?

hulecomte · 2021-07-07T15:28:54Z

Thanks for addressing these, @hulecomte! I still haven't managed to test this locally, but hope this is all still useful.

To respond to #1048 (comment), IMO we shouldn't hand-move things installed with another package manager (in this case, apt). If you try to use apt to remove it, I'm sure it'll take a bunch of other packages with it. IMO, this means we shouldn't try to do that, since it will just silently break other things. We already do this with nix, conda, etc - IMO that's what we should do here too.

When the container runs and the user tries to do pip install numpy, it should automatically use the 'correct' python installed - in this case, the guix one. For example, in the conda buildpack (

repo2docker/repo2docker/buildpacks/conda/__init__.py

Line 54 in f07111c

path.insert(0, "${CONDA_DIR}/bin")

), just the PATH is enough to 'do the right thing'. And /usr/bin/python3 is never used.

Let's try do the same here?

Thanks for working with me through this!

Hey yuvipanda! I'm currently trying that but it seems I still can use python while using bash script to test the buildpack...Maybe I am not doing it well enough? What do you think ?

Anyway, I don't now why lint test is failing and I don't know how it works too. How can I reproduce it locally ?

Thanks!

hulecomte · 2021-08-11T10:31:25Z

Hey @yuvipanda here are some updates:
-I modified the doc and the guix-install.bash as you suggested.
-Added a PATH 'do to the right thing' as we discussed it previously.

What do you think?

civodul · 2021-09-17T09:53:01Z

Hi @yuvipanda & all! Would be great if @hulecomte's work could get in. :-)

Is there anything left that you think we should do?

Thanks in advance!

civodul · 2022-01-18T21:42:43Z

Hi @yuvipanda, could you take another look at this PR?

Thanks in advance.

rekado · 2022-02-01T16:29:26Z

I'm very excited about this addition. Are there any other obstacles to getting this merged?

for more information, see https://pre-commit.ci

civodul · 2022-11-03T09:02:07Z

Hi! Any update here?

yuvipanda requested changes Jul 4, 2021

View reviewed changes

repo2docker/buildpacks/guix/__init__.py Outdated Show resolved Hide resolved

repo2docker/buildpacks/guix/__init__.py Outdated Show resolved Hide resolved

hulecomte force-pushed the master branch from 074a5e3 to fc4cc45 Compare July 5, 2021 16:40

hulecomte requested a review from yuvipanda July 5, 2021 16:46

yuvipanda requested changes Jul 5, 2021

View reviewed changes

hulecomte force-pushed the master branch from fc4cc45 to ffd2ddf Compare July 6, 2021 12:43

yuvipanda requested changes Jul 6, 2021

View reviewed changes

hulecomte requested a review from yuvipanda July 30, 2021 12:13

Add Guix buildpack

8edfa53

hulecomte force-pushed the master branch from ffd2ddf to 8edfa53 Compare August 11, 2021 10:21

consideRatio changed the title ~~[MRG] Add Guix buildpack~~ Add Guix buildpack Oct 30, 2022

consideRatio added new buildpack labels Oct 31, 2022

consideRatio closed this Oct 31, 2022

consideRatio reopened this Oct 31, 2022

[pre-commit.ci] auto fixes from pre-commit.com hooks

54c193f

for more information, see https://pre-commit.ci

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Guix buildpack #1048

Add Guix buildpack #1048

hulecomte commented Jun 21, 2021

welcome bot commented Jun 21, 2021

yuvipanda left a comment

civodul commented Jul 5, 2021

yuvipanda commented Jul 5, 2021

hulecomte commented Jul 5, 2021

yuvipanda left a comment

yuvipanda Jul 5, 2021

hulecomte Jul 6, 2021

yuvipanda left a comment

yuvipanda Jul 6, 2021

hulecomte Jul 7, 2021

yuvipanda Jul 6, 2021

hulecomte Jul 6, 2021

yuvipanda Jul 6, 2021

hulecomte Jul 7, 2021

hulecomte commented Jul 7, 2021

hulecomte commented Aug 11, 2021 •

edited

Loading

civodul commented Sep 17, 2021

civodul commented Jan 18, 2022

rekado commented Feb 1, 2022

civodul commented Nov 3, 2022

		@@ -0,0 +1,13 @@
		# This downloads and installs a pinned version of Guix using a pinned version of the installation script.

	# This downloads and installs a pinned version of Guix using a pinned version of the installation script.
	#!/bin/bash
	# This downloads and installs a pinned version of Guix using a pinned version of the installation script.

Add Guix buildpack #1048

Are you sure you want to change the base?

Add Guix buildpack #1048

Conversation

hulecomte commented Jun 21, 2021

welcome bot commented Jun 21, 2021

yuvipanda left a comment

Choose a reason for hiding this comment

civodul commented Jul 5, 2021

yuvipanda commented Jul 5, 2021

hulecomte commented Jul 5, 2021

yuvipanda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuvipanda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hulecomte commented Jul 7, 2021

hulecomte commented Aug 11, 2021 • edited Loading

civodul commented Sep 17, 2021

civodul commented Jan 18, 2022

rekado commented Feb 1, 2022

civodul commented Nov 3, 2022

hulecomte commented Aug 11, 2021 •

edited

Loading