[r-apt] "installBspm": true is not working on Debian

[eitsupi]

I have confirmed that install.packages("curl") fails on R started by a non-root user (vscode) on an image built from devcontainer.json as shown below.

{
	"image": "mcr.microsoft.com/devcontainers/base:debian",
	"features": {
		"ghcr.io/rocker-org/devcontainer-features/r-apt:0": {
			"installBspm": true
		}
	}
}

This setup corresponds to the following script run with INSTALLBSPM=true.
https://github.com/rocker-org/devcontainer-features/blob/583387a2831c181905c571b306f3a976affbe5cb/src/r-apt/install.sh

Perhaps the script contains improper settings somewhere?

Originally posted by @eitsupi in #169 (comment)

[Enchufa2]

Not sure what happens here... I see:

$ sudo /usr/local/lib/R/site-library/bspm/service/bspm.py discover
Traceback (most recent call last):
  File "/usr/local/lib/R/site-library/bspm/service/bspm.py", line 150, in <module>
    read_conf()
  File "/usr/local/lib/R/site-library/bspm/service/bspm.py", line 16, in read_conf
    conf = backend.discover()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/R/site-library/bspm/service/backend/apt.py", line 14, in discover
    cache = apt.Cache()
            ^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 170, in __init__
    self.open(progress)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 232, in open
    self._cache = apt_pkg.Cache(progress)
                  ^^^^^^^^^^^^^^^^^^^^^^^
apt_pkg.Error: E:The value 'testing' is invalid for APT::Default-Release as such a release is not available in the sources

But then, after running sudo apt-get update manually, everything works fine.

[eitsupi]

Thanks for looking at this.
Is there a problem with setting testing...?

devcontainer-features/src/r-apt/install.sh

Lines 157 to 160 in 583387a

	echo "Set up Debian testing..."
	echo "deb http://http.debian.net/debian testing main" >/etc/apt/sources.list.d/debian-testing.list
	echo "deb http://http.debian.net/debian unstable main" >/etc/apt/sources.list.d/debian-unstable.list
	echo 'APT::Default-Release "testing";' >/etc/apt/apt.conf.d/default

Do you think bspm should automatically run apt update?

[Enchufa2]

It does update the cache here:

https://github.com/Enchufa2/bspm/blob/4d3c8ce500c0c4f7b30ca322fdf51d9c26b23d9d/inst/service/backend/apt.py#L15

But note that the error is raised one line before that, when we are just instantiating the Cache object. So I suppose that apt update does something else on top of that, maybe to fix inconsistent states?

bspm runs just fine with debian:testing, but the base image you are using here is a stable one, right? Then I'd say it's your responsibility to ensure that the cache is in a consistent state after switching to testing.

[Enchufa2]

[Removing my previous comment since I forgot to add the repos. My last comment still holds].

[Enchufa2]

Here's a minimal reprex:

$ docker run --rm -it mcr.microsoft.com/devcontainers/base:debian
$ echo "deb http://http.debian.net/debian testing main" >/etc/apt/sources.list.d/debian-testing.list 
$ echo 'APT::Default-Release "testing";' >/etc/apt/apt.conf.d/default 
$ apt-get -y update && apt-get -y install python3-apt
$ rm -rf /var/lib/apt/lists/*
$ python3 -c "import apt; apt.Cache()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 170, in __init__
    self.open(progress)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 232, in open
    self._cache = apt_pkg.Cache(progress)
                  ^^^^^^^^^^^^^^^^^^^^^^^
apt_pkg.Error: E:The value 'testing' is invalid for APT::Default-Release as such a release is not available in the sources

I.e. bspm cannot run apt.Cache().update() if apt.Cache() cannot be instantiated in the first place.

[eitsupi]

Thanks,

Interestingly, a search on that error message turned up previous issues of the Rocker project. (rocker-org/rocker#167)

[Enchufa2]

It's strange, because, according to the docs,

import apt

# First of all, open the cache
cache = apt.Cache()
# Now, lets update the package list
cache.update()

should be equivalent to apt-get update, which succeeds. But opening the cache fails in the first place. Maybe this is a bug in the Python API?

[Enchufa2]

Another question is: do you really need to set APT::Default-Release "testing"; in the first place? If I spawn a debian:stable container and add the testing repo, package installations come from testing, and this bug is no more.

[eitsupi]

Thanks for taking a closer look at this.

Maybe this is a bug in the Python API?

This certainly looks like a bug in python-apt.

Another question is: do you really need to set APT::Default-Release "testing"; in the first place?

I simply wanted to provide a way to install the latest R. I don't know if this is the best way to do it.

[Enchufa2]

All package managers install the latest package versions by default, and APT is no exception. So if you just add the testing repositories, the latest R will be pulled.

From what I've seen, the typical use of APT::Default-Release would be the other way around: to pin packages by default to an older release when you add repos from a newer one. I.e. I want r-base from testing, but I don't want the rest of the packages to be automatically upgraded to testing, so I add the testing repos but set APT::Default-Release to stable.

TL;DR, unless I'm missing something (please @eddelbuettel correct me if I'm wrong), I think you can just drop that line and everything will work as intended here.

[eddelbuettel]

Good call by @Enchufa2

I usually just work with 'testing' and may at times select into 'unstable' -- this iss what Rocker's r-base and derived containers do. I have the suspicion that forcing 'stable' may be the issue but I just don't know. There are scary many options to apt and I cannot claim to have explored all of them. But simply not impossing 'stable' may work.

FWIW I just fired up rocker/r-bspm:testing and asked it to install.r RcppArmadillo. With (default) options(bspm.version.check=TRUE) it noticed the newer source (0.12.6.1.0) and attempted to compiled it. When I ctrl-c'ed out and set options(bspm.version.check=FALSE) it picked the testing binary 0.12.4.1.0 just fine. (With direct use of apt I can override this on the fly too. )

This setup has APT::Default-Release "testing"; in /etc/apt/apt.conf.d/default.

[eitsupi]

After #204 merging, it seems working now.
As @Enchufa2 pointed out, it seems that APT::Default-Release "testing"; was the cause. thank you!