Build fails to find many Ubuntu packages

[OutdoorsE]

Hi,
I'm trying to build docker-cartodb on RedHat 7, and the build fails early when it cannot find several supporting packages before it even gets to any of the main Carto components. What do I need to do to get docker-cartodb to build properly?

Error messages indicating which packages cannot be found:

E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/i/imagemagick/imagemagick-6-common_6.9.7.4+dfsg-16ubuntu6.3_all.deb  404  Not Found [IP: 91.189.88.161 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/i/imagemagick/libmagickcore-6.q16-3_6.9.7.4+dfsg-16ubuntu6.3_amd64.deb  404  Not Found [IP: 91.189.88.161 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/i/imagemagick/libmagickwand-6.q16-3_6.9.7.4+dfsg-16ubuntu6.3_amd64.deb  404  Not Found [IP: 91.189.88.161 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/l/linux/linux-libc-dev_4.15.0-34.37_amd64.deb  404  Not Found [IP: 91.189.88.161 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/m/mysql-5.7/libmysqlclient20_5.7.23-0ubuntu0.18.04.1_amd64.deb  404  Not Found [IP: 91.189.88.161 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/m/mysql-5.7/libmysqlclient-dev_5.7.23-0ubuntu0.18.04.1_amd64.deb  404  Not Found [IP: 91.189.88.161 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/git/git-man_2.17.1-1ubuntu0.1_all.deb  404  Not Found [IP: 91.189.88.161 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/git/git_2.17.1-1ubuntu0.1_amd64.deb  404  Not Found [IP: 91.189.88.161 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/i/imagemagick/imagemagick-6.q16_6.9.7.4+dfsg-16ubuntu6.3_amd64.deb  404  Not Found [IP: 91.189.88.161 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/i/imagemagick/imagemagick_6.9.7.4+dfsg-16ubuntu6.3_amd64.deb  404  Not Found [IP: 91.189.88.161 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/u/util-linux/uuid-dev_2.31.1-0.4ubuntu3.1_amd64.deb  404  Not Found [IP: 91.189.88.161 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Thanks for any guidance.

E

[sverhoeven]

Normally I get this error when apt-get update has not run in a while. apt tries to download old Debian packages that are not present anymore. Before the error does the docker build show something like

Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:4 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [6161 B]
Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [10.8 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [725 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [539 kB]
Get:13 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [2975 B]
Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [112 kB]
Get:15 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [1364 B]
Get:16 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [237 kB]
Fetched 15.0 MB in 1s (11.1 MB/s)`

[OutdoorsE]

Yes, the "Get" list starts like this:

Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 multiarch-support amd64 2.27-3ubuntu1 [6,916 B]
Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxau6 amd64 1:1.0.8-1 [8,376 B]
Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 libbsd0 amd64 0.8.7-1 [41.5 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxdmcp6 amd64 1:1.1.2-3 [10.7 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxcb1 amd64 1.13-1 [45.6 kB]

First error is "Get:25 (imagemagick-6-common)", and the rest are interspersed among others, ending at "Get:381".

I was under the impression that apt-get update is one of the first things to run via the Dockerfile. Is that something I need to run within the container somehow before trying the build again?

[sverhoeven]

It does, just checking that apt update actually ran OK for you. For me the Docker build is running successfully. I suspect it was timing issue.

Could you try to build again by first pulling the from image with docker pull ubuntu:18.04 and build the Carto image without cache with docker build -t=sverhoeven/cartodb --no-cache .?

[OutdoorsE]

OK, I tried running those 2 commands (from /cartodb/src/docker-cartodb, which is where I have the github clone). The build got much farther but failed while working on crankshaft and related packages. Here are the last several lines of output:

Installing collected packages: joblib, numpy, scipy, pysal, scikit-learn, crankshaft
  Found existing installation: numpy 1.13.3
    Uninstalling numpy-1.13.3:
Could not install packages due to an EnvironmentError: [Errno 39] Directory not empty: '/usr/lib/python2.7/dist-packages/numpy/compat'

make[1]: *** [install] Error 1
Makefile:5: recipe for target 'install' failed
make[1]: Leaving directory '/crankshaft/src/py'
Makefile:16: recipe for target 'install' failed
make: *** [install] Error 2
The command '/bin/sh -c cd / &&     curl https://bootstrap.pypa.io/get-pip.py | python &&     git clone https://github.com/CartoDB/crankshaft.git &&     cd /crankshaft &&     git checkout $CRANKSHAFT_VERSION &&     make install &&     pip install --force-reinstall --no-cache-dir scikit-learn==0.14.1 &&     cd ..' returned a non-zero code: 2

Scrolling back through the output, I'm also seeing a lot of warnings like this:

infback.c:471:25: warning: this statement may fall through [-Wimplicit-fallthrough=]
             state->mode = LEN;
             ~~~~~~~~~~~~^~~~~
infback.c:473:9: note: here
         case LEN:
         ^~~~

and this:

/bin/bash ../../libtool  --tag=CC   --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../..  -DMALLOC_PRODUCTION  -D__isthreaded=true -g -O2 -pthread -Wextra -Wno-missing-field-initializers -Wno-sign-compare -MT libjemalloc_mt_la-jemalloc_linux.lo -MD -MP -MF .deps/libjemalloc_mt_la-jemalloc_linux.Tpo -c -o libjemalloc_mt_la-jemalloc_linux.lo `test -f 'jemalloc_linux.c' || echo './'`jemalloc_linux.c
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -DMALLOC_PRODUCTION -D__isthreaded=true -g -O2 -pthread -Wextra -Wno-missing-field-initializers -Wno-sign-compare -MT libjemalloc_mt_la-jemalloc_linux.lo -MD -MP -MF .deps/libjemalloc_mt_la-jemalloc_linux.Tpo -c jemalloc_linux.c  -fPIC -DPIC -o .libs/libjemalloc_mt_la-jemalloc_linux.o
jemalloc_linux.c: In function ‘wrtmessage’:
jemalloc_linux.c:1242:2: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
  write(STDERR_FILENO, p1, strlen(p1));
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[OutdoorsE]

I'm new to both docker and ubuntu so am not sure what to try next.

Web search turned up a discussion of a similar directory not empty error where pip failed in a docker build, and the problem was related to the overlay filesystem and was eventually corrected in a newer kernel version. I'm wondering if my build failure is related to a similar problem with the platform I'm using. Much of Red Hat 7's software is older than the bleeding-edge packages preferred on Ubuntu, which makes it very difficult to install Carto natively on Red Hat. That's why I'm now trying the docker approach. Here is some platform info in case it is helpful....

$ uname -a
Linux cartodb.uvm.edu 3.10.0-862.14.4.el7.x86_64 #1 SMP Fri Sep 21 09:07:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ sudo docker info
...
Containers: 3
 Running: 0
 Paused: 0
 Stopped: 3
Images: 92
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: false
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 5eda6f6fd0c2884c2c8e78a6e7119e8d0ecedb77 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
 selinux
Kernel Version: 3.10.0-862.14.4.el7.x86_64
Operating System: Red Hat Enterprise Linux
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 4
Total Memory: 15.5 GiB
Name: cartodb.uvm.edu
ID: 6ZOS:HIHN:KXQT:RWFS:JGVU:F2PS:NR62:U6UO:YEFB:UL2O:Z6TT:TJWT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://registry.access.redhat.com/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: registry.access.redhat.com (secure), docker.io (secure)