Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alpine 3.14 images can fail on Docker versions older than 20.10 #351

Closed
stanhu opened this issue Jul 9, 2021 · 21 comments
Closed

Alpine 3.14 images can fail on Docker versions older than 20.10 #351

stanhu opened this issue Jul 9, 2021 · 21 comments
Labels
question Usability question, not directly related to an error with the image

Comments

@stanhu
Copy link

stanhu commented Jul 9, 2021

We run a slightly old Container-Optimized OS from Google (13310.1041.24) on thousands of Google N2D machines, and after the latest updates using Alpine v3.14, we noticed all sorts of strange failures:

cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:3.0.2-alpine3.13 bundle --version
Bundler version 2.2.22
cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:3.0.2-alpine3.14 bundle --version
`/root` is not writable.
Bundler will use `/tmp/bundler20210709-1-4lza5z1' as your home directory temporarily.
Bundler version 2.2.22
cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:2.7
irb(main):001:0>
cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:2.7-alpine
Traceback (most recent call last):
	6: from /usr/local/bin/irb:23:in `<main>'
	5: from /usr/local/bin/irb:23:in `load'
	4: from /usr/local/lib/ruby/gems/2.7.0/gems/irb-1.2.6/exe/irb:11:in `<top (required)>'
	3: from /usr/local/lib/ruby/2.7.0/irb.rb:393:in `start'
	2: from /usr/local/lib/ruby/2.7.0/irb/init.rb:18:in `setup'
	1: from /usr/local/lib/ruby/2.7.0/irb/init.rb:121:in `init_error'
/usr/local/lib/ruby/2.7.0/irb/locale.rb:121:in `load': No such file to load -- irb/error.rb (LoadError)
cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:3.0.2-alpine3.13
irb(main):001:0>
cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:3.0.2-alpine3.14
/usr/local/lib/ruby/3.0.0/irb/locale.rb:121:in `load': No such file to load -- irb/error.rb (LoadError)
	from /usr/local/lib/ruby/3.0.0/irb/init.rb:195:in `init_error'
	from /usr/local/lib/ruby/3.0.0/irb/init.rb:18:in `setup'
	from /usr/local/lib/ruby/3.0.0/irb.rb:402:in `start'
	from /usr/local/lib/ruby/gems/3.0.0/gems/irb-1.3.5/exe/irb:11:in `<top (required)>'
	from /usr/local/bin/irb:23:in `load'
	from /usr/local/bin/irb:23:in `<main>'

https://alpinelinux.org/posts/Alpine-3.14.0-released.html mentions:

The faccessat2 syscall has been enabled in musl. This can result in issues on docker hosts with older versions of docker (<20.10.0) and libseccomp (<2.4.4), which blocks this syscall.

Our version:

cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker --version
Docker version 19.03.9, build 9d98839

Can we rollback the Alpine v3.14 upgrade and make this opt-in instead of the default while we work to upgrade our systems?

I should note this problem did not occur on Google's supported cos-85-13310-1260-2 image:

image

Related links:

  1. https://bugzilla.redhat.com/show_bug.cgi?id=1900021
  2. Docker fix: Add openat2 and faccessat2 to default seccomp profile. moby/moby#41353
  3. Backport to Docker 19.03.16: [19.03 backport] seccomp profile updates moby/moby#41381
@stanhu stanhu changed the title Alpine 3.14 images can fail badly on certain Docker hosts Alpine 3.14 images can fail on Docker versions older than 19.03.15 Jul 9, 2021
@stanhu stanhu changed the title Alpine 3.14 images can fail on Docker versions older than 19.03.15 Alpine 3.14 images can fail on Docker versions older than 20.10 Jul 9, 2021
@J0WI
Copy link
Contributor

J0WI commented Jul 9, 2021

Can we rollback the Alpine v3.14 upgrade and make this opt-in instead of the default while we work to upgrade our systems?

just use the alpine3.13 tag

@jamesbrooks
Copy link

@stanhu thank you for this, I ran into this issue a few hours before this post and was finally at the part of looking at the alpine change as the cause (issues running containers on ECS Fargate) but am glad to see you having the same issue.

I'll use the 3.13 tag as @J0WI suggests.

Thanks!

@stanhu
Copy link
Author

stanhu commented Jul 9, 2021

just use the alpine3.13 tag

We run a CI service where users can choose any tag, and existing builds that use ruby:2.7-alpine or other builds abruptly have stopped working. We're now trying to upgrade the OS, but this issue seems like it could burn others.

@wglambert wglambert added the question Usability question, not directly related to an error with the image label Jul 9, 2021
@wglambert
Copy link

From another alpine-bug issue docker-library/php#1177 (comment)

The more specific aliases exist specifically to give users an escape hatch to stay on a specific underlying distribution release and control their own consumption and upgrade timing. Just like users using :latest, users using a bare alpine tag should expect some degree of breakage over time, and I don't think there's much we can reasonably do about that.

(This is also the common pattern of usage/tagging across all the official images -- PHP is not unique in this regard.)

I think the only thing unique about this particular bump is that the most common breakage is more disruptive than usual, but there's not really much we can do about that, given it's a change in musl/Alpine itself (https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0#faccessat2).

@simmerz
Copy link

simmerz commented Jul 12, 2021

We're running into this same issue with Docker 20.10.2 running on Ubuntu LTS. Docker is installed from the Docker ubuntu repo.

Clearly rolling down to 3.13 "solves" things, but these seem like broken 3.14 images to me.

EDIT: Resolved by upgrading docker-ce to 20.10.7.

@morgoth
Copy link

morgoth commented Jul 12, 2021

The same problem with alpine 3.14 on Fargate.
@jamesbrooks If you find a way to use the 3.14 on Fargate, please let us know.

@samnang
Copy link

samnang commented Jul 19, 2021

This problem happens to me on AWS Fargate (1.4) with ruby:2.7-alpine (1.14). It works when downgraded to ruby:2.7-alpine3.13.

@alexbrebu
Copy link

alexbrebu commented Jul 27, 2021

This problem happens to me on macOS Big Sur (Version 11.5.1) with ruby:2.7.4-alpine. Although, my colleagues do not encounter the same issue in Ubuntu.


Docker version 20.10.7, build f0df35096d
docker-compose version 1.29.2, build unknown
docker-machine version 0.16.2, build bd45ab1
Dinghy 4.6.5


Gemfile

source "https://rubygems.org"
git_source(:github) { |repo| "https://github.com/#{repo}.git" }

ruby "2.7.3"

gem "bootsnap", ">= 1.4.4", require: false
gem "puma", "~> 5.0"
gem "rails", "~> 6.1.4"
gem "rack-cors", "~> 1.1"

gem "active_interaction", "~> 4.0"

group :database, :default do
  gem "discard", "~> 1.2"
  gem "pg", "~> 1.1"
  gem "strong_migrations", "~> 0.7"
end

group :graphql, :default do
  # TODO: manage graphiql through docker specific image
  gem "graphql", "~> 1.12"
  gem "graphql-batch"
  gem "graphql-rails_logger", "~> 1.2"
end

group :development do
  gem "annotate"
  gem "listen", "~> 3.3"
  gem "spring"
end

group :test do
  gem "factory_bot_rails"
  gem "faker"
  gem "rspec-rails"
  gem "rswag-specs"
  gem "simplecov"
  gem "spring-commands-rspec"
end

group :code_quality, :development, :test do
  gem "bullet"
  gem "byebug"
  gem "pry"
  gem "pry-byebug"
  gem "pry-rails"
  gem "reek", require: false
  gem "rubocop", require: false
  gem "rubocop-graphql", require: false
  gem "rubocop-performance", require: false
  gem "rubocop-rails", require: false
  gem "rubocop-rspec", require: false
  gem "stackprof", require: false
end

group :instrumentation, :default do
  gem "ddtrace", require: "ddtrace/auto_instrument"
  gem "sentry-rails", "~> 4.2"
end


I have created this Dockerfile:

FROM ruby:2.7.4-alpine

RUN apk add build-base tzdata postgresql-dev postgresql-client

WORKDIR /usr/src/app
COPY Gemfile* ./
RUN bundle install

# Add a script to be executed every time the container starts.
COPY entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/entrypoint.sh
ENTRYPOINT ["entrypoint.sh"]
EXPOSE 3000

COPY . .

CMD ["rails", "server", "-b", "0.0.0.0"]

The error occurs when running the bundle install:

Building api
Sending build context to Docker daemon  438.8kB
Step 1/11 : FROM ruby:2.7.4-alpine
 ---> edd3dc5af692
Step 2/11 : RUN apk add build-base tzdata postgresql-dev postgresql-client
 ---> Using cache
 ---> 7370fb782090
Step 3/11 : WORKDIR /usr/src/app
 ---> Running in 4f66666b84ac
Removing intermediate container 4f66666b84ac
 ---> 20cc36d2f120
Step 4/11 : COPY Gemfile* ./
 ---> 63eee2b21da9
Step 5/11 : RUN bundle install
 ---> Running in e13ccb87957b
`/root` is not writable.
Bundler will use `/tmp/bundler20210727-1-1x1ft1f1' as your home directory temporarily.
Fetching gem metadata from https://rubygems.org/..........
Fetching rake 13.0.6
Installing rake 13.0.6
Gem::FilePermissionError: You don't have write permissions for the
/usr/local/bundle directory.
An error occurred while installing rake (13.0.6), and Bundler cannot continue.
Make sure that `gem install rake -v '13.0.6' --source 'https://rubygems.org/'`
succeeds before bundling.

In Gemfile:
  annotate was resolved to 3.1.1, which depends on
    rake
The command '/bin/sh -c bundle install' returned a non-zero code: 5
ERROR: Service 'api' failed to build : Build failed

no-reply pushed a commit to surfliner/surfliner-mirror that referenced this issue Aug 3, 2021
currently 3.14 breaks builds with older versions of docker: docker-library/ruby#351
@rohanmangal
Copy link

The same problem with alpine 3.14 on Fargate.
@jamesbrooks If you find a way to use the 3.14 on Fargate, please let us know.

This problem happens to me on AWS Fargate (1.4) with ruby:2.7-alpine (1.14). It works when downgraded to ruby:2.7-alpine3.13.

@morgoth @samnang > seems to be working on Fargate 1.4.0 now, tested with ruby:alpine

@Hello71
Copy link

Hello71 commented Aug 15, 2021

please see the alpine linux 3.14 release notes: https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0#faccessat2. you should always read the release notes when upgrading linux distro, or expect surprises.

if, after reading the release notes, it is still not working for anybody, please post:

  1. docker version (not docker --version) and docker info. if you are using hosted Docker, then your provider name and relevant information (e.g. advertised Linux distro and version).
  2. full docker invocation, input (e.g. Dockerfile or stdin), and output

to avoid cluttering the issue, please use <details> element for big output, e.g. https://gist.github.com/ericclemmons/b146fe5da72ca1f706b2ef72a20ac39d.

@alexbrebu see above, and if you are using docker desktop, what version?

@stanhu, docker 19.03 is officially EOL. however, you can continue using it (at your own risk) if you upgrade to runc v1.0.0-rc93 or apply the documented workaround. or, if you don't care about^W^W^Wdon't need security at the OS level, which is the only decent reason to be running without ten months of security updates, then just use --security-opt=seccomp=unconfined?

although i don't maintain this image, i don't think it is a good idea to downgrade the version. it would be unexpected for users to suddenly get a lower version, as well as a version not matching the latest version advertised on the alpine linux website. additionally, some people may be using the community repo, which is not supported beyond the latest alpine linux release.

@alexbrebu
Copy link

.

please see the alpine linux 3.14 release notes: https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0#faccessat2. you should always read the release notes when upgrading linux distro, or expect surprises.

if, after reading the release notes, it is still not working for anybody, please post:

  1. docker version (not docker --version) and docker info. if you are using hosted Docker, then your provider name and relevant information (e.g. advertised Linux distro and version).
  2. full docker invocation, input (e.g. Dockerfile or stdin), and output

to avoid cluttering the issue, please use <details> element for big output, e.g. https://gist.github.com/ericclemmons/b146fe5da72ca1f706b2ef72a20ac39d.

@alexbrebu see above, and if you are using docker desktop, what version?

@stanhu, docker 19.03 is officially EOL. however, you can continue using it (at your own risk) if you upgrade to runc v1.0.0-rc93 or apply the documented workaround. or, if you don't care about^W^W^Wdon't need security at the OS level, which is the only decent reason to be running without ten months of security updates, then just use --security-opt=seccomp=unconfined?

although i don't maintain this image, i don't think it is a good idea to downgrade the version. it would be unexpected for users to suddenly get a lower version, as well as a version not matching the latest version advertised on the alpine linux website. additionally, some people may be using the community repo, which is not supported beyond the latest alpine linux release.

I have installed the docker through homebrew

Brebus-MacBook-Pro:~ brebuandrei$ docker -v
Docker version 20.10.7, build f0df35096d
Brebus-MacBook-Pro:~ brebuandrei$ which docker
/usr/local/bin/docker

@tianon
Copy link
Member

tianon commented Aug 20, 2021

Closing because this isn't something we can fix in the images -- the proper solution is to either use the alpine3.13 images or update Docker, runc, and/or libseccomp (with the latter being the better option long term).

@dhnaranjo
Copy link

This does appear to be an issue on Azure App Service. Downgrading to Alpine 3.13 resolved the issue for me. Just leaving some breadcrumbs for anyone who finds this thread with the same issue.

@alexbrebu
Copy link

alexbrebu commented Oct 15, 2021

I solved this issue by replacing dinghy setup and docker (installed with homebrew) with Docker for Mac.

Step 1

dinghy destroy
brew uninstall docker-machine docker-compose dinghy docker docker-machine-nfs

Step 2
Check your configuration file and make sure you clean DOCKER_* envs if any

cat ~/.bash_profile
cat ~/.bashrc

@KashubaK
Copy link

I was running into an issue where installing a gem from a Git repo source failed. I would get the following error when running bundle install:

You need to install git to be able to use gems from git repositories

bundle exec git version would also fail, saying the command was not found.

Updating docker via sudo apt-get install docker-ce docker-ce-cli resolved the issue. Posting this just in case one other person has a similar problem.

Obsiye added a commit to ministryofjustice/laa-court-data-ui that referenced this issue Feb 8, 2022
This is an issue with our docker alpine image and an older docker host. Keeping to apline3.13 fixes the below error
Fix for :
While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for/usr/local/bundle

Solution:
docker-library/ruby#351
Obsiye added a commit to ministryofjustice/laa-court-data-ui that referenced this issue Feb 9, 2022
* Add cypress-axe and axe-core

Add cypress-axe as a yarn devDependency
Add axe-core as a yarn peerDependency of the cypress package

Add import 'cypress-axe' to the support file as it includes commands to be used in testing

* Added basic usage of checkA11y
Cypress test for sign in spec checks for accessibility issues on page visit

To do: Refactor so that we don't have to reproduce long task code on every spec

* Fix js linting issues

* Add Cypress command to log results to terminal

Add a terminal log command that logs cypress test results to the terminal.

Refactor code to move the function terminalLog into the above command.

Add initial rules to ignore the govuk-phase-banner accessiblity issue because it is Technial Debt

* Add accessiblity cypress checks on sign in

Add accessibility checks to the page load and dynamic changes on the page during the sign in cypress tests

* Added accessibility tests to cookie tests
Added axe a11y checks to cookie_settings and cookie_banner
Refactored runtime a11y checks into function composed of:
checkA11y and injectAxe for ease of use

* Added eslint for Javascript
Created config for eslint
Added Cypress eslint plugin
Linted all JS files in repo
Cleaned up test specs

* Changed validate:js command to ESLint
Ran linter on via command on app and cypress folder

* Update comment to ESLint from standardjs

* Upgrade ruby to 2.7.5

Upgrade ruby to 2.7.5 from 2.7.2

Upgrade circleci ruby docker imgae to 2.7.5 which comes with node version 16.13.0

This is so that the circleci docker image has the node of at least (^12.22.0, ^14.17.0, or >=16.0.0)  as described by ESlint.

This will allow for ESlint to be installed and work as our javascript linter

* Disable eslint error for cypress plugin/index.js

* Upgrade ruby to 2.7.5 in Dockerfile

* Fix bundle gems issue in docker

This is an issue with our docker alpine image and an older docker host. Keeping to apline3.13 fixes the below error
Fix for :
While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for/usr/local/bundle

Solution:
docker-library/ruby#351

* Configure Cypress Axe impact
Disabled checks for minor and moderate issues (only check for serious or critical)
Modified custom a11yCheck command to handle config
Applied injectAxe to all a11yChecks to avoid losing the instantiation (calling the function multiple times has no impact)

Co-authored-by: ivanELEC <[email protected]>
@luke-hill
Copy link

luke-hill commented Mar 1, 2022

Apologies for bumping a phantom thread (Although it's not that old).

The advice I found out through different mediums (Use alpine 3.13), works.... However .....

Ruby 3.1.x series in dockerhub doesn't feature an alpine 3.13 build combination. As such this issue now prohibits people from using Ruby 3.1.x

I personally get this issue when using Docker in Docker. And have raised a few issues (Probably in the wrong places)

Any advice from people on next steps? Asking whoever maintains said images under ruby:xxxx in docker hub to promote / create an alpine 3.13 one would fix the issue for 1year in theory and feels like a reasonably simple set of maintenance steps to give an immediate fix?

@tianon
Copy link
Member

tianon commented Mar 1, 2022

The correct solution here is to update Docker and libseccomp on your host.

@luke-hill
Copy link

is libseccomp a package. So something like apt-get libseccomp?

@tianon
Copy link
Member

tianon commented Mar 2, 2022

On Debian (and derivatives like Raspbian, Ubuntu, etc) the package name is libseccomp2.

@luke-hill
Copy link

FYI This doesn't fix the issue. But I've raised it elsewhere.

@David-Mysize
Copy link

=> CANCELED [internal] load build context 18.1s
=> => transferring context: 494.25MB 18.1s
=> [1/8] FROM docker.io/library/ruby:2.6-alpine@sha256:382ce92de42ef027bf1bfe382c3f3c3878042c41c07da8b8aa41855db0894762 0.0s
=> CACHED [2/8] RUN apk add git && apk add --update --no-cache jq curl firefox-esr xvfb mesa-dev mesa-gl chromium-chromedriver chromium build-ba 0.0s
=> ERROR [3/8] RUN gem install rails -v '5.2.4' 18.1s

[3/8] RUN gem install rails -v '5.2.4':
#6 17.73 ERROR: While executing gem ... (Gem::FilePermissionError)
#6 17.73 You don't have write permissions for the /usr/local/bundle directory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Usability question, not directly related to an error with the image
Projects
None yet
Development

No branches or pull requests