git-annex lost Amazon S3 support #60505

kousu · 2020-09-01T21:29:27Z

Bug report

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

ran brew update and can still reproduce the problem?
ran brew doctor, fixed all issues and can still reproduce the problem?
ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
Error: no logs ?
if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

brew config

    user@users-iMac homebrew-core % brew config
    HOMEBREW_VERSION: 2.4.16
    ORIGIN: https://github.com/Homebrew/brew
    HEAD: 2a4c9f610e69b9fed1eec61ac2c53e4b11a32e4c
    Last commit: 29 hours ago
    Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
    Core tap HEAD: 41a4db231ecd421c896b57ad802611c590ab33a2
    Core tap last commit: 15 minutes ago
    Core tap branch: master
    HOMEBREW_PREFIX: /usr/local
    HOMEBREW_CASK_OPTS: []
    HOMEBREW_MAKE_JOBS: 4
    CPU: quad-core 64-bit penryn
    Homebrew Ruby: 2.6.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
    Clang: 11.0 build 1103
    Git: 2.24.2 => /Library/Developer/CommandLineTools/usr/bin/git
    Curl: 7.64.1 => /usr/bin/curl
    macOS: 10.15.4-x86_64
    CLT: 11.4.1.0.1.1586360307
    Xcode: N/A
    user@users-iMac homebrew-core % brew doctor
    Your system is ready to brew.

What you were trying to do (and why)

Use git-annex to share scientific data.

What happened (include command output)

We have a dataset whose content is on AWS, via git-annex (https://github.com/spine-generic/data-multi-subject/); all the datasets in https://openneuro.org and most in https://portal.conp.ca are the same.

But since the middle of the month or so this has been happening:

Command output

julien-macbook:~/code/spine-generic/data-multi-subject $ git checkout -b jca/17-defacing
Switched to a new branch 'jca/17-defacing'
julien-macbook:~/code/spine-generic/data-multi-subject $ git remote
amazon
origin
julien-macbook:~/code/spine-generic/data-multi-subject $ 
julien-macbook:~/code/spine-generic/data-multi-subject $ gs
 M sub-cmrra02/anat/sub-cmrra02_T2w.nii.gz
 M sub-cmrra04/anat/sub-cmrra04_T2w.nii.gz
 M sub-cmrra05/anat/sub-cmrra05_T2w.nii.gz
 M sub-oxfordFmrib07/anat/sub-oxfordFmrib07_T1w.nii.gz
 M sub-stanford06/anat/sub-stanford06_T1w.nii.gz
julien-macbook:~/code/spine-generic/data-multi-subject $ gc
(recording state in git...)
[jca/17-defacing 4ea269a7] Fixed defacing in a few subjects
 5 files changed, 5 insertions(+), 5 deletions(-)
julien-macbook:~/code/spine-generic/data-multi-subject $ git annex sync --content amazon
git-annex: there is no available git remote named "amazon"
julien-macbook:~/code/spine-generic/data-multi-subject $ git remote
amazon
origin
julien-macbook:~/code/spine-generic/data-multi-subject $

and this:

julien-macbook:~/code/spine-generic/data-multi-subject $ git annex enableremote amazon
enableremote amazon 
git-annex: Unknown remote type S3 (pick from: git gcrypt p2p bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external)
failed
git-annex: enableremote: 1 failed

The most underlying issue is that git-annex has lost its S3 support. The telling symptom is that this doesn't say "S3" or "aws" anywhere:

git annex version                                         
git-annex version: 8.20200720.1
build flags: Assistant Webapp Pairing WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite
dependency versions: bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.6.5 http-client-0.7.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
operating system: darwin x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7

What you expected to happen

A working copy should say "build flags: ... S3 ..." and "dependency versions: aws-0.22":

git annex version
git-annex version: 8.20200720.1
build flags: Assistant Webapp Pairing S3 WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.6.5 http-client-0.7.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
operating system: darwin x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7

This was working for us a month ago.

Step-by-step reproduction instructions (by running `brew install` commands)

brew update
brew install git-annex
git annex version

The text was updated successfully, but these errors were encountered:

claui · 2020-09-01T21:35:14Z

Pinging fellow maintainer @chenrui333, who has authored #59330 and may be able to help.

kousu · 2020-09-01T22:21:54Z

Oh great, thanks for checking in so quick. I do indeed think it was #59330

log for git-annex.rb

user@users-iMac ~ % cd $(brew --repository)/Library/Taps/homebrew/homebrew-core
user@users-iMac homebrew-core % git log -p master -- Formula/git-annex.rb
commit 9591758fd8a565abf98d13ffd87c66e5e12ad579
Author: Nanda H Krishna <[email protected]>
Date:   Fri Aug 28 21:06:02 2020 +0530

    Add livecheck blocks from homebrew-livecheck

diff --git a/Formula/git-annex.rb b/Formula/git-annex.rb
index 1de6a6ccd6..8aa02aaeea 100644
--- a/Formula/git-annex.rb
+++ b/Formula/git-annex.rb
@@ -5,6 +5,10 @@ class GitAnnex < Formula
   sha256 "e631c9d52e440f80e9d305c95a078dcae71f200125bca91e49d5b8e2d864c6f3"
   head "git://git-annex.branchable.com/"
 
+  livecheck do
+    url :stable
+  end
+
   bottle do
     cellar :any
     sha256 "b2da7a293056e721b82cd6cfbc706203611321677ae78f084d24531df3423b04" => :catalina

commit 7a99e3cb8d541d62cdb9a423b6141db7b90cd682
Author: BrewTestBot <[email protected]>
Date:   Wed Aug 12 01:13:12 2020 +0000

    git-annex: update 8.20200810 bottle.

diff --git a/Formula/git-annex.rb b/Formula/git-annex.rb
index b4dfc9b309..1de6a6ccd6 100644
--- a/Formula/git-annex.rb
+++ b/Formula/git-annex.rb
@@ -7,9 +7,9 @@ class GitAnnex < Formula
 
   bottle do
     cellar :any
-    sha256 "f67c811fac6f043d7e07012868c314e1b96657c74d147c92504c4bc36f5fced5" => :catalina
-    sha256 "2fa4166aa125e2c0e620b18506ddd7649d4ae4baf8bc22bb9b29d167ca33a04a" => :mojave
-    sha256 "33d83e7d1dca6016fea3539c5f0daa6bbba831f7eaa3f69301d68f8da7b060d4" => :high_sierra
+    sha256 "b2da7a293056e721b82cd6cfbc706203611321677ae78f084d24531df3423b04" => :catalina
+    sha256 "cb3c028e6f987813c02dd7d1e3382721896ddfce92924776d3715e1912323766" => :mojave
+    sha256 "969571512b9bbf02b5e1492a0b36501b8d3050e1c243c322accfc18eeba4dbe2" => :high_sierra
   end
 
   depends_on "cabal-install" => :build

commit 7e90adf034c134b895fee3983f5ba2d017628843
Author: Nick Golder <[email protected]>
Date:   Tue Aug 11 06:55:22 2020 -0600

    git-annex 8.20200810
    
    Closes #59470.
    
    Signed-off-by: chenrui <[email protected]>
    Signed-off-by: BrewTestBot <[email protected]>

diff --git a/Formula/git-annex.rb b/Formula/git-annex.rb
index 7ee16b17e2..b4dfc9b309 100644
--- a/Formula/git-annex.rb
+++ b/Formula/git-annex.rb
@@ -1,9 +1,8 @@
 class GitAnnex < Formula
   desc "Manage files with git without checking in file contents"
   homepage "https://git-annex.branchable.com/"
-  url "https://hackage.haskell.org/package/git-annex-8.20200720.1/git-annex-8.20200720.1.tar.gz"
-  sha256 "d6252697151ab01cc8f7a492dc5eb4b4e773c0d48771df360c82a7cca7a39c3c"
-  revision 1
+  url "https://hackage.haskell.org/package/git-annex-8.20200810/git-annex-8.20200810.tar.gz"
+  sha256 "e631c9d52e440f80e9d305c95a078dcae71f200125bca91e49d5b8e2d864c6f3"
   head "git://git-annex.branchable.com/"
 
   bottle do

commit 6370ed052ee3ca595f77bbf5988689ddec04ff17
Author: chenrui <[email protected]>
Date:   Sat Aug 8 17:39:58 2020 +0000

    git-annex: update 8.20200720.1_1 bottle.

diff --git a/Formula/git-annex.rb b/Formula/git-annex.rb
index f3f5ee35e6..7ee16b17e2 100644
--- a/Formula/git-annex.rb
+++ b/Formula/git-annex.rb
@@ -8,9 +8,9 @@ class GitAnnex < Formula
 
   bottle do
     cellar :any
-    sha256 "0dc6cab32110246d38e26753141ac0772557cf303cf7a92f52edf1b4b2c9f625" => :catalina
-    sha256 "15cc50337736d3a405ca28665dc89e850e29d622a7233d0071249fa5fbe66eba" => :mojave
-    sha256 "215ae839c10f9fa39a236d0f7cd3bd9630b6c719a2d9e593bdd03b692c6bbc6e" => :high_sierra
+    sha256 "f67c811fac6f043d7e07012868c314e1b96657c74d147c92504c4bc36f5fced5" => :catalina
+    sha256 "2fa4166aa125e2c0e620b18506ddd7649d4ae4baf8bc22bb9b29d167ca33a04a" => :mojave
+    sha256 "33d83e7d1dca6016fea3539c5f0daa6bbba831f7eaa3f69301d68f8da7b060d4" => :high_sierra
   end
 
   depends_on "cabal-install" => :build

commit 23a4f9fb8513c805fba508f3b769075fb2e090b6
Author: Rui Chen <[email protected]>
Date:   Sat Aug 8 09:19:11 2020 -0400

    git-annex: deprecate cabal v1
    
    Closes #59330.
    
    Signed-off-by: chenrui <[email protected]>

diff --git a/Formula/git-annex.rb b/Formula/git-annex.rb
index 0523bb2604..f3f5ee35e6 100644
--- a/Formula/git-annex.rb
+++ b/Formula/git-annex.rb
@@ -1,12 +1,9 @@
-require "language/haskell"
-
 class GitAnnex < Formula
-  include Language::Haskell::Cabal
-
   desc "Manage files with git without checking in file contents"
   homepage "https://git-annex.branchable.com/"
   url "https://hackage.haskell.org/package/git-annex-8.20200720.1/git-annex-8.20200720.1.tar.gz"
   sha256 "d6252697151ab01cc8f7a492dc5eb4b4e773c0d48771df360c82a7cca7a39c3c"
+  revision 1
   head "git://git-annex.branchable.com/"
 
   bottle do
@@ -24,10 +21,8 @@ class GitAnnex < Formula
   depends_on "quvi"
 
   def install
-    install_cabal_package "--constraint", "http-conduit>=2.3",
-                          "--constraint", "network>=2.6.3.0",
-                          using: ["alex", "happy", "c2hs"],
-                          flags: ["s3", "webapp"]
+    system "cabal", "v2-update"
+    system "cabal", "v2-install", *std_cabal_v2_args
     bin.install_symlink "git-annex" => "git-annex-shell"
   end

The version before PR #59330, 8.20200720.1, is working:

user@users-iMac homebrew-core % git checkout -b git-annex-old 23a4f9fb8513c805fba508f3b769075fb2e090b6^ # note this is the step *BEFORE* the PR #59330
Updating files: 100% (2606/2606), done.
Switched to a new branch 'git-annex-old'
user@users-iMac homebrew-core % HOMEBREW_NO_AUTO_UPDATE=1 brew install git-annex                       
==> Downloading https://homebrew.bintray.com/bottles/git-annex-8.20200720.1.catalina.bottle.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/0dc6cab32110246d38e26753141ac0772557cf303cf7a92f52edf1b4b2c9f625?response-content-disposition=attachment%3Bfilename%3D%2
######################################################################## 100.0%
==> Pouring git-annex-8.20200720.1.catalina.bottle.tar.gz
==> Caveats
To have launchd start git-annex now and restart at login:
  brew services start git-annex
Or, if you don't want/need a background service you can just run:
  git annex assistant --autostart
==> Summary
🍺  /usr/local/Cellar/git-annex/8.20200720.1: 113 files, 42.5MB
user@users-iMac homebrew-core % git annex version
git-annex version: 8.20200720.1
build flags: Assistant Webapp Pairing S3 WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite
dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.6.5 http-client-0.7.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
operating system: darwin x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7

while the version before the next PR #59470, git-annex-8.20200720.1_1, is not:

user@users-iMac homebrew-core % git reset --hard 7e90adf034c134b895fee3983f5ba2d017628843^
Updating files: 100% (2576/2576), done.
HEAD is now at 65be49a3c0 helm: update 3.3.0 bottle.
user@users-iMac homebrew-core % brew uninstall git-annex                                  
Uninstalling /usr/local/Cellar/git-annex/8.20200810... (9 files, 53.3MB)
user@users-iMac homebrew-core % HOMEBREW_NO_AUTO_UPDATE=1 brew install git-annex          
==> Downloading https://homebrew.bintray.com/bottles/git-annex-8.20200720.1_1.catalina.bottle.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/f67c811fac6f043d7e07012868c314e1b96657c74d147c92504c4bc36f5fced5?response-content-disposition=attachment%3Bfilename%3D%2
######################################################################## 100.0%
==> Pouring git-annex-8.20200720.1_1.catalina.bottle.tar.gz
==> Caveats
To have launchd start git-annex now and restart at login:
  brew services start git-annex
Or, if you don't want/need a background service you can just run:
  git annex assistant --autostart
==> Summary
🍺  /usr/local/Cellar/git-annex/8.20200720.1_1: 9 files, 53.2MB
user@users-iMac homebrew-core % git annex version                                         
git-annex version: 8.20200720.1
build flags: Assistant Webapp Pairing WebDAV FsEvents TorrentParser MagicMime Feeds Testsuite
dependency versions: bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.6.5 http-client-0.7.1 persistent-sqlite-2.10.6.2 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.0.2
key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL
remote types: git gcrypt p2p bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs hook external
operating system: darwin x86_64
supported repository versions: 8
upgrade supported from repository versions: 0 1 2 3 4 5 6 7

I don't know about cabal but I can try to fix it.

For comparison, Arch uses the included Makefile instead of cabal, and makes sure to depend on haskell-aws:

https://github.com/archlinux/svntogit-community/blob/511d422f8aaa8ada1290994dc855c37fb1cacef9/trunk/PKGBUILD#L36-L41

kousu · 2020-09-01T22:48:18Z

This caused spine-generic/data-multi-subject#24 and spine-generic/data-multi-subject#33 and spine-generic/data-multi-subject#18. They're all actually the same, it's just that git-annex isn't that clear about reporting errors.

jonchang · 2020-09-11T05:53:10Z

Fixed in 548be03

This was referenced Sep 4, 2020

git-annex: readd S3 support. #60667

Closed

git-annex: there is no available git remote named "amazon" spine-generic/data-multi-subject#18

Closed

jonchang closed this as completed Sep 11, 2020

BrewTestBot added the outdated PR was locked due to age label Dec 1, 2020

Homebrew locked as resolved and limited conversation to collaborators Dec 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

git-annex lost Amazon S3 support #60505

git-annex lost Amazon S3 support #60505

kousu commented Sep 1, 2020 •

edited

Loading

claui commented Sep 1, 2020

kousu commented Sep 1, 2020 •

edited

Loading

kousu commented Sep 1, 2020 •

edited

Loading

jonchang commented Sep 11, 2020

git-annex lost Amazon S3 support #60505

git-annex lost Amazon S3 support #60505

Comments

kousu commented Sep 1, 2020 • edited Loading

Bug report

What you were trying to do (and why)

What happened (include command output)

What you expected to happen

Step-by-step reproduction instructions (by running brew install commands)

claui commented Sep 1, 2020

kousu commented Sep 1, 2020 • edited Loading

kousu commented Sep 1, 2020 • edited Loading

jonchang commented Sep 11, 2020

kousu commented Sep 1, 2020 •

edited

Loading

Step-by-step reproduction instructions (by running `brew install` commands)

kousu commented Sep 1, 2020 •

edited

Loading

kousu commented Sep 1, 2020 •

edited

Loading