solana-test-validator fails by default on macOS, as hardened_unpack() expects GNU tar

[mikemaccana]

Problem

So far 18 people on Solana Stackexchange have encountered this error. On a default install of MacOS, BSD tar is the default:

$ tar --help | grep bsdtar
tar(bsdtar): manipulate archive files

Running solana-test-validator will fail with:

[2024-01-02T20:08:25.911047000Z INFO  solana_accounts_db::hardened_unpack] Extracting "test-ledger/genesis.tar.bz2"...
Error: failed to start validator: Failed to create ledger at test-ledger: blockstore error

A lot of mac users have Apple's 'Xcode Command Line Tools' installed which provides git, grep, less, cat, etc but does not include GNU tar. Additionally 'blockstore error' is a bit vague.

Proposed Solution

Ideal solution

Avoid the GNU tar dependendency. Either detect and work with BSD tar command out of the box, or use a rust implementation of tar.

Less ideal solution

Detect BSD tar and ask users to install GNU tar.

[mikemaccana]

See anza-xyz#2840

[codertjay]

This is because the BSD tar program is not compatible with the GNU tar program.

To fix it: you need to install and export the executable path of your GNU tar program in your .zshrc file.

Mac with Apple Silicon

brew install gnu-tar

Put this in ~/.zshrc

export PATH="/opt/homebrew/opt/gnu-tar/libexec/gnubin:$PATH"

Intel-based Mac

brew install gnu-tar

Put this in ~/.zshrc

export PATH="/usr/local/opt/gnu-tar/libexec/gnubin:$PATH"
the only reason there are separate exports for intel-based macs and macs with apple silicon is that, by default, homebrew executables are stored at different locations for macs with respective chipset.

[mikemaccana]

Looks like this is being worked on, see anza-xyz#2840

[steviez]

I think this issue was actually resolved with #35213 which first landed in the v2.0 branch. As noted in anza-xyz#3079, the issue seems to show on an Apple M1, but not Apple M3 despite the M1 and M3 having the same OS / bsdtar / supporting libs. This doesn't really make sense to me, but that is a rabbit hole that I'm not sure I want to jump down at the moment

Can anyone who has encountered the issue please try with a v2.0 version ?

[ferric-sol]

not the OP but just wanted to report that on my M1 Pro it is as @steviez said, 1.18.25 fails but 2.0.13 works perfectly fine

[steviez]

Going to close this one out. Like I said above, #35213 resolved this in the v2.0 line for agave; the change just never landed in solana labs or agave v1.18.

If someone still runs into this with agave v2.0, happy to continue the conversation.