PyCall build error owing to JSONDecodeError

[SimonEnsemble]

was working a few days ago, but after an up, build PyCall fails, with the following build log.

Collecting package metadata (current_repodata.json): ...working... failed

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/exceptions.py", line 1132, in __call__
        return func(*args, **kwargs)
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/cli/main.py", line 69, in main_subshell
        exit_code = do_call(args, p)
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/cli/conda_argparse.py", line 122, in do_call
        return getattr(module, func_name)(args, parser)
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/notices/core.py", line 121, in wrapper
        return func(*args, **kwargs)
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/cli/main_install.py", line 20, in execute
        install(args, parser, 'install')
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/cli/install.py", line 264, in install
        unlink_link_transaction = solver.solve_for_transaction(
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/core/solve.py", line 134, in solve_for_transaction
        unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/core/solve.py", line 177, in solve_for_diff
        final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/core/solve.py", line 282, in solve_final_state
        ssc = self._collect_all_metadata(ssc)
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/common/io.py", line 84, in decorated
        return f(*args, **kwds)
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/core/solve.py", line 449, in _collect_all_metadata
        index, r = self._prepare(prepared_specs)
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/core/solve.py", line 1064, in _prepare
        reduced_index = get_reduced_index(self.prefix, self.channels,
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/core/index.py", line 267, in get_reduced_index
        new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs,
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/core/subdir_data.py", line 169, in query_all
        result = tuple(chain.from_iterable(executor.map(subdir_query, channel_urls)))
      File "/Users/cokes/.julia/conda/3/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
        yield _result_or_cancel(fs.pop())
      File "/Users/cokes/.julia/conda/3/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
        return fut.result(timeout)
      File "/Users/cokes/.julia/conda/3/lib/python3.10/concurrent/futures/_base.py", line 451, in result
        return self.__get_result()
      File "/Users/cokes/.julia/conda/3/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
        raise self._exception
      File "/Users/cokes/.julia/conda/3/lib/python3.10/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/core/subdir_data.py", line 158, in subdir_query
        return tuple(
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/core/subdir_data.py", line 174, in query
        self.load()
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/core/subdir_data.py", line 264, in load
        _internal_state = self._load()
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/core/subdir_data.py", line 315, in _load
        cache.load_state()  # XXX should this succeed even if FileNotFound?
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/gateways/repodata/__init__.py", line 570, in load_state
        self.load(state_only=True)
      File "/Users/cokes/.julia/conda/3/lib/python3.10/site-packages/conda/gateways/repodata/__init__.py", line 528, in load
        state = json.loads(state_file.read())
      File "/Users/cokes/.julia/conda/3/lib/python3.10/json/__init__.py", line 346, in loads
        return _default_decoder.decode(s)
      File "/Users/cokes/.julia/conda/3/lib/python3.10/json/decoder.py", line 340, in decode
        raise JSONDecodeError("Extra data", s, end)
    json.decoder.JSONDecodeError: Extra data: line 9 column 2 (char 289)

`$ /Users/cokes/.julia/conda/3/bin/conda install -y numpy`

  environment variables:
                 CIO_TEST=<not set>
                  CONDARC=/Users/cokes/.julia/conda/3/condarc-julia.yml
             CONDA_PREFIX=/Users/cokes/.julia/conda/3
               CONDA_ROOT=/Users/cokes/.julia/conda/3
           CURL_CA_BUNDLE=<not set>
                 GEM_PATH=/Users/cokes/.gem/ruby/3.1.3:/Users/cokes/.rubies/ruby-
                          3.1.3/lib/ruby/gems/3.1.0
                 INFOPATH=/opt/homebrew/share/info:
          JULIA_LOAD_PATH=@:/var/folders/b3/p558slm10qj7tzmkx22rhyc80000gn/T/jl_YR0kPl
               LD_PRELOAD=<not set>
                  MANPATH=/opt/homebrew/share/man::
                     PATH=/Users/cokes/.gem/ruby/3.1.3/bin:/Users/cokes/.rubies/ruby-
                          3.1.3/lib/ruby/gems/3.1.0/bin:/Users/cokes/.rubies/ruby-
                          3.1.3/bin:/Users/cokes/opt/anaconda3/bin:/Users/cokes/opt/anaconda3/co
                          ndabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cry
                          ptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/
                          usr/local/go/bin:/var/run/com.apple.security.cryptexd/codex.system/boo
                          tstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system
                          /bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/b
                          ootstrap/usr/appleinternal/bin
         PYTHONIOENCODING=UTF-8
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>

     active environment : base
    active env location : /Users/cokes/.julia/conda/3
       user config file : /Users/cokes/.condarc
 populated config files : /Users/cokes/.julia/conda/3/.condarc
                          /Users/cokes/.condarc
                          /Users/cokes/.julia/conda/3/condarc-julia.yml
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.10.8.final.0
       virtual packages : __archspec=1=arm64
                          __osx=13.3.1=0
                          __unix=0=0
       base environment : /Users/cokes/.julia/conda/3  (writable)
      conda av data dir : /Users/cokes/.julia/conda/3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/cokes/.julia/conda/3/pkgs
                          /Users/cokes/.conda/pkgs
       envs directories : /Users/cokes/.julia/conda/3/envs
                          /Users/cokes/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.3.1 requests/2.28.1 CPython/3.10.8 Darwin/22.4.0 OSX/13.3.1
                UID:GID : 501:20
             netrc file : None
           offline mode : False

An unexpected error has occurred. Conda has prepared the above report.

Upload successful.
┌ Info: Using the Python distribution in the Conda package by default.
└ To use a different Python version, set ENV["PYTHON"]="pythoncommand" and re-run Pkg.build("PyCall").
[ Info: Running `conda install -y numpy` in root environment
ERROR: LoadError: failed process: Process(setenv(`/Users/cokes/.julia/conda/3/bin/conda install -y numpy`,["_CE_M=", "PATH=/Users/cokes/.gem/ruby/3.1.3/bin:/Users/cokes/.rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/bin:/Users/cokes/.rubies/ruby-3.1.3/bin:/Users/cokes/opt/anaconda3/bin:/Users/cokes/opt/anaconda3/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/go/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin", "MANPATH=/opt/homebrew/share/man::", "ITERM_PROFILE=Default", "USER=cokes", "RUBY_ENGINE=ruby", "RUBYOPT=", "SHELL=/bin/zsh", "LC_TERMINAL_VERSION=3.4.19", "COLORFGBG=7;0"  …  "SECURITYSESSIONID=186a3", "TERM_PROGRAM_VERSION=3.4.19", "COMMAND_MODE=unix2003", "PWD=/Users/cokes/.julia/packages", "RUBY_ROOT=/Users/cokes/.rubies/ruby-3.1.3", "TERM_PROGRAM=iTerm.app", "GEM_HOME=/Users/cokes/.gem/ruby/3.1.3", "CONDA_PREFIX=/Users/cokes/.julia/conda/3", "TERM_SESSION_ID=w0t7p0:B4D22DC8-2AB8-4DAA-96B7-7F7B563EE985", "JULIA_LOAD_PATH=@:/var/folders/b3/p558slm10qj7tzmkx22rhyc80000gn/T/jl_YR0kPl"]), ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:565 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:480
 [3] run
   @ ./process.jl:477 [inlined]
 [4] runconda(args::Cmd, env::String)
   @ Conda ~/.julia/packages/Conda/kOnIE/src/Conda.jl:128
 [5] add(pkg::String, env::String; channel::String)
   @ Conda ~/.julia/packages/Conda/kOnIE/src/Conda.jl:224
 [6] add (repeats 2 times)
   @ ~/.julia/packages/Conda/kOnIE/src/Conda.jl:222 [inlined]
 [7] top-level scope
   @ ~/.julia/packages/PyCall/twYvK/deps/build.jl:79
 [8] include(fname::String)
   @ Base.MainInclude ./client.jl:476
 [9] top-level scope
   @ none:5
in expression starting at /Users/cokes/.julia/packages/PyCall/twYvK/deps/build.jl:43

version info

• PyCall v1.95.1
• Conda v1.8.0

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.5.0)
  CPU: 8 × Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 1 on 4 virtual cores

[SimonEnsemble]

oddly, setting ENV["PYTHON"] = "/Users/cokes/opt/anaconda3/bin/python" (output of which python), then build PyCall fails too. 👀 '

UPDATE: solved by (i) uninstalling anaconda, (ii) reinstalling proper anaconda for my M1 arch. 😄 above is still an issue tho, but using my own Python installation and in business now! 🚀

ERROR: Error building `PyCall`:
DEBUG is_windows = False
DEBUG is_apple = True
DEBUG Candidate: None
DEBUG Not found.
DEBUG Candidate: /Users/cokes/opt/anaconda3/lib/python3.9/config-3.9-darwin/libpython3.9.a
DEBUG Not found.
DEBUG Candidate: /Users/cokes/opt/anaconda3/lib/python3.9/config-3.9-darwin/libpython3.9.dylib
DEBUG Not found.
DEBUG Candidate: /Users/cokes/opt/anaconda3/lib/python3.9/config-3.9-darwin/libpython3.dylib
DEBUG Not found.
DEBUG Candidate: /Users/cokes/opt/anaconda3/lib/python3.9/config-3.9-darwin/libpython.dylib
DEBUG Not found.
DEBUG Candidate: /Users/cokes/opt/anaconda3/lib/libpython3.9.a
DEBUG Not found.
DEBUG Candidate: /Users/cokes/opt/anaconda3/lib/libpython3.9.dylib
DEBUG Found: /Users/cokes/opt/anaconda3/lib/libpython3.9.dylib
DEBUG Candidate: /Users/cokes/opt/anaconda3/lib/libpython3.dylib
DEBUG Not found.
DEBUG Candidate: /Users/cokes/opt/anaconda3/lib/libpython.dylib
DEBUG Not found.
DEBUG Candidate: /Users/cokes/opt/anaconda3/libpython3.9.a
DEBUG Not found.
DEBUG Candidate: /Users/cokes/opt/anaconda3/libpython3.9.dylib
DEBUG Not found.
DEBUG Candidate: /Users/cokes/opt/anaconda3/libpython3.dylib
DEBUG Not found.
DEBUG Candidate: /Users/cokes/opt/anaconda3/libpython.dylib
DEBUG Not found.
DEBUG Candidate: /Users/cokes/opt/anaconda3/bin/../lib/libpython3.9.dylib
DEBUG Found: /Users/cokes/opt/anaconda3/lib/libpython3.9.dylib
dlopen(/Users/cokes/opt/anaconda3/lib/libpython3.9.dylib) ==> ErrorException("could not load library \"/Users/cokes/opt/anaconda3/lib/libpython3.9.dylib\"\ndlopen(/Users/cokes/opt/anaconda3/lib/libpython3.9.dylib, 0x0009): tried: '/Users/cokes/opt/anaconda3/lib/libpython3.9.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/cokes/opt/anaconda3/lib/libpython3.9.dylib' (no such file), '/Users/cokes/opt/anaconda3/lib/libpython3.9.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))")
dlopen(libpython3.9) ==> ErrorException("could not load library \"libpython3.9\"\ndlopen(libpython3.9.dylib, 0x0009): tried: 'libpython3.9.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpython3.9.dylib' (no such file), '/Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libpython3.9.dylib' (no such file), '/Applications/Julia-1.8.app/Contents/Resources/julia/lib/libpython3.9.dylib' (no such file), '/usr/lib/libpython3.9.dylib' (no such file, not in dyld cache), 'libpython3.9.dylib' (no such file), '/usr/local/lib/libpython3.9.dylib' (no such file), '/usr/lib/libpython3.9.dylib' (no such file, not in dyld cache)")
dlopen(libpython3.9) ==> ErrorException("could not load library \"libpython3.9\"\ndlopen(libpython3.9.dylib, 0x0009): tried: 'libpython3.9.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpython3.9.dylib' (no such file), '/Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libpython3.9.dylib' (no such file), '/Applications/Julia-1.8.app/Contents/Resources/julia/lib/libpython3.9.dylib' (no such file), '/usr/lib/libpython3.9.dylib' (no such file, not in dyld cache), 'libpython3.9.dylib' (no such file), '/usr/local/lib/libpython3.9.dylib' (no such file), '/usr/lib/libpython3.9.dylib' (no such file, not in dyld cache)")
dlopen(libpython3) ==> ErrorException("could not load library \"libpython3\"\ndlopen(libpython3.dylib, 0x0009): tried: 'libpython3.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpython3.dylib' (no such file), '/Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libpython3.dylib' (no such file), '/Applications/Julia-1.8.app/Contents/Resources/julia/lib/libpython3.dylib' (no such file), '/usr/lib/libpython3.dylib' (no such file, not in dyld cache), 'libpython3.dylib' (no such file), '/usr/local/lib/libpython3.dylib' (no such file), '/usr/lib/libpython3.dylib' (no such file, not in dyld cache)")
dlopen(libpython) ==> ErrorException("could not load library \"libpython\"\ndlopen(libpython.dylib, 0x0009): tried: 'libpython.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpython.dylib' (no such file), '/Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libpython.dylib' (no such file), '/Applications/Julia-1.8.app/Contents/Resources/julia/lib/libpython.dylib' (no such file), '/usr/lib/libpython.dylib' (no such file, not in dyld cache), 'libpython.dylib' (no such file), '/usr/local/lib/libpython.dylib' (no such file), '/usr/lib/libpython.dylib' (no such file, not in dyld cache)")
ERROR: LoadError: Couldn't find libpython; check your PYTHON environment variable.

The python executable we tried was /Users/cokes/opt/anaconda3/bin/python (= version 3.9).
Re-building with
    ENV["PYCALL_DEBUG_BUILD"] = "yes"
may provide extra information for why it failed.

Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] find_libpython(python::String; _dlopen::typeof(Base.Libc.Libdl.dlopen))
   @ Main ~/.julia/packages/PyCall/twYvK/deps/buildutils.jl:72
 [3] find_libpython(python::String)
   @ Main ~/.julia/packages/PyCall/twYvK/deps/buildutils.jl:40
 [4] top-level scope
   @ ~/.julia/packages/PyCall/twYvK/deps/build.jl:82
 [5] include(fname::String)
   @ Base.MainInclude ./client.jl:476
 [6] top-level scope
   @ none:5
in expression starting at /Users/cokes/.julia/packages/PyCall/twYvK/deps/build.jl:43