Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add http/socks5 proxy support for package manager #15048

Open
binarycraft007 opened this issue Mar 22, 2023 · 10 comments · May be fixed by #22458
Open

add http/socks5 proxy support for package manager #15048

binarycraft007 opened this issue Mar 22, 2023 · 10 comments · May be fixed by #22458
Labels
enhancement Solving this issue will likely involve adding new logic or components to the codebase. zig build system std.Build, the build runner, `zig build` subcommand, package management
Milestone

Comments

@binarycraft007
Copy link
Contributor

Not all people have unrestricted network access to package source, for example: github is censored by some countries, it would be good to add proxy support for package manager so that user under restricted network can fetch package successfully.

@nektro
Copy link
Contributor

nektro commented Mar 22, 2023

personally more interested in this so that Zig supports Tor

@andrewrk andrewrk added enhancement Solving this issue will likely involve adding new logic or components to the codebase. zig build system std.Build, the build runner, `zig build` subcommand, package management labels Apr 10, 2023
@andrewrk andrewrk added this to the 0.12.0 milestone Apr 10, 2023
@Tesseract22
Copy link

Tesseract22 commented Jul 27, 2023

What is the progress on this? I am also suffering from censorship of github.
Besides directly setting a proxy on zig package manager, is there a way to manually download the dependencies and have the package manager skip over the downloading part?

@snoire
Copy link
Contributor

snoire commented Jul 28, 2023

Besides directly setting a proxy on zig package manager, is there a way manually to download the dependencies and have the package manager skips over the downloading part?

You can download the dependency into ~/.cache/zig/p, unzip it, and rename the folder to the hash value. Then zig build will use the local copies instead of trying to download.

@tizee
Copy link

tizee commented Aug 21, 2023

Here is my workaround for the proxy issue. After building my patched zig from source, I could use zig build --build-proxy http://127.0.0.1:1080 to fetch pacakages via proxy.

I'm still confused that there are two zig build commands. One is for fetch packages and another is for build runner.

https://github.com/tizee/zig/tree/feature/http-socks5-proxy-support

@alberic89
Copy link

zig build should detect and use http_proxy and https_proxy environment variables. My entire network is behind a proxy that doesn't let anything else through, and I assume this is also the case in many universities and work offices. It's a shame not to be able to use the Zig build system simply because it can't read the signposts of the internet route.

@abrasumente233
Copy link

My temporary workaround is to manually download the package, host it locally and change the dependency URL accordingly:

{
  .binned_allocator = .{
    .url = "http://localhost:8080/8372900fcc09e38d7b0b6bbaddad3904-6c3321e0969ff2463f8335da5601986cf2108690.tar.gz",
    .hash = "1220363c7e27b2d3f39de6ff6e90f9537a0634199860fea237a55ddb1e1717f5d6a5",
  },
}

If anyone wants to implement http_proxy, https_proxy and no_proxy properly, it should be relatively easy, since zig's http client already supports proxies, and there is Go's proxy parsing and matching reference implementation: https://cs.opensource.google/go/x/net/+/refs/tags/v0.15.0:http/httpproxy/proxy.go

@konosubakonoakua
Copy link

any update?

@Jack-Ji
Copy link
Contributor

Jack-Ji commented Apr 4, 2024

A dirty script to workaround the issue.

#!/bin/bash

set -e

forceupdate=false
if [ "$1" = "-f" ]; then
    forceupdate=true
    shift
fi

do_fetch() {
    for d in `grep -o 'url *=.*' $1 | cut -d = -f 2`; do
        d=`echo $d | grep -o 'https://[^"]*'`
        echo -e "\n>>> Deal with $d"
        if echo $d | grep -q '\.tar\.gz$'; then
            url=$d
        elif echo $d | grep -q '#[.0-9a-z]*$'; then
            url_base=`echo $d | awk -F \# '{print $1}'`
            url_base=${url_base%.git}
            url_commit=`echo $d | awk -F \# '{print $2}'`
            url="${url_base}/archive/${url_commit}.tar.gz"
        else
            echo ">>> Ignored $d, unable to resolve it!"
            continue
        fi
        hash=`grep -m 1 -A 1 "$d" $1 | grep hash |  awk -F \" '{print $(NF-1)}'`
        if [ -z "$hash" ]; then
          forceupdate=true
        fi
        if ! $forceupdate && [ -e ~/.cache/zig/p/$hash ]; then
          echo ">>> Found $url in cache, ignored"
          continue
        fi
        wget $url
        tarfile=${url##*/}
        hash=`zig fetch --debug-hash $tarfile | tail -n 1`
        echo ">> hash of $d:"
        echo -e "\t$hash"
        rm $tarfile
        if [ -e ~/.cache/zig/p/$hash/build.zig.zon ]; then
            do_fetch ~/.cache/zig/p/$hash/build.zig.zon
        fi
    done

    for d in `grep -o 'path *=.*' $1 | cut -d = -f 2`; do
        path=`echo $d | awk -F \" '{print $(NF-1)}'`
        if [ -e $path/build.zig.zon ]; then
            do_fetch $path/build.zig.zon
        fi
    done
}

zonfile=$1
if [ -z "$zonfile" ]; then
    zonfile=build.zig.zon
fi

if ! [ -e $zonfile ]; then
    echo "can't find build.zig.zon!"
    exit 1
fi

do_fetch $zonfile

EDIT 1: Eliminate redundant downloading
EDIT 2: Force update if hash is empty, output checksum hash for reference

@konosubakonoakua
Copy link

or manually manage plugins like this project.
https://github.com/Srekel/tides-of-revival/blob/2aec79d644bbf5358ff502952386e138ebbf247f/sync_external.py#L63

@tw4452852
Copy link
Contributor

A dirty script to workaround the issue.

#!/bin/bash

set -e

forceupdate=false
if [ "$1" = "-f" ]; then
    forceupdate=true
    shift
fi

do_fetch() {
    for d in `grep -o 'url *=.*' $1 | cut -d = -f 2`; do
        d=`echo $d | grep -o 'https://[^"]*'`
        echo -e "\n>>> Deal with $d"
        if echo $d | grep -q '\.tar\.gz$'; then
            url=$d
        elif echo $d | grep -q '#[.0-9a-z]*$'; then
            url_base=`echo $d | awk -F \# '{print $1}'`
            url_base=${url_base%.git}
            url_commit=`echo $d | awk -F \# '{print $2}'`
            url="${url_base}/archive/${url_commit}.tar.gz"
        else
            echo ">>> Ignored $d, unable to resolve it!"
            continue
        fi
        hash=`grep -m 1 -A 1 "$d" $1 | grep hash |  awk -F \" '{print $(NF-1)}'`
        if [ -z "$hash" ]; then
          forceupdate=true
        fi
        if ! $forceupdate && [ -e ~/.cache/zig/p/$hash ]; then
          echo ">>> Found $url in cache, ignored"
          continue
        fi
        wget $url
        tarfile=${url##*/}
        hash=`zig fetch --debug-hash $tarfile | tail -n 1`
        echo ">> hash of $d:"
        echo -e "\t$hash"
        rm $tarfile
        if [ -e ~/.cache/zig/p/$hash/build.zig.zon ]; then
            do_fetch ~/.cache/zig/p/$hash/build.zig.zon
        fi
    done

    for d in `grep -o 'path *=.*' $1 | cut -d = -f 2`; do
        path=`echo $d | awk -F \" '{print $(NF-1)}'`
        if [ -e $path/build.zig.zon ]; then
            do_fetch $path/build.zig.zon
        fi
    done
}

zonfile=$1
if [ -z "$zonfile" ]; then
    zonfile=build.zig.zon
fi

if ! [ -e $zonfile ]; then
    echo "can't find build.zig.zon!"
    exit 1
fi

do_fetch $zonfile

EDIT 1: Eliminate redundant downloading EDIT 2: Force update if hash is empty, output checksum hash for reference

For someone using this script, a little fix to find the global caching directory instead of hardcoded ~/.cache/zig: global_cache_dir=$(zig env | jq -r .global_cache_dir)

@thislight thislight linked a pull request Jan 10, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Solving this issue will likely involve adding new logic or components to the codebase. zig build system std.Build, the build runner, `zig build` subcommand, package management
Projects
None yet
Development

Successfully merging a pull request may close this issue.