Patch and forbid download urls

[KUGA2]

Hi everyone.

At my company we have a strict policy that everything we use needs to be persisted at our servers. Therefore the regular download from github etc mechanism is not allowed. We have mirror servers that redirect to original server but cache and persist the files downloaded.

I have found a way to redirect the all downloads in vcpkg to our servers at quite a central position: vcpkg_download_distfile.cmake.

So I added and called this function there:

function(patch_urls VAR_NAME)
    set(URLS_OLD "${${VAR_NAME}}")
    string(REPLACE "https://github.com" "https://ourserver.com/github-proxy" URLS "${URLS_OLD}")
    string(REPLACE "https://dev-www.libreoffice.org" "https://ourserver.com/libreoffice-proxy" URLS "${URLS}")
    string(REPLACE "https://repo.msys2.org" "https://ourserver.com/msys2-proxy" URLS "${URLS}")
    if(NOT URLS_OLD STREQUAL URLS)
        message(STATUS "Patched urls: ${URLS_OLD} -> ${URLS}")
    endif()
    set(${VAR_NAME} "${URLS}" PARENT_SCOPE)
endfunction()

To prevent that other urls are used (sneaking by our proxy settings) I also added and used:

function(check_urls VAR_NAME)
    set(URLS "${${VAR_NAME}}")
    set(ALLOWED_URL "https://ourserver.com")
    foreach(URL IN LISTS URLS)
        if(NOT URL MATCHES ${ALLOWED_URL})
            message(FATAL_ERROR "Illegal url '${URL}' found. Does not match allowed url: '${ALLOWED_URL}'!")
        endif()
    endforeach()
endfunction()

Of course, the patch-pairs and allowed urls (plural!) should be a CMake variable instead of being hard coded.

1. What do you think of this solution?
2. Is there other code where files are downloaded?
3. Do you think this could make it mainline?
4. What would those variables be called? I suggest VCPKG_DOWNLOAD_PATCH_URLS and VCPKG_DOWNLOAD_ALLOWED_URLS.
5. If I create a PR, where do I need to document the new variables?

[Osyotr]

I believe your use case is covered by asset caching https://learn.microsoft.com/en-us/vcpkg/users/assetcaching combined with x-block-origin.

[dg0yt]

as 1) Isn't this a complicated way to achieve what a proxy could do in a less explicit way?

ad 2) Yes. E.g. vcpkg_from_git.

Did you consider vcpkg's asset caching?
https://learn.microsoft.com/en-us/vcpkg/users/assetcaching
And if you want to use local files only, there is --no-downloads:
https://learn.microsoft.com/en-us/vcpkg/commands/install#--no-downloads

[KUGA2]

@Osyotr @dg0yt

I cant work with x-azurl since I do not have a Azure Blob Storage or a fileshare. This would also prohibit user to add/try new packages before someone uploads them. --no-downloads has the same limitiations, but additionaly it does not work in manifest mode, which we use.

However, I could run my search and replace through x-script to my download tool. This is quite nice as I can also handle authentification there more easy. But i guess, vcpkg_from_git will then also be a problem.

[dg0yt]

--no-downloads has the same limitiations, but additionaly it does not work in manifest mode, which we use.

It can be used with VCPKG_INSTALL_OPTIONS (CMake) / VcpkgAdditionalInstallOptions (MSBuild).

FTR with --only-downloads, you can fetch the downloads without starting a build (but YMMV). Then you could move it to a shared location. After renaming based on SHA512, the assets might even be made accessible for reading from a file or http server.

[KUGA2]

Can you explain what you and the documentation page means by "renaming based on SHA512"? Can vcpkg download sources by SHA512? How?

When I set --no-downloads, I must download the sources somehow before starting the build. How can I know which ones I need without keeping a list?

I still think my "redirect to my server that caches" (with x-script) is the best solution for usability even if it does not support all download mechanisms. vcpkg_from_git( is only found in very few ports. If that is needed I can still download this manually beforehand.

[KUGA2]

Hi again,

so as announced, I used X_VCPKG_ASSET_SOURCES x-script. Is it correct, that my logs in that script are not printed as long as it succeeds?