Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bootstrappable Builds #8929

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+3,352 −1,565
Open

Bootstrappable Builds #8929

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 0 additions & 49 deletions .github/workflows/gitian.yml
View file
Open in desktop

This file was deleted.

104 changes: 104 additions & 0 deletions .github/workflows/guix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
name: ci/gh-actions/guix

on:
push:
paths:
- 'contrib/depends/**'
- 'contrib/guix/**'
- '!contrib/**.md'
- '.github/workflows/guix.yml'
pull_request:
paths:
- 'contrib/depends/**'
- 'contrib/guix/**'
- '!contrib/**.md'
- '.github/workflows/guix.yml'

jobs:
cache-sources:
runs-on: ubuntu-24.04
tobtoht marked this conversation as resolved.
Show resolved Hide resolved
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: depends sources cache
id: cache
uses: actions/cache@v4
with:
path: contrib/depends/sources
key: sources-${{ hashFiles('contrib/depends/packages/*') }}
- name: download depends sources
if: steps.cache.outputs.cache-hit != 'true'
run: make -C contrib/depends download

build-guix:
runs-on: ubuntu-24.04
needs: [cache-sources]
strategy:
fail-fast: false
matrix:
toolchain:
- target: "x86_64-linux-gnu"
- target: "aarch64-linux-gnu"
- target: "arm-linux-gnueabihf"
- target: "riscv64-linux-gnu"
- target: "i686-linux-gnu"
- target: "x86_64-w64-mingw32"
- target: "x86_64-unknown-freebsd"
- target: "x86_64-apple-darwin"
- target: "aarch64-apple-darwin"
- target: "aarch64-linux-android"
- target: "arm-linux-androideabi"

name: ${{ matrix.toolchain.target }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: recursive
- name: remove bundled packages
# This is faster than rm -rf
run: |
sudo mkdir /empty
sudo rsync -a --delete /empty/ /usr/local
- name: depends sources cache
uses: actions/cache/restore@v4
with:
path: contrib/depends/sources
key: sources-${{ hashFiles('contrib/depends/packages/*') }}
- name: install dependencies
run: sudo apt update; sudo apt -y install guix git ca-certificates
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be worth investigating if using the installer script is faster.

https://guix.gnu.org/manual/en/html_node/Installation.html

- name: apparmor workaround
# https://bugs.launchpad.net/ubuntu/+source/guix/+bug/2064115
run: |
sudo tee /etc/apparmor.d/guix << EOF
abi <abi/4.0>,
include <tunables/global>
profile guix /usr/bin/guix flags=(unconfined) {
userns,
include if exists <local/guix>
}
EOF
sudo /etc/init.d/apparmor reload
sudo aa-enforce guix || true
sudo apt purge apparmor
- name: build
run: SUBSTITUTE_URLS='http://bordeaux.guix.gnu.org' HOSTS="${{ matrix.toolchain.target }}" ./contrib/guix/guix-build
- uses: actions/upload-artifact@v4
with:
name: ${{ matrix.toolchain.target }}
path: |
guix/guix-build-*/output/${{ matrix.toolchain.target }}/*
guix/guix-build-*/logs/${{ matrix.toolchain.target }}/*

bundle-logs:
runs-on: ubuntu-24.04
needs: [build-guix]
steps:
- uses: actions/download-artifact@v4
with:
merge-multiple: true
- uses: actions/upload-artifact@v4
with:
name: "logs"
path: '**/logs/**'
6 changes: 2 additions & 4 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,8 @@ miniupnpcstrings.h
version/
ClangBuildAnalyzerSession.txt

# gitian
contrib/gitian/builder/
contrib/gitian/docker/
contrib/gitian/sigs/
# guix
/guix
Copy link
Contributor Author

@tobtoht tobtoht Feb 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finished builds are available in ./guix/guix-build-<COMMIT>/

├── guix-build-222ea1a30058
│   ├── build
│   │   └── distsrc-222ea1a30058-x86_64-linux-gnu  # build directory
│   ├── logs
│   │   └── x86_64-linux-gnu  # various logs for reproducibility debugging
│   │       ├── depends-hashes.txt
│   │       ├── depends-packages.txt
│   │       ├── guix-env.txt
│   │       ├── guix-hashes.txt
│   │       └── SHA256SUMS.part
│   ├── output
│   │   └── x86_64-linux-gnu
│   │       └── monero-x86_64-linux-gnu-222ea1a30058.tar.bz2  # binary tarball
│   └── var
│       ├── precious_dirs  # directories to keep when running `./contrib/guix/guix-clean`
│       └── profiles


# Created by https://www.gitignore.io

Expand Down
2 changes: 1 addition & 1 deletion CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1113,7 +1113,7 @@ if(MINGW)
elseif(APPLE OR OPENBSD OR ANDROID)
set(EXTRA_LIBRARIES "")
elseif(FREEBSD)
set(EXTRA_LIBRARIES execinfo)
set(EXTRA_LIBRARIES execinfo elf)
elseif(DRAGONFLY)
find_library(COMPAT compat)
set(EXTRA_LIBRARIES execinfo ${COMPAT})
Expand Down
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Portions Copyright (c) 2012-2013 The Cryptonote developers.
- [Release staging schedule and protocol](#release-staging-schedule-and-protocol)
- [Compiling Monero from source](#compiling-monero-from-source)
- [Dependencies](#dependencies)
- [Gitian builds](#gitian-builds)
- [Guix builds](#guix-builds)
- [Internationalization](#Internationalization)
- [Using Tor](#using-tor)
- [Pruning](#Pruning)
Expand Down Expand Up @@ -599,9 +599,9 @@ USE_DEVICE_TREZOR=OFF make release

For more information, please check out Trezor [src/device_trezor/README.md](src/device_trezor/README.md).

### Gitian builds
### Guix builds

See [contrib/gitian/README.md](contrib/gitian/README.md).
See [contrib/guix/README.md](contrib/guix/README.md).

## Installing Monero from a package

Expand Down
11 changes: 8 additions & 3 deletions contrib/depends/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ include builders/$(build_os).mk
include builders/default.mk
include packages/packages.mk

ifeq ($(GUIX_ENVIRONMENT),)
build_id_string:=$(BUILD_ID_SALT)
build_id_string+=$(shell $(build_CC) --version 2>/dev/null)
build_id_string+=$(shell $(build_AR) --version 2>/dev/null)
Expand All @@ -98,6 +99,10 @@ $(host_arch)_$(host_os)_id_string+=$(shell $(host_AR) --version 2>/dev/null)
$(host_arch)_$(host_os)_id_string+=$(shell $(host_CXX) --version 2>/dev/null)
$(host_arch)_$(host_os)_id_string+=$(shell $(host_RANLIB) --version 2>/dev/null)
$(host_arch)_$(host_os)_id_string+=$(shell $(host_STRIP) --version 2>/dev/null)
else
build_id_string:=$(realpath $(GUIX_ENVIRONMENT))
$(host_arch)_$(host_os)_id_string:=$(realpath $(GUIX_ENVIRONMENT))
Comment on lines +103 to +104
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any change to manifest.scm invalidates the cache.

endif

packages += $($(host_arch)_$(host_os)_packages) $($(host_os)_packages)
native_packages += $($(host_arch)_$(host_os)_native_packages) $($(host_os)_native_packages)
Expand All @@ -111,7 +116,7 @@ $(host_arch)_$(host_os)_native_toolchain?=$($(host_os)_native_toolchain)
include funcs.mk

toolchain_path=$($($(host_arch)_$(host_os)_native_toolchain)_prefixbin)
final_build_id_long+=$(shell $(build_SHA256SUM) toolchain.cmake.in)
final_build_id_long+=:[sha256sum]:$(shell $(build_SHA256SUM) toolchain.cmake.in)
tobtoht marked this conversation as resolved.
Show resolved Hide resolved
final_build_id+=$(shell echo -n "$(final_build_id_long)" | $(build_SHA256SUM) | cut -c-$(HASH_LENGTH))
$(host_prefix)/.stamp_$(final_build_id): $(native_packages) $(packages)
$(AT)rm -rf $(@D)
Expand All @@ -124,8 +129,8 @@ $(host_prefix)/.stamp_$(final_build_id): $(native_packages) $(packages)
$(host_prefix)/share/toolchain.cmake : toolchain.cmake.in $(host_prefix)/.stamp_$(final_build_id)
$(AT)@mkdir -p $(@D)
$(AT)sed -e 's|@HOST@|$(host)|' \
-e 's|@CC@|$(toolchain_path)$(host_CC)|' \
-e 's|@CXX@|$(toolchain_path)$(host_CXX)|' \
-e 's|@CC@|$(host_CC)|' \
-e 's|@CXX@|$(host_CXX)|' \
-e 's|@AR@|$(toolchain_path)$(host_AR)|' \
-e 's|@RANLIB@|$(toolchain_path)$(host_RANLIB)|' \
-e 's|@NM@|$(toolchain_path)$(host_NM)|' \
Expand Down
38 changes: 23 additions & 15 deletions contrib/depends/funcs.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,25 @@
define int_vars
#Set defaults for vars which may be overridden per-package
$(1)_cc=$($($(1)_type)_CC)
$(1)_cxx=$($($(1)_type)_CXX)
$(1)_objc=$($($(1)_type)_OBJC)
$(1)_objcxx=$($($(1)_type)_OBJCXX)
$(1)_ar=$($($(1)_type)_AR)
$(1)_ranlib=$($($(1)_type)_RANLIB)
$(1)_libtool=$($($(1)_type)_LIBTOOL)
$(1)_nm=$($($(1)_type)_NM)
$(1)_cflags=$($($(1)_type)_CFLAGS) $($($(1)_type)_$(release_type)_CFLAGS)
$(1)_cxxflags=$($($(1)_type)_CXXFLAGS) $($($(1)_type)_$(release_type)_CXXFLAGS)
$(1)_arflags=$($($(1)_type)_ARFLAGS) $($($(1)_type)_$(release_type)_ARFLAGS)
$(1)_ldflags=$($($(1)_type)_LDFLAGS) $($($(1)_type)_$(release_type)_LDFLAGS) -L$($($(1)_type)_prefix)/lib
$(1)_cppflags=$($($(1)_type)_CPPFLAGS) $($($(1)_type)_$(release_type)_CPPFLAGS) -I$($($(1)_type)_prefix)/include
$(1)_cc=$$($$($(1)_type)_CC)
$(1)_cxx=$$($$($(1)_type)_CXX)
$(1)_objc=$$($$($(1)_type)_OBJC)
$(1)_objcxx=$$($$($(1)_type)_OBJCXX)
$(1)_ar=$$($$($(1)_type)_AR)
$(1)_ranlib=$$($$($(1)_type)_RANLIB)
$(1)_libtool=$$($$($(1)_type)_LIBTOOL)
$(1)_nm=$$($$($(1)_type)_NM)
$(1)_cflags=$$($$($(1)_type)_CFLAGS) \
$$($$($(1)_type)_$$(release_type)_CFLAGS)
$(1)_cxxflags=$$($$($(1)_type)_CXXFLAGS) \
$$($$($(1)_type)_$$(release_type)_CXXFLAGS)
$(1)_arflags=$$($$($(1)_type)_ARFLAGS) \
$$($$($(1)_type)_$(release_type)_ARFLAGS)
$(1)_ldflags=$$($$($(1)_type)_LDFLAGS) \
$$($$($(1)_type)_$$(release_type)_LDFLAGS) \
-L$$($($(1)_type)_prefix)/lib
$(1)_cppflags=$$($$($(1)_type)_CPPFLAGS) \
$$($$($(1)_type)_$$(release_type)_CPPFLAGS) \
-I$$($$($(1)_type)_prefix)/include
Comment on lines +3 to +22
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delay expansion of package variables.

$(1)_recipe_hash:=
endef

Expand All @@ -37,6 +44,7 @@ endef

define int_get_build_recipe_hash
$(eval $(1)_all_file_checksums:=$(shell $(build_SHA256SUM) $(meta_depends) packages/$(1).mk $(addprefix $(PATCHES_PATH)/$(1)/,$($(1)_patches)) | cut -d" " -f1))
final_build_id_long+=:[$(1)_all_file_checksums]$(foreach checksum,$($(1)_all_file_checksums),$(shell echo ":$(checksum)")):
$(eval $(1)_recipe_hash:=$(shell echo -n "$($(1)_all_file_checksums)" | $(build_SHA256SUM) | cut -d" " -f1))
endef

Expand All @@ -46,7 +54,7 @@ $(eval $(1)_all_dependencies:=$(call int_get_all_dependencies,$(1),$($($(1)_type
$(foreach dep,$($(1)_all_dependencies),$(eval $(1)_build_id_deps+=$(dep)-$($(dep)_version)-$($(dep)_recipe_hash)))
$(eval $(1)_build_id_long:=$(1)-$($(1)_version)-$($(1)_recipe_hash)-$(release_type) $($(1)_build_id_deps) $($($(1)_type)_id_string))
$(eval $(1)_build_id:=$(shell echo -n "$($(1)_build_id_long)" | $(build_SHA256SUM) | cut -c-$(HASH_LENGTH)))
final_build_id_long+=$($(package)_build_id_long)
final_build_id_long+=:[recipe]:$(1)-$($(1)_version)-$($(1)_recipe_hash)-$(release_type):[deps]$(foreach dep,$($(1)_build_id_deps),$(shell echo ":$(dep)")):[$($(1)_type)_id]:$($($(1)_type)_id_string):
Comment on lines -49 to +57
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Increase verbosity of final_build_id_long to ease debugging reproducibility issues.


#compute package-specific paths
$(1)_build_subdir?=.
Expand Down Expand Up @@ -267,4 +275,4 @@ $(foreach package,$(all_packages),$(eval $(call int_config_attach_build_config,$
$(foreach package,$(all_packages),$(eval $(call int_add_cmds,$(package))))

#special exception: if a toolchain package exists, all non-native packages depend on it
$(foreach package,$(packages),$(eval $($(package)_unpacked): |$($($(host_arch)_$(host_os)_native_toolchain)_cached) ))
$(foreach package,$(packages),$(eval $($(package)_extracted): |$($($(host_arch)_$(host_os)_native_toolchain)_cached) ))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_unpacked target doesn't exist.

62 changes: 52 additions & 10 deletions contrib/depends/hosts/darwin.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +1,65 @@
OSX_MIN_VERSION=10.13
OSX_SDK_VERSION=11.0
XCODE_VERSION=12.2
XCODE_BUILD_ID=12B45b
LD64_VERSION=609
ifeq (aarch64, $(host_arch))
CC_target=arm64-apple-$(host_os)
else
CC_target=$(host)
endif
darwin_CC=clang -target $(CC_target) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(host_prefix)/native/SDK/ -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks -mlinker-version=$(LD64_VERSION) -B$(host_prefix)/native/bin/$(host)-
darwin_CXX=clang++ -target $(CC_target) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(host_prefix)/native/SDK/ -iwithsysroot/usr/include/c++/v1 -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks -mlinker-version=$(LD64_VERSION) -stdlib=libc++ -B$(host_prefix)/native/bin/$(host)-

OSX_SDK=$(host_prefix)/native/SDK

darwin_native_toolchain=darwin_sdk native_cctools

clang_prog=$(shell $(SHELL) $(.SHELLFLAGS) "command -v clang")
clangxx_prog=$(shell $(SHELL) $(.SHELLFLAGS) "command -v clang++")

# Flag explanations:
#
# -mlinker-version
#
# Ensures that modern linker features are enabled. See here for more
# details: https://github.com/bitcoin/bitcoin/pull/19407.
#
# -B$(build_prefix)/bin
#
# Explicitly point to our binaries (e.g. cctools) so that they are
# ensured to be found and preferred over other possibilities.
#
# -isysroot$(OSX_SDK) -nostdlibinc
#
# Disable default include paths built into the compiler as well as
# those normally included for libc and libc++. The only path that
# remains implicitly is the clang resource dir.
#
# -iwithsysroot / -iframeworkwithsysroot
#
# Adds the desired paths from the SDK
#

darwin_CC=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
-u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
-u LIBRARY_PATH \
$(clang_prog) --target=$(host) -mmacosx-version-min=$(OSX_MIN_VERSION) \
-B$(build_prefix)/bin -mlinker-version=$(LD64_VERSION) \
-isysroot$(OSX_SDK) \
-isysroot$(OSX_SDK) -nostdlibinc \
-iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks

darwin_CXX=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
-u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
-u LIBRARY_PATH \
$(clangxx_prog) --target=$(host) -mmacosx-version-min=$(OSX_MIN_VERSION) \
-B$(build_prefix)/bin -mlinker-version=$(LD64_VERSION) \
-isysroot$(OSX_SDK) -nostdlibinc \
-iwithsysroot/usr/include/c++/v1 \
-iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks

darwin_CFLAGS=-pipe
darwin_CXXFLAGS=$(darwin_CFLAGS)
darwin_ARFLAGS=cr

darwin_release_CFLAGS=-O1
darwin_release_CFLAGS=-O2
darwin_release_CXXFLAGS=$(darwin_release_CFLAGS)

darwin_debug_CFLAGS=-O1
darwin_debug_CXXFLAGS=$(darwin_debug_CFLAGS)

darwin_native_toolchain=native_cctools darwin_sdk

darwin_cmake_system=Darwin
13 changes: 11 additions & 2 deletions contrib/depends/hosts/freebsd.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
freebsd_CC=clang-8
freebsd_CXX=clang++-8
clang_prog=$(shell $(SHELL) $(.SHELLFLAGS) "command -v clang")
clangxx_prog=$(shell $(SHELL) $(.SHELLFLAGS) "command -v clang++")

freebsd_CC=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
-u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
-u LIBRARY_PATH $(clang_prog) --target=$(host) --sysroot=$(host_prefix)/native -iwithsysroot/usr/include
freebsd_CXX=env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH \
-u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH \
-u LIBRARY_PATH $(clangxx_prog) --target=$(host) -stdlib=libc++ --sysroot=$(host_prefix)/native \
-iwithsysroot/usr/include/c++/v1 -iwithsysroot/usr/include

freebsd_AR=ar
freebsd_RANLIB=ranlib
freebsd_NM=nm
Expand Down
Loading
Loading