-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tools/docker/syzbot: use LLVM archive instead apt #5442
base: master
Are you sure you want to change the base?
Conversation
I've also uploaded the output of docker build here: |
cea291a
to
66b5200
Compare
66b5200
to
522f052
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, but please fix the comments.
gunzip -d $LLVM_VERSION.tar | ||
gpg --trust-model tofu --verify $LLVM_VERSION.tar.sign $LLVM_VERSION.tar | ||
|
||
# untar binaries to the output |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ditto.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! PTAL
@a-nogikh , @tarasmadan, @dvyukov , please review as well. As this should close #5375 PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Following up
fea7db6
to
7afa39b
Compare
Replace clang, llvm apt packages in syzbot Dockerfile with the stable kernel version. Reduce Dockerfile code (less RUN layers). Use stable LLVM version that Linux kernel doc suggests [1]. [1] https://mirrors.edge.kernel.org/pub/tools/llvm/ These 2 binaries are missing in the archive: "clang-format, clang-tidy", hence install them as the current apt package. This patch increases syzbot docker image size by 230MB, see the comparison between "syz_master" (current master branch built image), and with the applied patch "syz_master_diff". $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE syz_master_diff latest 40b209ef582e 6 hours ago 5.54GB syz_master latest 292faad84de9 13 hours ago 5.31GB The archive is 123M size and I've built the image 3 times with removing the images as the fresh install, and curl downloading for my 500Mbit/s Ethernet downlink took: 1st build: 0 hour 0 min 50 sec 2nd build: 1 hour 3 min 5 sec 3rd build: 0 hour 1 min 5 sec Hereby, I'm concerned if the archive can be mirrored from "mirrors.edge.kernel.org" to more stable/reliable storage with more distributed CDN like dl.google.com, or it's something local with my downlink. Per the 3rd build, the patch also speeds up (~1 min faster) the docker image build time: ... Step 7/27 : RUN curl -L <redacted> ---> Running in ae2853dfc0fc % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 123M 100 123M 0 0 1916k 0 0:01:05 0:01:05 --:--:-- 2513k ... Successfully built 387ae7e06dfa Successfully tagged syz_master_diff:latest real 6m23.013s user 0m0.145s sys 0m0.094s and the original "syz_master" fresh build: ... Successfully built 843592c9ce1b Successfully tagged syz_master:latest real 7m18.333s user 0m0.152s sys 0m0.090s Verified the binaries' versions in the container as well: $ docker run --rm --name syz_master_diff -it syz_master_diff bash /🤖 for e in {clang,clang++,clang-tidy,clang-format,ld.lld,llvm-nm,\ > llvm-ar,llvm-objcopy,llvm-objdump,llvm-addr2line,llvm-readelf,\ > llvm-strip}; do echo; $e --version; done Debian clang version 15.0.6 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin Debian clang version 15.0.6 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin Debian LLVM version 15.0.6 Optimized build. Default target: x86_64-pc-linux-gnu Host CPU: skylake Debian clang-format version 15.0.6 ClangBuiltLinux LLD 15.0.7 (compatible with GNU linkers) llvm-nm, compatible with GNU nm LLVM (http://llvm.org/): LLVM version 15.0.7 Optimized build. Default target: x86_64-unknown-linux-gnu Host CPU: skylake LLVM (http://llvm.org/): LLVM version 15.0.7 Optimized build. Default target: x86_64-unknown-linux-gnu Host CPU: skylake llvm-objcopy, compatible with GNU objcopy LLVM (http://llvm.org/): LLVM version 15.0.7 Optimized build. Default target: x86_64-unknown-linux-gnu Host CPU: skylake LLVM (http://llvm.org/): LLVM version 15.0.7 Optimized build. Default target: x86_64-unknown-linux-gnu Host CPU: skylake Registered Targets: aarch64 - AArch64 (little endian) aarch64_32 - AArch64 (little endian ILP32) aarch64_be - AArch64 (big endian) arm - ARM arm64 - ARM64 (little endian) arm64_32 - ARM64 (little endian ILP32) armeb - ARM (big endian) bpf - BPF (host endian) bpfeb - BPF (big endian) bpfel - BPF (little endian) hexagon - Hexagon mips - MIPS (32-bit big endian) mips64 - MIPS (64-bit big endian) mips64el - MIPS (64-bit little endian) mipsel - MIPS (32-bit little endian) ppc32 - PowerPC 32 ppc32le - PowerPC 32 LE ppc64 - PowerPC 64 ppc64le - PowerPC 64 LE riscv32 - 32-bit RISC-V riscv64 - 64-bit RISC-V systemz - SystemZ thumb - Thumb thumbeb - Thumb (big endian) x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 llvm-addr2line LLVM (http://llvm.org/): LLVM version 15.0.7 Optimized build. Default target: x86_64-unknown-linux-gnu Host CPU: skylake LLVM (http://llvm.org/): LLVM version 15.0.7 Optimized build. Default target: x86_64-unknown-linux-gnu Host CPU: skylake llvm-strip, compatible with GNU strip LLVM (http://llvm.org/): LLVM version 15.0.7 Optimized build. Default target: x86_64-unknown-linux-gnu Host CPU: skylake Fixes: google#5385
7afa39b
to
93131d7
Compare
Replace clang, llvm apt packages in syzbot Dockerfile with the stable kernel version. Reduce Dockerfile code (less RUN layers).
Use stable LLVM version that Linux kernel doc suggests [1].
These 2 binaries are missing in the archive: "clang-format, clang-tidy", hence install them as the current apt package.
This patch increases syzbot docker image size by 230MB, see the comparison between "syz_master" (current master branch built image), and with the applied patch "syz_master_diff".
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
syz_master_diff latest 40b209ef582e 6 hours ago 5.54GB
syz_master latest 292faad84de9 13 hours ago 5.31GB
The archive is 123M size and I've built the image 3 times with removing the images as the fresh install, and curl downloading for my 500Mbit/s Ethernet downlink took:
1st build: 0 hour 0 min 50 sec
2nd build: 1 hour 3 min 5 sec
3rd build: 0 hour 1 min 5 sec
Hereby, I'm concerned if the archive can be mirrored from "mirrors.edge.kernel.org" to more stable/reliable storage with more distributed CDN like dl.google.com, or it's something local with my downlink.
Per the 3rd build, the patch also speeds up (~1 min faster) the docker image build time:
...
Step 7/27 : RUN curl -L
---> Running in ae2853dfc0fc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 123M 100 123M 0 0 1916k 0 0:01:05 0:01:05 --:--:-- 2513k
...
Successfully built 387ae7e06dfa
Successfully tagged syz_master_diff:latest
and the original "syz_master" fresh build:
...
Successfully built 843592c9ce1b
Successfully tagged syz_master:latest
Verified the binaries' versions in the container as well:
[1] https://mirrors.edge.kernel.org/pub/tools/llvm/
Fixes: #5385