Otherwise we can't build the eBPF probe when the kmod is already built.

While we still have some hacks for e.g. choosing linux-modules
when available instead of linux-image, this crawler should be
much more robust than the old html-scraping based one.

Also includes support for PhotonOS and Oracle Linux 6/7/8.

Doesn't support Fedora Atomic and CoreOS, since these two distros
are dead by now.

These are the variants we haven't built before so exclude them
here to avoid an explosion in built probes.

Otherwise we get conflicting package names under different URLs
which made us include e.g. backported kernels (which we explicitly
skip) because they had the same package name as a non-backported
kernel we do want to build.

To unblock the probe builder job which doesn't have sqlite 3.8 installed

Otherwise we just reused whatever probe was lying there

We no longer download many of the linux-image-* packages
so we need the linux-modules-* ones to trigger the build.

This means we'll potentially run some builds twice but
the second one should immediately notice that the probes
are already built and exit.

… linux-headers Makefile format

Linux headers for Debian 11 use a new format for the architectur-specific
Makefile.  Enhance the Makefile relocation patching logic to handle this
new format as well as the old one.

… support CustomDebian operation

- Install libelf-dev in Debian dockerfiles
- Fix linux-kbuild finder logic to accomodate package location in CustomDebian mode

[SSPROD-9627] Enhance build-probe-binaries Debian logic to handle new linux-headers Makefile format

…n kernels

[SSPROD-9627] Fix README.md to document use of CustomDebian option

… to sysdig-probe-loader

Allows user to specify the full URL, including host, directory, AND filename,
to be used by sysdig-probe-loader in naming and retrieving a kernel module
or eBPF probe.

Useful if
- Kernel config file is not available on host - normally needed to calculate hash value
  component of probe filename
- Probe is to be downloaded from locally-hosted system, and the user is unable
  or unwilling to obey the Sysdig naming/directory hierarchy conventions

[SSPROD-9709] Add SYSDIG_PROBE_OVERRIDE_FULL_URL environment variable to sysdig-probe-loader

The eBPF probe builds successfully with clang 13 but then fails
the verifier test upon loading.

Fedora 34 and 35 have the same gcc version (at least at this point
in time) but are otherwise incompatible:
- fc34 can't build probes for fc35 kernels because fc35 needs a newer glibc
- fc35 can't build eBPF probes (at all) because its clang is too new

1. Try to parse the kernel version to look for distro-specific tags.
If one is found, look for a matching dockerfile. If there is one, use it.
Otherwise, fall back to gcc version based choice.

2. Add a fc34 builder

add a non-busybox sed to the container used to call
the top-level script build-probe-binaries

This is necessary to patch the Makefiles used when
building debian probes.

When defining the environment variable SYSDIG_FORCE_DOWNLOAD_PROBE,
the probe loader skips the building phase and tries to directly download the probe.

When defining the environment variable SYSDIG_FORCE_BUILD_PROBE,
the probe loader avoids downloading the probe and tries to directly build it.

It's been long dead now

It will eventually come back as FlatCar but CoreOS itself is dead.

This requires either a `pip install` from the probe builder directory
or (more likely) a Docker container.

The CLI is slightly changed: to actually build the probes, you need
to add the `build` keyword before the command line arguments.

To run the kernel crawler, run the builder with `crawl <distro>`.

The Docker entrypoint is adapted and usage in a container should not
require any changes.

vault.centos.org servers are renowned for having very
limited bandwidth. Fix the current error handling so that:
a) a stack trace is printed
b) remaining, working repositories can be processed

Some RPM-based distributions (namely CentOS) might at some point get
deprecated, so that their content gets removed completely from
the official mirrors, while leaving the base directory (e.g.  present
with just a readme file within it, e.g.:

  http://mirror.centos.org/centos/7.8.2003/readme

Filter out such distributions (where the final path is non-existing)
so that they're left out of the official list of repos to crawl.
This way, we prevent stack traces introduced by the new error handling
from cluttering the output.

- remove CentOS 8 BaseOS
- use CentOS vault mirror from cern.ch

some debian kernel packages might disappear from the server
while still being listed in the repository metadata.
This causes an exception in the downloader thread, which will
abort the thread, leaving the main thread stuck while
waiting for the queue to join.
Add some exception handler to the downlaoder thread so that
the caller doesn't get stuck.

current handling of debian packages (e.g. because
they failed to download and are empty) will cause the
unpacking to fail completely.
Handle that case by just skipping the directories for which
the unpacking failed.

debian and ubuntu have a very weird package naming which
requires us to discard the versioning information our crawler
obtains from the repository metadata, and recur to parsing
filenames, which is what we would be doing on a local distro
anyway.
Notice how this also fixes unpack_kernels for debian, which
expects a list of tuples instead of the dictionary returned
by the crawler.

some debian kernel packages have an additional version suffix
(examples I found are -rt and -cloud)
which is somehow buried into the name, and that is particularly
weird for the the ones having the "common" arch in the name:

linux-headers-5.16.0-1-amd64_5.16.7-2_amd64.deb
linux-headers-5.16.0-1-cloud-amd64_5.16.7-2_amd64.deb
linux-headers-5.16.0-1-common_5.16.7-2_all.deb
linux-headers-5.16.0-1-common-rt_5.16.7-2_all.deb
linux-headers-5.16.0-1-rt-amd64_5.16.7-2_amd64.deb

Refactor the regexp so to capture the combination of arch and variant
(in whatever order) by stopping at the first "_", and handle the
exception of the "common" pseudo-architecture accordingly.

Notice how we now have different combinations of variant+architecture
(or just architecture with no variant) but they are all treated as
if they were an architecture (with the exception of possibly having
an hypen in between, like cloud-amd64).
For this reason, use a different separator ":" between the version
and the variant+architecture so that the version can be
easily reconstructed.

Packages like:
linux-headers-5.4.0-1063-azure-cvm_5.4.0-1063.66+cvm3_amd64.deb
were not handled correctly in that release_id was considered
as '5.4.0-1063-azure'
which would also contain packages like:
linux-headers-5.4.0-1063-azure_5.4.0-1063.66_amd64.deb

leading to the following error:
ValueError: Unexpected version 5.4.0-1063/66+cvm3 from package
/workspace/ubuntu/linux-headers-5.4.0-1063-azure-cvm_5.4.0-1063.66+cvm3_amd64.deb
(expected 5.4.0-1063/66)

Change the regex so to swallow such strings and therefore yield
a different release_id and avoid such mismatch.

this way, in addition to:
=== 5.6.6-300.fc32.x86_64 ===
=== 5.8.15-301.fc33.x86_64 ===
=== 5.11.12-300.fc34.x86_64 ===
=== 5.14.10-300.fc35.x86_64 ===

we also get:
=== 5.11.22-100.fc32.x86_64 ===
=== 5.14.18-100.fc33.x86_64 ===
=== 5.16.9-100.fc34.x86_64 ===
=== 5.16.9-200.fc35.x86_64 ===

Fix several issues found during the first road test of the new builder in python

crawling ubuntu mirrors as debian-like has the effect
of resolving all dependencies across all repositories.
This creates a global namespace for all ubuntu packages
(across all distributions), which creates a problem when
the same kernel release (e.g. 5.4.0-1065-68) is packaged
within packages of the same name:
- linux-headers-5.4.0-1065-aws
but with different versions:
- 5.4.0-1065.68~18.04.1 (within bionic repositories)
- 5.4.0-1065.68 (within focal repositories)

This has the side effect of one version potentially masking all others.

This patch essentially restores the default distro behavior, by which
dependencies are only resolved looking at packages coming from the same
repo.

This way we can build all combinations of (kernel_release, pkgs_version),
jumping from 1568 to 1958 kernels.

Notice how we were not necessarily skipping backported kernels;
in some cases newer versions would be masked by older ones, for no
obvious reason.

with ubuntu no longer being a debian-like mirror, there's no
point keeping it around.
Drop it and add a few words why some special behavior is needed
for debian mirrors in the first place.

kernel_dirs has so far been treated as a dictionary with the release
as the key, and the target directory as the value.
However, for ubuntu we have the need to build the exact same release
onto multiple target directories (i.e. package versions).
Rework the dictionary as a list of tuples.

when batching packages, instead of using the plain release
'5.4.0-100-generic' as the key, use the tuple (release, version),
allowing for the storage of two separate set of packages within
the releases dictionary:

('5.4.0-100-generic', '5.4.0-100.113~18.04.1'):
  ['/workspace/ubuntu/linux-modules-5.4.0-100-generic_5.4.0-100.113~18.04.1_amd64.deb',
   '/workspace/ubuntu/linux-headers-5.4.0-100-generic_5.4.0-100.113~18.04.1_amd64.deb']

('5.4.0-100-generic', '5.4.0-100.113'):
   ['/workspace/ubuntu/linux-modules-5.4.0-100-generic_5.4.0-100.113_amd64.deb',
   '/workspace/ubuntu/linux-headers-5.4.0-100-generic_5.4.0-100.113_amd64.deb']

which will eventually be returned as two tuples in kernels:

('5.4.0-100-generic',
   ['/workspace/ubuntu/linux-modules-5.4.0-100-generic_5.4.0-100.113_amd64.deb',
   '/workspace/ubuntu/linux-headers-5.4.0-100-generic_5.4.0-100.113_amd64.deb'])

('5.4.0-100-generic',
  ['/workspace/ubuntu/linux-modules-5.4.0-100-generic_5.4.0-100.113~18.04.1_amd64.deb',
   '/workspace/ubuntu/linux-headers-5.4.0-100-generic_5.4.0-100.113~18.04.1_amd64.deb'])

add a missing return statement, probably forgotten

after running the builder image, check that the probe file
actually exists. If not, print the resulting output.
This becomes useful, for instance, when the eBPF probe does
not get built because the builder image does not contain
clang / llvm.

Re-add clang and llvm to the builder container for Fedora 35,
which were initially removed because at the time they would
generate code that would not pass the verifier test upon loading.

The version now present in the image does not seem to suffer from
this problem anymore, so re-add them.

This reverts commit 66724e2.

Improve logging

add -f <filter> option to the build command, so that filter could be
applied to building in addition to crawling.
Notice how for 'crawl' it's an argument, whereas for 'build' it's
an option.

Also refactor the argument all around so that it's called
'filter' as opposed to 'version', and add a few comments about the logic.

This makes for a slightly quicker test cycle by running, for instance:

... -k Ubuntu -f 4.4.0-1089
... -k Debian -f 4.19.0-0
... -k Fedora -f 5.6.6-300.fc32

…#40)

deb-based kernels (ubuntu and debian) are highly convoluted.
Add some debugging statements and comments to help understand.

revert the previous change where we would flatten everything
and always batch the whole pool of crawled kernels.

Instead, redefine the crawl() method only where needed:
- on debian, flatten all packages and batch them in
  a single heap (so, behaviour unchanged)
- on ubuntu, perform batching per general version
  (i.e., the release which does not include the variant)
  and pass that information to the batching logic.
  This will become clearer in the next commit.

There's one specific ubuntu kernel, namely 5.15.0-1001-gke,
which comes off as a set of packages which all have a
common version 5.15.0-1001.1, except for
linux-image-5.15.0-1001-gke which has version 5.15.0-1001.1+2.
This is somewhat an exception as normally, when you resolve
packages within the same repo, all kernel packages have
the exact same version, and this is the logic applied by
batch_packages() and unpack_kernels().

However, since we have repo metadata available, we can also leverage
that information, e.g.

  === 5.15.0-1001/1 ===
  http://security.ubuntu.com/ubuntu/pool/main/l/linux-gke/linux-gke-headers-5.15.0-1001_5.15.0-1001.1_amd64.deb
  http://security.ubuntu.com/ubuntu/pool/main/l/linux-signed-gke/linux-image-5.15.0-1001-gke_5.15.0-1001.1+2_amd64.deb
  http://security.ubuntu.com/ubuntu/pool/main/l/linux-gke/linux-modules-5.15.0-1001-gke_5.15.0-1001.1_amd64.deb
  http://security.ubuntu.com/ubuntu/pool/main/l/linux-gke/linux-headers-5.15.0-1001-gke_5.15.0-1001.1_amd64.deb

so that it only needs to be checked that the expected version is at least a prefix
of the package version.
Same goes for unpack_kernels() -- which only works because the "longest" package
does not happen to be the first one, and we surely hope this will not change.

Note: this can be tested by running:

...build ... -k Ubuntu -f 5.15.0-1001

Take care of the following error when building for Ubuntu:

```
FileNotFoundError: [Errno 2] No such file or directory: '/workspace/build/ubuntu/5.15.0-1001/1+2/boot/config-5.15.0-1001-gke'
```

Please see individual commits.

* optimize download

In order to handle incomplete downloads, the current implementation
will always try to append any remaining bytes to an already existing files.
But this means issuing at least one HTTP request per file, which might
slow down the process dramatically.
Instead, use temporary .part files and rename to the their filename
once complete -- this way we can assume that a file with the right name
is always complete and we can skip the download altogether.

* crawler: only use http for downloading

this will speed up subsequent downloads significantly
if we use a proxy

Signed-off-by: Federico Di Pierro <[email protected]>

…OSes.

Moreover, added support for 'arm' keyword to list all arm kernels (debian-like use 'arm64' while others use 'aarch64').

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

…of supported kernels.

Output can be found for each platform in kernels/ subdir.

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Multiarch support + github action to auto build lists of supported kernels on sunday midnight

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Ubuntu/debian/flatcar will fix it up for themselves.
Updated github action.

Signed-off-by: Federico Di Pierro <[email protected]>

Driverkit config out

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Fixed flatcar importing repo.

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Moreover, added an `headers` field to output driverkit config struct.
Finally, updated readme.

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>

Signed-off-by: Federico Di Pierro <[email protected]>