Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new(ci): run update-kernels daily. #179

Merged
merged 4 commits into from
Oct 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 25 additions & 78 deletions .github/workflows/update-kernels.yml
Original file line number Diff line number Diff line change
@@ -1,37 +1,10 @@
---
name: Update Kernels

on:

Check warning on line 4 in .github/workflows/update-kernels.yml

View workflow job for this annotation

GitHub Actions / Lints and Scans

4:1 [truthy] truthy value should be one of [false, true]
workflow_dispatch:
inputs:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't support single input anymore:

  • since dbg-go is super quick to generate dbg configs (a matter of seconds)
  • since we will run the crawler daily
    we have no more needs for it.

distro:
description: distro of which to perform the update
type: choice
required: true
options:
- '*'
- AliyunLinux
- AlmaLinux
- AmazonLinux
- AmazonLinux2
- AmazonLinux2022
- AmazonLinux2023
- ArchLinux
- BottleRocket
- CentOS
- Debian
- Fedora
- Flatcar
- Minikube
- OpenSUSE
- OracleLinux
- PhotonOS
- Redhat
- RockyLinux
- Talos
- Ubuntu
schedule:
- cron: '30 6 * * 1'
- cron: '30 6 * * *'

# Checks if any concurrent jobs is running for kernels CI and eventually cancel it.
concurrency:
Expand All @@ -45,71 +18,45 @@
image: falcosecurity/kernel-crawler:latest
options: -u root
permissions:
contents: write
pull-requests: write
contents: read
pages: write
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will directly deploy pages without pushing anything to any branch.

id-token: write
steps:
- name: Checkout crawler
uses: actions/checkout@v3
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We checkout default branch that now has the index.html file.

with:
ref: kernels

- name: Fetch distro
run: |
INPUT_DISTRO=${{ inputs.distro }}
DISTRO=${INPUT_DISTRO:-'*'}
echo "distro=$DISTRO" >> $GITHUB_ENV

- name: Setup jq
run: apt-get update && apt-get install jq -y

- name: Prepare commit body
id: prep_commit_msg
run: |
COMMIT_MSG="This PR updates the list of kernels from the latest crawling \
$(if [ "${{ env.distro }}" != '*' ]; then echo " for distro ${{ env.distro }}"; fi). \
Do not edit this PR."
echo "commit_msg=$COMMIT_MSG" >> $GITHUB_OUTPUT

- name: Run crawler for x86_64
run: |
mkdir $RUNNER_TEMP/x86_64
kernel-crawler crawl --distro=${{ env.distro }} > $RUNNER_TEMP/x86_64/list.json
kernel-crawler crawl --distro="*" > $RUNNER_TEMP/x86_64/list.json

- name: Run crawler for aarch64
run: |
mkdir $RUNNER_TEMP/aarch64
kernel-crawler crawl --distro=${{ env.distro }} --arch=aarch64 > $RUNNER_TEMP/aarch64/list.json
kernel-crawler crawl --distro="*" --arch=aarch64 > $RUNNER_TEMP/aarch64/list.json

- name: Single distro update
if: ${{ github.event_name == 'workflow_dispatch' && inputs.distro != '*' }}
- name: Install deps
run: |
jq --arg distroKey "${{ inputs.distro }}" \
--slurpfile newValues $RUNNER_TEMP/x86_64/list.json \
'if .[$distroKey] then .[$distroKey] = $newValues[][] else . end' \
x86_64/list.json > $RUNNER_TEMP/x86_64/output.json
jq --arg distroKey "${{ inputs.distro }}" \
--slurpfile newValues $RUNNER_TEMP/aarch64/list.json \
'if .[$distroKey] then .[$distroKey] = $newValues[][] else . end' \
aarch64/list.json > $RUNNER_TEMP/aarch64/output.json
mv $RUNNER_TEMP/x86_64/output.json $RUNNER_TEMP/x86_64/list.json
mv $RUNNER_TEMP/aarch64/output.json $RUNNER_TEMP/aarch64/list.json
apt update
apt install -y jq

- name: Update json lists
- name: Validate jsons
run: |
mv $RUNNER_TEMP/x86_64/list.json x86_64/list.json
mv $RUNNER_TEMP/aarch64/list.json aarch64/list.json
cat $RUNNER_TEMP/x86_64/list.json | jq empty
cat $RUNNER_TEMP/aarch64/list.json | jq empty

- name: Update last run distro
- name: Move everything under site folder
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use site folder as root for the pages deployment.

run: |
echo "${{ env.distro }}" > last_run_distro.txt

- name: Create Pull Request
uses: peter-evans/create-pull-request@v5-rc
mkdir site/
mv index.html site/
mkdir site/aarch64/
$RUNNER_TEMP/aarch64/list.json site/aarch64
mkdir site/x86_64/
mv $RUNNER_TEMP/x86_64/list.json site/x86_64/

- uses: actions/upload-pages-artifact@v1
with:
signoff: true
branch: update/kernels
base: kernels
title: 'update(kernels): update kernel json lists.'
body: ${{ steps.prep_commit_msg.outputs.commit_msg }}
commit-message: 'update(kernels): update kernel json lists.'
token: ${{ secrets.GITHUB_TOKEN }}
path: 'site'

- id: deployment
uses: actions/deploy-pages@v1g
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@
![Architectures](https://img.shields.io/badge/ARCHS-x86__64%7Caarch64-blueviolet?style=for-the-badge)

It is a tool used to crawl supported kernels by multiple distros, and generate a [driverkit](https://github.com/falcosecurity/driverkit)-like config json.
Output json can be found, for each supported architecture, under [kernels](https://github.com/falcosecurity/kernel-crawler/tree/kernels) branch and on gh pages: https://falcosecurity.github.io/kernel-crawler/.
Output json can be found, for each supported architecture, on gh pages: https://falcosecurity.github.io/kernel-crawler/:
* [aarch64](https://falcosecurity.github.io/kernel-crawler/aarch64/list.json)
* [x86_64](https://falcosecurity.github.io/kernel-crawler/x86_64/list.json)

A weekly [github action workflow](https://github.com/falcosecurity/kernel-crawler/actions/workflows/update-kernels.yml) will open a PR on this repo to update the json.
As soon as the PR is merged and the json updated, a [prow job](https://github.com/falcosecurity/test-infra/blob/master/config/jobs/update-dbg/update-dbg.yaml) will create a PR on [test-infra](https://github.com/falcosecurity/test-infra) to generate the new Driverkit configs from the updated json.
Expand Down
151 changes: 151 additions & 0 deletions index.html
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file was completely copy/pasted by the kernels branch.

Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Kernel Crawler</title>
<meta name="author" content="">
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/5.1.3/css/bootstrap.min.css">
<link rel="stylesheet" href="https://cdn.datatables.net/1.12.1/css/dataTables.bootstrap5.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/font/bootstrap-icons.css">
<script type="text/javascript" src="https://code.jquery.com/jquery-3.5.1.js"></script>
<script type="text/javascript" src="https://cdn.datatables.net/1.12.1/js/jquery.dataTables.min.js"></script>
<script type="text/javascript" src="https://cdn.datatables.net/1.12.1/js/dataTables.bootstrap5.min.js"></script>
<style>
.dataTables_filter {
padding-right: 10px;
}
.btn {
padding-bottom: 15px;
}
</style>
</head>
<body>
<div>
<img src="https://sysdig.com/wp-content/uploads/2018/10/Falco-horizontal-logo-teal_2x.png" height="55" alt="falco logo">
</div>
<div id="archs" style="padding-left: 10px;">
Architecture:
</div>
<div id="targets" style="padding-left: 10px;">
Target:
</div>
<table id="kernels" class="table table-striped table-condensed" style="padding-left: 10px;">
<thead>
<tr>
<th>Target</th>
<th>Kernel Release</th>
<th>Kernel Version</th>
<th>Headers</th>
<th>Config</th>
<th>Link</th>
</tr>
</thead>
</table>
</body>
<script>
var url = new URL(window.location);
var arch = url.searchParams.get('arch');
if (arch == null) {
arch = "x86_64";
url.searchParams.set('arch', arch);
};
var target = url.searchParams.get('target')
if (target == null) {
if (arch == "x86_64") {
target = "AmazonLinux";
}
if (arch == "aarch64") {
target = "AmazonLinux2";
}
url.searchParams.set('target', target);
};
var search = url.searchParams.get('search')
if (search == null) {
search = "";
};
window.history.pushState({}, '', url);

const archs = ['x86_64', 'aarch64']
archs.forEach ((item) => {
var searchPrms = new URLSearchParams(window.location.search);
searchPrms.delete('search');
var selected = searchPrms.get('arch');
searchPrms.set('arch', item);
var element = document.createElement('a');
element.className = "btn btn-outline-primary btn-sm";
if (selected == item) {
element.className = "btn btn-primary btn-sm";
}
element.style = "margin: 5px; padding-top: 8px; padding-bottom: 10px;"
element.text = item;
element.href = "?"+searchPrms.toString();
document.getElementById('archs').appendChild(element);
});

$.getJSON(arch+'/list.json', function(data) {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now points to a local (relative) file.

Object.keys(data).forEach ((key) => {
var searchPrms = new URLSearchParams(window.location.search);
searchPrms.delete('search');
var selected = searchPrms.get('target');
searchPrms.set('target', key);
var element = document.createElement('a');
element.className = "btn btn-outline-primary btn-sm";
if (selected == key) {
element.className = "btn btn-primary btn-sm";
}
element.style = "margin: 5px; padding-top: 8px; padding-bottom: 10px;"
element.text = key;
element.href = "?"+searchPrms.toString();
document.getElementById('targets').appendChild(element);
});
});

$(document).ready(function() {
$('#kernels').DataTable({
"search": {"search": search },
"paging": false,
// "data" : data,
"order": [[ 1, "desc" ]],
ajax: {
url: arch+'/list.json',
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

dataSrc: target,
},
columns : [
{ "data" : "target"},
{ "data" : "kernelrelease"},
{ "data" : "kernelversion"},
{
"data": 'headers',
render: function (data, type) {
if (data === undefined) {
return '';
}
let s = '';
for (const value of Object.values(data)) {
s += '<a href="'+value+'" download="'+value+'"><i class="bi bi-download" style="margin-right: 8px;"></i></a>'
}
return s;
},
},
{
"data": 'kernelconfigdata',
render: function (data, type) {
if (data === undefined) {
return '';
}
return '<a href="data:application/octet-stream;charset=utf-8;base64,'+data+'" download="config.txt"><i class="bi bi-download"></a>'
},
},
{
"data" : "kernelrelease",
render: function (data, type, row) {
return '<a href="?arch='+arch+'&target='+target+'&search='+data+'"><i class="bi bi-link"></i></a>';
}
},
]
});
});
</script>
</html>
Loading