Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Konflux(ROX-22195): Install RHEL RPMs with subscription manager workaround #1573

Merged
merged 114 commits into from
Feb 29, 2024
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
7ff08c8
add custom Dockerfile for rhtap
tommartensen Oct 16, 2023
1c6a49d
Red Hat Trusted App Pipeline update collector (#1367)
red-hat-konflux[bot] Oct 16, 2023
69200d7
RHTAP: fix workspace size for checkouts (#1368)
tommartensen Oct 16, 2023
7410e30
broken dnf installs
tommartensen Oct 17, 2023
4399cad
Update RHTAP references (#1375)
red-hat-konflux[bot] Oct 19, 2023
37b0c74
Update RHTAP references (#1378)
red-hat-konflux[bot] Oct 23, 2023
80d6c38
use centos instead of ubi images
tommartensen Oct 25, 2023
8cc0a9a
Merge branch 'tm/rhtap-onboarding' of github.com:stackrox/collector i…
tommartensen Oct 25, 2023
8b4d052
Update RHTAP references (#1387)
red-hat-konflux[bot] Nov 6, 2023
3a55302
Merge branch 'master' into tm/rhtap-onboarding
tommartensen Nov 6, 2023
6945989
update pipelines
tommartensen Nov 6, 2023
a1811d5
increase storage for shared volume in pipeline
tommartensen Nov 6, 2023
157b8d0
falco submodule
tommartensen Nov 6, 2023
b26118c
update TODOs and LABELs
tommartensen Nov 6, 2023
807015c
restore falco
tommartensen Nov 6, 2023
6ade98a
clean up
tommartensen Nov 9, 2023
0d4cef0
Red Hat Trusted App Pipeline update collector-slim (#1414)
red-hat-konflux[bot] Nov 9, 2023
1ea2eb2
rename collector -> collector-slim
tommartensen Nov 9, 2023
3db110e
finish up
tommartensen Nov 13, 2023
56abbd3
Merge branch 'master' into tm/rhtap-onboarding
tommartensen Nov 13, 2023
346e7a4
fix build after rebase
tommartensen Nov 13, 2023
49f6bcd
Update RHTAP references (#1415)
red-hat-konflux[bot] Nov 13, 2023
f4d9d03
more oomph
tommartensen Nov 13, 2023
4f2e97e
Merge branch 'tm/rhtap-onboarding' of github.com:stackrox/collector i…
tommartensen Nov 13, 2023
a225b95
Update RHTAP references (#1420)
red-hat-konflux[bot] Nov 14, 2023
a8c55ee
attempt with default buildah size
tommartensen Nov 14, 2023
4f41bb5
fix task ref
tommartensen Nov 14, 2023
7ec670a
Update RHTAP references (#1421)
red-hat-konflux[bot] Nov 14, 2023
3957e05
Update RHTAP references (#1422)
red-hat-konflux[bot] Nov 15, 2023
a7bff4d
RHTAP Onboarding: 2nd attempt (#1425)
tommartensen Nov 17, 2023
e2bf0b3
move dockerfile
tommartensen Nov 20, 2023
b1eea97
emptyg
tommartensen Nov 20, 2023
780e997
disable prefetch-input
tommartensen Nov 20, 2023
bb8cfa8
remove guard on prefetch-dependencies task
tommartensen Nov 20, 2023
f4512eb
clean up Dockerfile
tommartensen Nov 20, 2023
119ff0a
add CODEOWNERS
tommartensen Nov 20, 2023
4c57a27
Merge branch 'master' into tm/rhtap-onboarding
tommartensen Nov 21, 2023
156c0e1
rename Dockerfile for slim
tommartensen Nov 21, 2023
6b393ab
Update RHTAP references (#1444)
red-hat-konflux[bot] Nov 27, 2023
3f5a901
Update .tekton/collector-slim-pull-request.yaml
tommartensen Nov 27, 2023
b867082
some recommendations from PR
tommartensen Nov 27, 2023
e3ea3cc
restrict pipeline for *rhtap* branches
tommartensen Nov 28, 2023
f62607e
cleanup
tommartensen Dec 4, 2023
ce0f293
Update RHTAP references (#1452)
red-hat-konflux[bot] Dec 4, 2023
fc214da
empty
tommartensen Dec 4, 2023
5adbce7
empty to check new pod limits
tommartensen Dec 4, 2023
9e2bcb1
Update RHTAP references (#1455)
red-hat-konflux[bot] Dec 5, 2023
ac2e9d8
empty commit
tommartensen Dec 5, 2023
8d41a38
test with CPU limits
tommartensen Dec 5, 2023
0710d12
Update RHTAP references (#1463)
red-hat-konflux[bot] Dec 6, 2023
cc2fd34
Update RHTAP references (#1464)
red-hat-konflux[bot] Dec 11, 2023
7a5b8a1
Update RHTAP references (#1467)
red-hat-konflux[bot] Dec 12, 2023
05a86d4
update Dockerfile with midstream changes
tommartensen Dec 12, 2023
bc4405f
Merge branch 'tm/rhtap-onboarding' of github.com:stackrox/collector i…
tommartensen Dec 12, 2023
bda0de0
Apply suggestions from code review
tommartensen Dec 12, 2023
bc5cf90
Merge branch 'master' into tm/rhtap-onboarding
tommartensen Dec 12, 2023
d44cd27
Merge branch 'tm/rhtap-onboarding' of github.com:stackrox/collector i…
tommartensen Dec 12, 2023
fb4ded4
fix comment
tommartensen Dec 12, 2023
1541146
Update RHTAP references (#1471)
red-hat-konflux[bot] Dec 13, 2023
0fbf75f
setup Snyk for collector
tommartensen Dec 13, 2023
be05ea1
bump to get green RHTAP CI
tommartensen Dec 13, 2023
4801067
empty commit for retrigger
tommartensen Dec 14, 2023
6b38abe
Update RHTAP references (#1474)
red-hat-konflux[bot] Dec 18, 2023
de9d423
Update RHTAP references (#1477)
red-hat-konflux[bot] Dec 20, 2023
9178b08
Update RHTAP references (#1478)
red-hat-konflux[bot] Jan 3, 2024
11e5321
add source image build task
tommartensen Jan 3, 2024
e66d192
test enabled builds
tommartensen Jan 3, 2024
ce6314a
fix workspaces
tommartensen Jan 3, 2024
ea19451
hack the rpm installation
tommartensen Jan 3, 2024
cbd01bc
skip cleanup
tommartensen Jan 3, 2024
86278b0
bodge
tommartensen Jan 3, 2024
f0099d9
add the three dependencies that are RHEL RPM only
tommartensen Jan 3, 2024
d6d4e6a
empty commit
tommartensen Jan 3, 2024
b3d216d
fix missing dependency installation
tommartensen Jan 3, 2024
8bd5588
give it a shot w/ the midstream Dockerfile
tommartensen Jan 4, 2024
242bfc0
fix installations in second stage by using ubi-nomrla
tommartensen Jan 4, 2024
fc6fc25
empty commit
tommartensen Jan 4, 2024
2a0117e
Merge branch 'master' into tm/rhtap-onboarding
tommartensen Jan 4, 2024
6681437
Merge branch 'tm/rhtap-onboarding' into tm/rhtap-dnf-enablement
tommartensen Jan 4, 2024
c84c61b
inject collector_version
tommartensen Jan 4, 2024
7bf4bdc
empty commit
tommartensen Jan 4, 2024
1fed215
empty to retrigger
tommartensen Jan 11, 2024
2acdd3e
format according to collector team's preferences
tommartensen Jan 11, 2024
8afcd6d
try with a ubi-minimal final stage
tommartensen Jan 11, 2024
902307e
fix last stage
tommartensen Jan 11, 2024
42207c9
empty
tommartensen Jan 15, 2024
e0d3640
Merge branch 'master' into tm/rhtap-dnf-enablement
tommartensen Jan 19, 2024
7bdaf04
Merge branch 'master' into tm/rhtap-dnf-enablement
tommartensen Feb 22, 2024
923b6f6
update subscription-manager-bro.sh script
tommartensen Feb 22, 2024
3572405
use scratch instead for final stage
tommartensen Feb 22, 2024
20050e5
fix
tommartensen Feb 22, 2024
a564841
2nd part
tommartensen Feb 22, 2024
ad7fb33
fix kernel-modules directory
tommartensen Feb 22, 2024
53b88d9
empty commit
tommartensen Feb 26, 2024
a00eb62
empty
tommartensen Feb 26, 2024
37dcf9e
add missing labels
tommartensen Feb 26, 2024
23d4abc
rename stage
tommartensen Feb 27, 2024
49c9860
cosmetic reorder
tommartensen Feb 27, 2024
5728876
rename script .rhtap -> .konflux
tommartensen Feb 27, 2024
e56e81a
port changes to all tekton pipelines
tommartensen Feb 27, 2024
82f3846
reduce builder to single stage
tommartensen Feb 27, 2024
cd34e5f
Revert "reduce builder to single stage"
tommartensen Feb 27, 2024
f1e1666
update full Dockerfile
tommartensen Feb 27, 2024
5520ffa
make the shfmt linter happy
tommartensen Feb 27, 2024
b447995
register
tommartensen Feb 27, 2024
08f069d
desperation
tommartensen Feb 27, 2024
ae2ff4f
remove gitignore
tommartensen Feb 27, 2024
dc1144e
Merge branch 'master' into tm/konflux-scratch-final-stage
tommartensen Feb 27, 2024
8aa4c0e
make rhtap maintainers owners of the konflux dir
tommartensen Feb 27, 2024
bb164dd
Update .tekton/collector-pull-request.yaml
tommartensen Feb 28, 2024
79d1c99
apply suggestion for workspace path
tommartensen Feb 28, 2024
c41368a
make prefetch and custom tasks run in parallel
tommartensen Feb 28, 2024
2eadede
link instead of copy; remove redundant info
tommartensen Feb 28, 2024
87f70d2
use a staging dir to avoid overriding existing content in builder image
tommartensen Feb 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ RELEASED_VERSIONS.unsupported @stackrox/collector-team
# The RHTAP maintainers for ACS review all changes related to the RHTAP pipelines, such as new pipelines,
# parameter changes or automated task updates.
/.tekton/ @stackrox/rhtap-maintainers
/.konflux/ @stackrox/rhtap-maintainers
3 changes: 3 additions & 0 deletions .konflux/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# .konflux

This is a directory with misc files needed for builds on Konflux.
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
set -euo pipefail

SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")"
SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd)"

SECRET_NAME_IN_KONFLUX="subscription-manager-activation-key"
SECRET_KONFLUX_WORKSPACE_PATH="/workspace/${SECRET_NAME_IN_KONFLUX}"
Expand All @@ -30,10 +30,9 @@ TARGET_BACKUP_PATHS=(
var/cache/ldconfig
)


function main {
if [[ "$#" == "0" ]] ; then
>&2 echo "Error: command is missing. See the usage below."
if [[ "$#" == "0" ]]; then
echo >&2 "Error: command is missing. See the usage below."
usage
exit 2
fi
Expand All @@ -44,26 +43,32 @@ function main {
local fn

case "$cmd" in
"help" | "--help" | "-h")
fn=usage ;;
"smuggle")
fn=smuggle ;;
"register")
fn=register ;;
"cleanup")
fn=cleanup ;;
"self-test")
fn=self_test ;;
"diff")
fn=assert_diff ;;
*)
>&2 echo "Error: unknown command '$1'; call '$SCRIPT_NAME help' to see the usage."
exit 2
;;
"help" | "--help" | "-h")
fn=usage
;;
"smuggle")
fn=smuggle
;;
"register")
fn=register
;;
"cleanup")
fn=cleanup
;;
"self-test")
fn=self_test
;;
"diff")
fn=assert_diff
;;
*)
echo >&2 "Error: unknown command '$1'; call '$SCRIPT_NAME help' to see the usage."
exit 2
;;
esac

if [[ "$#" -gt "0" && "$cmd" != "diff" && "$cmd" != "register" ]]; then
>&2 echo "Error: too many arguments; call '$SCRIPT_NAME help' to see the usage."
echo >&2 "Error: too many arguments; call '$SCRIPT_NAME help' to see the usage."
exit 2
fi

Expand Down Expand Up @@ -117,24 +122,24 @@ function usage {
}

function smuggle {
mkdir -p "$(dirname "${SECRET_LOCAL_PATH}" )"
mkdir -p "$(dirname "${SECRET_LOCAL_PATH}")"
cp --verbose "${SECRET_KONFLUX_WORKSPACE_PATH}/${SECRET_KEY}" "${SECRET_LOCAL_PATH}"
}

function register {
if [[ ! -s "${SECRET_LOCAL_PATH}" ]]; then
>&2 echo "Error: it does not look like the activation key is present in ${SECRET_LOCAL_PATH}"
echo >&2 "Error: it does not look like the activation key is present in ${SECRET_LOCAL_PATH}"
exit 3
fi
local secret
secret="$(cat "${SECRET_LOCAL_PATH}")"

if [[ "$#" -lt 1 ]]; then
>&2 echo "Error: target path(s) must be provided for the 'register' command."
echo >&2 "Error: target path(s) must be provided for the 'register' command."
exit 2
fi

local target_dirs=( "$@" )
local target_dirs=("$@")

check_targets_and_store_paths_for_cleanup "${target_dirs[@]}"

Expand Down Expand Up @@ -170,17 +175,17 @@ function register {
}

function check_targets_and_store_paths_for_cleanup {
local target_dirs=( "$@" )
local target_dirs=("$@")

for target_dir in "${target_dirs[@]}"; do
if [[ ! -d "${target_dir}/etc" ]]; then
>&2 echo "Error: Looks like the target system is not placed at ${target_dir}"
echo >&2 "Error: Looks like the target system is not placed at ${target_dir}"
exit 4
fi
done

if [[ -f "${TARGETS_LIST_FILE}" ]]; then
>&2 echo "Error: ${TARGETS_LIST_FILE} already exists. Are you trying to register again without doing a cleanup?"
echo >&2 "Error: ${TARGETS_LIST_FILE} already exists. Are you trying to register again without doing a cleanup?"
exit 5
fi

Expand Down Expand Up @@ -254,7 +259,7 @@ function self_test {

function assert_diff {
if [[ "$#" != "2" ]]; then
>&2 echo "Error: expecting two arguments: expected and actual paths"
echo >&2 "Error: expecting two arguments: expected and actual paths"
exit 2
fi

Expand All @@ -265,7 +270,7 @@ function assert_diff {
failed_check_file="$(mktemp)"

echo "Comparing /etc"
if ! diff --brief --recursive --no-dereference --exclude='ld.so.cache' "$expected/etc" "$actual/etc" ; then
if ! diff --brief --recursive --no-dereference --exclude='ld.so.cache' "$expected/etc" "$actual/etc"; then
echo 1 >> "$failed_check_file"
fi

Expand All @@ -284,8 +289,8 @@ function assert_diff {
echo '/var/cache(: |/)dnf'
} >> "$var_exclusions"

if { diff --brief --recursive --no-dereference "$expected/var" "$actual/var" || true; } | \
grep -vEf "$var_exclusions" | { grep '.'; }; then
if { diff --brief --recursive --no-dereference "$expected/var" "$actual/var" || true; } \
| grep -vEf "$var_exclusions" | { grep '.'; }; then
msugakov marked this conversation as resolved.
Show resolved Hide resolved
echo 2 >> "$failed_check_file"
fi

Expand All @@ -299,7 +304,7 @@ function assert_diff {
done

if [[ -s "$failed_check_file" ]]; then
>&2 echo "Error: differences detected"
echo >&2 "Error: differences detected"
exit 6
fi

Expand Down
1 change: 0 additions & 1 deletion .rhtap/.gitignore

This file was deleted.

3 changes: 0 additions & 3 deletions .rhtap/README.md

This file was deleted.

21 changes: 20 additions & 1 deletion .tekton/collector-pull-request.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ spec:
- name: git-auth
secret:
secretName: '{{ git_auth_secret }}'
- name: subscription-manager-activation-key
secret:
secretName: subscription-manager-activation-key

# The pipeline regularly takes >1h to finish.
timeouts:
Expand Down Expand Up @@ -173,6 +176,7 @@ spec:
workspaces:
- name: workspace
- name: git-auth
- name: subscription-manager-activation-key

tasks:

Expand Down Expand Up @@ -221,12 +225,27 @@ spec:
- name: basic-auth
workspace: git-auth

- name: prepare-rhel-rpm-subscriptions
runAfter:
- clone-repository
workspaces:
- name: source
workspace: workspace
- name: subscription-manager-activation-key
workspace: subscription-manager-activation-key
taskSpec:
steps:
# TODO(ROX-20651): use content sets instead of subscription manager for access to RHEL RPMs once available.
- name: smuggle-activation-key
image: registry.access.redhat.com/ubi8/ubi:latest
script: exec /workspace/source/source/.konflux/scripts/subscription-manager-bro.sh smuggle
tommartensen marked this conversation as resolved.
Show resolved Hide resolved

- name: prefetch-dependencies
params:
- name: input
value: $(params.prefetch-input)
runAfter:
- clone-repository
- prepare-rhel-rpm-subscriptions
msugakov marked this conversation as resolved.
Show resolved Hide resolved
taskRef:
params:
- name: name
Expand Down
21 changes: 20 additions & 1 deletion .tekton/collector-push.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ spec:
- name: git-auth
secret:
secretName: '{{ git_auth_secret }}'
- name: subscription-manager-activation-key
secret:
secretName: subscription-manager-activation-key

# The pipeline regularly takes >1h to finish.
timeouts:
Expand Down Expand Up @@ -173,6 +176,7 @@ spec:
workspaces:
- name: workspace
- name: git-auth
- name: subscription-manager-activation-key

tasks:

Expand Down Expand Up @@ -221,12 +225,27 @@ spec:
- name: basic-auth
workspace: git-auth

- name: prepare-rhel-rpm-subscriptions
runAfter:
- clone-repository
workspaces:
- name: source
workspace: workspace
- name: subscription-manager-activation-key
workspace: subscription-manager-activation-key
taskSpec:
steps:
# TODO(ROX-20651): use content sets instead of subscription manager for access to RHEL RPMs once available.
- name: smuggle-activation-key
image: registry.access.redhat.com/ubi8/ubi:latest
script: exec /workspace/source/source/.konflux/scripts/subscription-manager-bro.sh smuggle

- name: prefetch-dependencies
params:
- name: input
value: $(params.prefetch-input)
runAfter:
- clone-repository
- prepare-rhel-rpm-subscriptions
taskRef:
params:
- name: name
Expand Down
6 changes: 3 additions & 3 deletions .tekton/collector-slim-pull-request.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,7 @@ spec:
- name: basic-auth
workspace: git-auth

- name: download-external-content
- name: prepare-rhel-rpm-subscriptions
runAfter:
- clone-repository
workspaces:
Expand All @@ -230,14 +230,14 @@ spec:
# TODO(ROX-20651): use content sets instead of subscription manager for access to RHEL RPMs once available.
- name: smuggle-activation-key
image: registry.access.redhat.com/ubi8/ubi:latest
script: exec /workspace/source/source/.rhtap/scripts/subscription-manager-bro.sh smuggle
script: exec /workspace/source/source/.konflux/scripts/subscription-manager-bro.sh smuggle

- name: prefetch-dependencies
params:
- name: input
value: $(params.prefetch-input)
runAfter:
- download-external-content
- prepare-rhel-rpm-subscriptions
taskRef:
params:
- name: name
Expand Down
21 changes: 20 additions & 1 deletion .tekton/collector-slim-push.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ spec:
- name: git-auth
secret:
secretName: '{{ git_auth_secret }}'
- name: subscription-manager-activation-key
secret:
secretName: subscription-manager-activation-key

pipelineSpec:

Expand Down Expand Up @@ -167,6 +170,7 @@ spec:
workspaces:
- name: workspace
- name: git-auth
- name: subscription-manager-activation-key

tasks:

Expand Down Expand Up @@ -213,12 +217,27 @@ spec:
- name: basic-auth
workspace: git-auth

- name: prepare-rhel-rpm-subscriptions
runAfter:
- clone-repository
workspaces:
- name: source
workspace: workspace
- name: subscription-manager-activation-key
workspace: subscription-manager-activation-key
taskSpec:
steps:
# TODO(ROX-20651): use content sets instead of subscription manager for access to RHEL RPMs once available.
- name: smuggle-activation-key
image: registry.access.redhat.com/ubi8/ubi:latest
script: exec /workspace/source/source/.konflux/scripts/subscription-manager-bro.sh smuggle

- name: prefetch-dependencies
params:
- name: input
value: $(params.prefetch-input)
runAfter:
- clone-repository
- prepare-rhel-rpm-subscriptions
taskRef:
params:
- name: name
Expand Down
Loading
Loading