Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Jenkinsfile to also build webhook and controller images #6

Closed
wants to merge 2 commits into from
Closed

Fix Jenkinsfile to also build webhook and controller images #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dbab8fe
[#288] feat(operator): Supports mounting PVC for shuffleserver (#1813)
jerqi Jun 20, 2024
652a453
Fix Jenkinsfile to also build webhook and controller images
gibchikafa Oct 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ pipeline {
environment {
VERSION = readFile "${env.WORKSPACE}/version.txt"
BUILD_BRANCH = readFile "${env.WORKSPACE}/build_branch.txt"
DOCKER_REGISTRY = "docker.hops.works"
CONTROLLER_IMAGE = "${DOCKER_REGISTRY}/hopsworks/rss-controller:${VERSION}"
WEBHOOK_IMAGE = "${DOCKER_REGISTRY}/hopsworks/rss-webhook:${VERSION}"
}
stages {
stage("checkout") {
Expand All @@ -26,18 +29,26 @@ pipeline {
sh """
set -ex
echo "Building RSS version ${VERSION} on branch ${BUILD_BRANCH}"
docker login -u ${NEXUS_CREDS_USR} -p ${NEXUS_CREDS_PSW} docker.hops.works
docker login -u ${NEXUS_CREDS_USR} -p ${NEXUS_CREDS_PSW} $DOCKER_REGISTRY

./build_distribution.sh --spark3-profile spark3 --hadoop-profile hadoop3.2 --without-dashboard
cd deploy/kubernetes/docker || exit
./build.sh --hadoop-version 3.2.0.14-EE-RC0 --registry docker.hops.works --nexus-user $NEXUS_CREDS_USR --nexus-password $NEXUS_CREDS_PSW
./build.sh --hadoop-version 3.2.0.14-EE-RC0 --registry $DOCKER_REGISTRY --nexus-user $NEXUS_CREDS_USR --nexus-password $NEXUS_CREDS_PSW
cd ../../..

mkdir -p /opt/repository/master/rss/${VERSION}/
cp client-spark/spark3-shaded/target/rss-client-spark3-shaded-${VERSION}.jar /opt/repository/master/rss/${VERSION}/

# build the controller and webhook images
cd deploy/kubernetes/operator || exit 1
docker build . --progress=plain -t $CONTROLLER_IMAGE --build-arg MODULE=controller -f hack/Dockerfile
docker build . --progress=plain -t $WEBHOOK_IMAGE --build-arg MODULE=webhook -f hack/Dockerfile
# push the controller and webhook images
docker push $CONTROLLER_IMAGE
docker push $WEBHOOK_IMAGE
"""
}
}
}
}
}
}
2 changes: 1 addition & 1 deletion deploy/kubernetes/docker/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ GIT_COMMIT=$(git describe --dirty --always --tags | sed 's/-/./g')
IMAGE_VERSION=$(cat $RSS_DIR/version.txt | awk '{$1=$1; print}' | sed '/^$/d')
IMAGE=$REGISTRY/rss-server:$IMAGE_VERSION
echo "image version: ${IMAGE_VERSION}"
IMAGE=$REGISTRY/rss:$IMAGE_VERSION
IMAGE=$REGISTRY/hopsworks/rss:$IMAGE_VERSION
echo "building image: $IMAGE"

DOCKER_BUILDKIT=1 docker build --network=host -t "$IMAGE" \
Expand Down
23 changes: 23 additions & 0 deletions deploy/kubernetes/operator/api/uniffle/v1alpha1/remoteshuffleservice_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,29 @@ type ShuffleServerConfig struct {

// UpgradeStrategy defines upgrade strategy of shuffle servers.
UpgradeStrategy *ShuffleServerUpgradeStrategy `json:"upgradeStrategy"`

// volumeClaimTemplates is a list of claims that pods are allowed to reference.
// The StatefulSet controller is responsible for mapping network identities to
// claims in a way that maintains the identity of a pod. Every claim in
// this list must have at least one matching (by name) volumeMount in one
// container in the template. A claim in this list takes precedence over
// any volumes in the template, with the same name.
// +optional
VolumeClaimTemplates []ShuffleServerPersistentVolumeClaimTemplate `json:"volumeClaimTemplates,omitempty" protobuf:"bytes,4,rep,name=volumeClaimTemplates"`
}

type ShuffleServerPersistentVolumeClaimTemplate struct {
// May contain labels and annotations that will be copied into the PVC
// when creating it. No other fields are allowed and will be rejected during
// validation.
//
VolumeNameTemplate *string `json:"volumeNameTemplate"`

// The specification for the PersistentVolumeClaim. The entire content is
// copied unchanged into the PVC that gets created from this
// template. The same fields as in a PersistentVolumeClaim
// are also valid here.
Spec corev1.PersistentVolumeClaimSpec `json:"spec" protobuf:"bytes,2,name=spec"`
}

// ShuffleServerUpgradeStrategy defines upgrade strategy of shuffle servers.
Expand Down
28 changes: 28 additions & 0 deletions deploy/kubernetes/operator/api/uniffle/v1alpha1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

199 changes: 199 additions & 0 deletions deploy/kubernetes/operator/config/crd/bases/uniffle.apache.org_remoteshuffleservices.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8362,6 +8362,205 @@ spec:
required:
- type
type: object
volumeClaimTemplates:
description: volumeClaimTemplates is a list of claims that pods
are allowed to reference. The StatefulSet controller is responsible
for mapping network identities to claims in a way that maintains
the identity of a pod. Every claim in this list must have at
least one matching (by name) volumeMount in one container in
the template. A claim in this list takes precedence over any
volumes in the template, with the same name.
items:
properties:
spec:
description: The specification for the PersistentVolumeClaim.
The entire content is copied unchanged into the PVC that
gets created from this template. The same fields as in
a PersistentVolumeClaim are also valid here.
properties:
accessModes:
description: 'accessModes contains the desired access
modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1'
items:
type: string
type: array
dataSource:
description: 'dataSource field can be used to specify
either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot)
* An existing PVC (PersistentVolumeClaim) If the provisioner
or an external controller can support the specified
data source, it will create a new volume based on
the contents of the specified data source. If the
AnyVolumeDataSource feature gate is enabled, this
field will always have the same contents as the DataSourceRef
field.'
properties:
apiGroup:
description: APIGroup is the group for the resource
being referenced. If APIGroup is not specified,
the specified Kind must be in the core API group.
For any other third-party types, APIGroup is required.
type: string
kind:
description: Kind is the type of resource being
referenced
type: string
name:
description: Name is the name of resource being
referenced
type: string
required:
- kind
- name
type: object
x-kubernetes-map-type: atomic
dataSourceRef:
description: 'dataSourceRef specifies the object from
which to populate the volume with data, if a non-empty
volume is desired. This may be any local object from
a non-empty API group (non core object) or a PersistentVolumeClaim
object. When this field is specified, volume binding
will only succeed if the type of the specified object
matches some installed volume populator or dynamic
provisioner. This field will replace the functionality
of the DataSource field and as such if both fields
are non-empty, they must have the same value. For
backwards compatibility, both fields (DataSource and
DataSourceRef) will be set to the same value automatically
if one of them is empty and the other is non-empty.
There are two important differences between DataSource
and DataSourceRef: * While DataSource only allows
two specific types of objects, DataSourceRef allows
any non-core object, as well as PersistentVolumeClaim
objects. * While DataSource ignores disallowed values
(dropping them), DataSourceRef preserves all values,
and generates an error if a disallowed value is specified.
(Beta) Using this field requires the AnyVolumeDataSource
feature gate to be enabled.'
properties:
apiGroup:
description: APIGroup is the group for the resource
being referenced. If APIGroup is not specified,
the specified Kind must be in the core API group.
For any other third-party types, APIGroup is required.
type: string
kind:
description: Kind is the type of resource being
referenced
type: string
name:
description: Name is the name of resource being
referenced
type: string
required:
- kind
- name
type: object
x-kubernetes-map-type: atomic
resources:
description: 'resources represents the minimum resources
the volume should have. If RecoverVolumeExpansionFailure
feature is enabled users are allowed to specify resource
requirements that are lower than previous value but
must still be higher than capacity recorded in the
status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources'
properties:
limits:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
description: 'Limits describes the maximum amount
of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
type: object
requests:
additionalProperties:
anyOf:
- type: integer
- type: string
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
x-kubernetes-int-or-string: true
description: 'Requests describes the minimum amount
of compute resources required. If Requests is
omitted for a container, it defaults to Limits
if that is explicitly specified, otherwise to
an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
type: object
type: object
selector:
description: selector is a label query over volumes
to consider for binding.
properties:
matchExpressions:
description: matchExpressions is a list of label
selector requirements. The requirements are ANDed.
items:
description: A label selector requirement is a
selector that contains values, a key, and an
operator that relates the key and values.
properties:
key:
description: key is the label key that the
selector applies to.
type: string
operator:
description: operator represents a key's relationship
to a set of values. Valid operators are
In, NotIn, Exists and DoesNotExist.
type: string
values:
description: values is an array of string
values. If the operator is In or NotIn,
the values array must be non-empty. If the
operator is Exists or DoesNotExist, the
values array must be empty. This array is
replaced during a strategic merge patch.
items:
type: string
type: array
required:
- key
- operator
type: object
type: array
matchLabels:
additionalProperties:
type: string
description: matchLabels is a map of {key,value}
pairs. A single {key,value} in the matchLabels
map is equivalent to an element of matchExpressions,
whose key field is "key", the operator is "In",
and the values array contains only "value". The
requirements are ANDed.
type: object
type: object
x-kubernetes-map-type: atomic
storageClassName:
description: 'storageClassName is the name of the StorageClass
required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1'
type: string
volumeMode:
description: volumeMode defines what type of volume
is required by the claim. Value of Filesystem is implied
when not included in claim spec.
type: string
volumeName:
description: volumeName is the binding reference to
the PersistentVolume backing this claim.
type: string
type: object
volumeNameTemplate:
description: May contain labels and annotations that will
be copied into the PVC when creating it. No other fields
are allowed and will be rejected during validation.
type: string
required:
- spec
- volumeNameTemplate
type: object
type: array
volumeMounts:
description: VolumeMounts indicates describes mountings of volumes
within shuffle servers' container.
Expand Down
Loading
Loading