The Dataset Lifecycle Framework enables users or administrators of Kubernetes clusters to easily link applications with data sources. Thanks to the new Dataset CRD (Custom Resource Definition) all you need to do is to create a dataset, and include the dataset ID as a label in your pods specification. Annotated pods will have access to tha data with no need for providing any futher information on tha data sources. Our framework takes care of all the dirty details of mounting or giving your pods access to the data. Once a dataset exists in a Kubernetes cluster, users will just need to reference it using the unique ID defined at creation time.
This framework targets any Kubernetes 1.15+ installation; it is built on the Operator SDK and is extensible to support any CSI enabled storage system.
- Support for NFS directories as Datasets pull/3
- Support lookup for Dataset information in Hive metastore pull/4
- Integration with Multi-Cloud Object Gateway (Noobaa)
- Pluggable caching mechanism
- Optimizations for data-aware scheduling
If you prefer to watch a quick demo of its functionality, have a look at the recording: Demo
The following steps demonstrate how to quickly getting started with our framework using minikube. Check the minikube documentation for instructions about how to install it. In case you want to deploy our framework on a proper kubernetes cluster inspect the Makefile to tailor your Dataset Lifecycle Framework installation.
- Docker
- git
- Kubernetes CLI (kubectl)
Before starting this step, please make sure your Kubernetes CLI (kubectl) is
properly configured to interact with your minikube environment. The command make minikube-install will
take a bit as it builds the framework's components from scratch.
Make also sure our minikube environment has enough resources, below the minimal configuration to ensure a successful run:
$ minikube start --memory='6000mb' --cpus=4To start the installation:
$ make minikube-installVerify the installation by making sure the following pods are running:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
csi-attacher-nfsplugin-0 2/2 Running 0 15s
csi-attacher-s3-0 1/1 Running 0 17s
csi-nodeplugin-nfsplugin-25x92 2/2 Running 0 15s
csi-provisioner-s3-0 2/2 Running 0 18s
csi-s3-qvbfm 2/2 Running 0 18s
dataset-operator-54b74d5885-zb78z 1/1 Running 0 15s
example-noobaa-data-9rnxd 0/1 Completed 0 6m4s
my-pv-bs-noobaa-noobaa-0 1/1 Running 0 8m19s
my-pv-bs-noobaa-noobaa-1 1/1 Running 0 8m15s
my-pv-bs-noobaa-noobaa-2 1/1 Running 0 8m10s
noobaa-core-0 2/2 Running 0 9m22s
noobaa-operator-7d479b7f7b-m98fs 1/1 Running 0 9m44s
As part of the minikube installation we deployed NooBaa and
added sample data for demo purposes. An S3 bucket is created in NooBaa using an
object bucket claim (OBC).
We also download the NooBaa CLI (v2.0.10) in ./examples/noobaa/noobaa.
As a user now you can use any Dataset stored on NooBaa inside your pods.
To check the NooBaa installation configuration run the following command:
$ ./examples/noobaa/noobaa status
#------------------#
#- Mgmt Addresses -#
#------------------#
ExternalDNS : []
ExternalIP : []
NodePorts : [https://192.168.39.122:30827]
InternalDNS : [https://noobaa-mgmt.default.svc:443]
InternalIP : [https://10.102.146.205:443]
PodPorts : [https://172.17.0.5:8443]
#--------------------#
#- Mgmt Credentials -#
#--------------------#
email : [email protected]
password : HM7HwZJ+DG+1MO5FnJSONA==
#----------------#
#- S3 Addresses -#
#----------------#
ExternalDNS : []
ExternalIP : []
NodePorts : [https://192.168.39.122:32434]
InternalDNS : [https://s3.default.svc:443]
InternalIP : [https://10.103.161.163:443]
PodPorts : [https://172.17.0.5:6443]
#------------------#
#- S3 Credentials -#
#------------------#
AWS_ACCESS_KEY_ID : M62tJattECtRRseKLkb0
AWS_SECRET_ACCESS_KEY : wSMf8YVqF0cyFIYc2APwMLVBQL7rUfmaZ9ekOsi8
#------------------#
#- Backing Stores -#
#------------------#
NAME TYPE TARGET-BUCKET PHASE AGE
my-pv-bs pv-pool Ready 5m1s
#------------------#
#- Bucket Classes -#
#------------------#
NAME PLACEMENT PHASE AGE
noobaa-default-bucket-class {Tiers:[{Placement: BackingStores:[my-pv-bs]}]} Ready 4m51s
#-----------------#
#- Bucket Claims -#
#-----------------#
NAMESPACE NAME BUCKET-NAME STORAGE-CLASS BUCKET-CLASS PHASE
default my-bucket-claim my-bucket-66d89807-3dda-426f-a5df-4c3b790acff5 default.noobaa.io noobaa-default-bucket-class Bound
The above screenshot shows the bucket created for this demo and some other information such as S3 endpoint and credentials that you'll need later in this tutorial.
To inspect the newly created NooBaa bucket run:
$ ./examples/noobaa/noobaa ui
INFO[0000] ✅ Exists: NooBaa "noobaa"
INFO[0000] ✅ Exists: Service "noobaa-mgmt"
INFO[0000] ✅ Exists: Secret "noobaa-operator"
NooBaa UI (credentials unless using Openshift SSO):
url : https://localhost:44261
email : [email protected]
password : HM7HwZJ+DG+1MO5FnJSONA==
A browser session will open up. Login with the credentials provided. You'll see something like the following screenshot if looking at the bucket info.
Let's try to access this bucket via a Dataset. What you'll need now is the
information on NooBaa configuration (AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY), the S3 endpoint and the bucket name.
No worries, we have a script that collects all the info needed and generates
a ready-to-go Dataset description from our templates:
$ ./examples/noobaa/create_dataset_desc.shthis will generate the file ./examples/noobaa/dataset-noobaa.yaml that will
look like something like this:
apiVersion: com.ie.ibm.hpsys/v1alpha1
kind: Dataset
metadata:
name: example-dataset
spec:
local:
type: "COS"
accessKeyID: "iQkv3FABR0eywcEeyJAQ"
secretAccessKey: "MIK3FPER+YQgb2ug26osxP/c8htr/05TVNJYuwmy"
endpoint: "http://192.168.39.245:31772"
bucket: "my-bucket-d4078283-dc35-4f12-a1a3-6f32571b0d62"
region: "" #it can be empty
If curious about how the various fields were retrieved please have a look at the create_dataset_desc.sh script. If instead of the provided NooBaa installation you want to test with another S3 based Cloud Object Storage bucket, feel free to do it. Just make sure the endpoint, accessKeyID, bucket and secretAccessKey fields are properly filled to connect to your bucket.
It is now time to create the dataset and submit a pod that uses it:
$ kubectl create -f ./examples/noobaa/dataset-noobaa.yaml
$ kubectl create -f ./examples/templates/example-pod.yamlVerify the dataset was properly created:
$ kubectl get datasets
NAME AGE
example-dataset 5sNow inspect the example-pod to see how to use the newly created example-dataset
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
dataset.0.id: "example-dataset"
dataset.0.useas: "mount"
spec:
containers:
- name: nginx
image: nginx
volumeMounts:
- mountPath: "/mount/dataset1" #optional, if not specified it would be mounted in /mnt/datasets/example-dataset
name: "example-dataset"
With the following command you can inspect the running nginx pod to verify the dataset is mounted at the provided
path
$ kubectl exec -it nginx ls /mount/dataset1
file1.txt file2.txt
Feel free to test adding new content to the NooBaa bucket and verify it is immediately available to the pod by re executing the above command.
Notice the way we annotate the pod to make it aware of the datasets. For instance if we wanted to use multiple datasets, in the labels section we would have something like this:
dataset.0.id: dataset-0
dataset.0.useas: mount
dataset.1.id: dataset-1
dataset.1.useas: mount
dataset.2.id: dataset-2
dataset.2.useas: mount
The mountPath and name in volumeMounts is optional and should be used if
the user wants to mount a dataset in a specific location.
If the user doesn't specify the mount point, as a convention we will mount the
dataset on /mnt/datasets/<dataset-name>.
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 825061.
