To build and install a development version of the driver:
GCE_PD_CSI_STAGING_IMAGE=gcr.io/path/to/driver/image:dev # Location to push dev image to
make push-container
# Modify controller.yaml and node.yaml in ./deploy/kubernetes/dev to use dev image
GCE_PD_DRIVER_VERSION=dev
./deploy/kubernetes/deploy-driver.sh
To bring down driver:
./deploy/kubernetes/delete-driver.sh
The following environment variables will be required for setup and all subsequent test runs.
export PROJECT=$USER-e2e # GCP project used for testing
export GCE_PD_SA_NAME=$PROJECT-pd-sa # Service account name
export IAM_NAME=$GCE_PD_SA_NAME@$PROJECT.iam.gserviceaccount.com # Service account IAM name
export GCE_PD_SA_DIR=~/credentials # Service account directory
export GOOGLE_APPLICATION_CREDENTIALS=$GCE_PD_SA_DIR/cloud-sa.json # Service account key
The following steps will need to be completed once per project.
To create a service account with correct permissions, your test project must:
- Be assigned Project IAM Admin and Role Administrator IAM roles.
- Be part of an organization that allows creation of service account keys.
- For example, a Google internal developer project will not work for E2E testing because the org restricts SA key creation.
export ENABLE_KMS=true
export ENABLE_KMS_ADMIN=true
./deploy/setup-project.sh
- Follow instructions to
generate an SSH key
with the following parameters:
KEY_FILENAME: "google_compute_engine"
USERNAME: "$USER"
- Add your public key (
cat ~/.ssh/google_compute_engine.pub
) to your GCP project metatdata.
./test/run-e2e-local.sh
Local stderr will show you all GCE calls and SSH commands that were performed by the test.
You may see errors during instance creation. It is likely your service account is missing permissions.
- Make sure the resources are showing up in pantheon under [Compute Engine > VM Instances].
- Look for errors in pantheon under [Logging > Logs Explorer].
- Double check and re-run the service account creation step.
You may see errors during driver deployment.
- Double check that you have an SSH key file in the correct directory.
gcloud compute ssh $VM_NAME --zone $ZONE
into the VM instance and attempt to execute the last few commands and look for errors.
./test/run-sanity.sh
./test/run-unit.sh
Performance tests are run in automated testing frameworks, but you may wish to
run them locally to benchmark changes. For a given test configuration, you need
to modify the test-config.yaml
file automatically generated from running one
of the run-k8s-integration...
or run-windows-k8s-integration.sh
scripts to
enable performance tests.
DriverInfo:
...
PerformanceTestOptions:
ProvisioningOptions: # deploy 50 1Gi PVCs and measure latency/throughput
VolumeSize: 1Gi
Count: 50
ExpectedMetrics:
AvgLatency: 10000000000 # 10 seconds
Throughput: 5
You may modify the parameter values to customize the test.
You also need to modify the StorageClass
file pointed to in test-config.yaml
to set volumeBindingMode: Immediate
, as the performance tests only support
this mode.
We will be running the Kubernetes integration tests directly from its
repository. Install kubetest
and set up a test cluster with the driver
installed. Now, cd into $GOPATH/src/k8s.io/kubernetes
and run
# pwd=k/k
kubetest <custom flags based on your provider> \
--check-version-skew=false \
--test \
--test_args="--ginkgo.focus=External.Storage.*volume-lifecycle-performance --allowed-not-ready-nodes=10 --node-os-distro=<linux or windows> --storage.testdriver=<path-to-test-config>"
Use dep
dep ensure
To modify dependencies or versions change ./Gopkg.toml
We use https://github.com/go-delve/delve and its feature for remote debugging. This feature is only available in the PD CSI Controller (which runs in a linux node).
Requirements:
Steps:
- Build the PD CSI driver with additional compiler flags.
export GCE_PD_CSI_STAGING_VERSION=latest
export GCE_PD_CSI_STAGING_IMAGE=image/repo/gcp-compute-persistent-disk-csi-driver
make build-and-push-multi-arch-debug
- Update
deploy/kubernetes/overlays/noauth-debug/kustomization.yaml
to match the repo you wrote above e.g.
images:
- name: gke.gcr.io/gcp-compute-persistent-disk-csi-driver
newName: image/repo/gcp-compute-persistent-disk-csi-driver
newTag: latest
- Delete and deploy the driver with this overlay
./deploy/kubernetes/delete-driver.sh && \
GCE_PD_DRIVER_VERSION=noauth-debug ./deploy/kubernetes/deploy-driver.sh
At this point you could verify that delve is running in the controller logs:
API server listening at: [::]:2345
2021-04-15T18:28:51Z info layer=debugger launching process with args: [/go/src/sigs.k8s.io/gcp-compute-persistent-disk-csi-driver/bin/gce-pd-csi-driver --v=5 --endpoint=unix:/csi/csi.sock]
2021-04-15T18:28:53Z debug layer=debugger continuing
- Enable port forwading of the PD CSI controller of port 2345
kubectl -n gce-pd-csi-driver get pods | grep controller | awk '{print $1}' | xargs -I % kubectl -n gce-pd-csi-driver port-forward % 2345:2345
- Connect to the headless server and issue commands
dlv connect localhost:2345
Type 'help' for list of commands.
(dlv) clearall
(dlv) break pkg/gce-pd-csi-driver/controller.go:509
Breakpoint 1 set at 0x159ba32 for sigs.k8s.io/gcp-compute-persistent-disk-csi-driver/pkg/gce-pd-csi-driver.(*GCEControllerServer).ListVolumes() ./pkg/gce-pd-csi-driver/controller.go:509
(dlv) c
> sigs.k8s.io/gcp-compute-persistent-disk-csi-driver/pkg/gce-pd-csi-driver.(*GCEControllerServer).ListVolumes() ./pkg/gce-pd-csi-driver/controller.go:509 (hits goroutine(69):1 total:1) (PC: 0x159ba32)
Warning: debugging optimized function
504: }
505: }
506:
507: func (gceCS *GCEControllerServer) ListVolumes(ctx context.Context, req *csi.ListVolumesRequest) (*csi.ListVolumesResponse, error) {
508: // https//cloud.google.com/compute/docs/reference/beta/disks/list
=> 509: if req.MaxEntries < 0 {
510: return nil, status.Error(codes.InvalidArgument, fmt.Sprintf(
511: "ListVolumes got max entries request %v. GCE only supports values between 0-500", req.MaxEntries))
512: }
513: var maxEntries int64 = int64(req.MaxEntries)
514: if maxEntries > 500 {
(dlv) req
Command failed: command not available
(dlv) p req
*github.com/container-storage-interface/spec/lib/go/csi.ListVolumesRequest {
MaxEntries: 0,
StartingToken: "",
XXX_NoUnkeyedLiteral: struct {} {},
XXX_unrecognized: []uint8 len: 0, cap: 0, nil,
XXX_sizecache: 0,}
(dlv)
See #742 for the implementation details