LifeCycle test fails since 'unpublish' is not run

[tikolsky]

On branch master - in "volume lifecycle" test (func VolumeLifecycle), seems like the last method running is ControllerPublishVolumeRequest and nothing after that, which causes the test to fail since it expects unpublish.

on the other hand, on branch release-3.1, Lifecycle tests pass.

This is the failure from master branch:

CSI Driver Test Suite.Controller Service [Controller Server] volume lifecycle should workFailing for the past 2 builds (Since #580 )Took 0.44 sec.add descriptionStacktrace/usr/local/go/src/github.com/kubernetes-csi/csi-test/pkg/sanity/controller.go:119
recorded 1 failure(s)
/go/pkg/mod/github.com/onsi/[email protected]/internal/leafnodes/runner.go:113Standard Output�[1mSTEP�[0m: reusing connection to CSI driver at /tmp/k8s_dir/nodecsi
�[1mSTEP�[0m: reusing connection to CSI driver controller at /tmp/k8s_dir/f
�[1mSTEP�[0m: creating mount and staging directories
�[1mSTEP�[0m: getting node information
�[1mSTEP�[0m: creating a single node writer volume
�[1mSTEP�[0m: calling controllerpublish on that volume
cleanup volumes: deleting A9000:6001738CFC9035EA000000000078E5C7
cleanup volumes: DeleteVolume failed: rpc error: code = Internal desc = an internal exception occurred : The volume mapped to a host cannot be deleted.

and:

CSI Driver Test Suite.Controller Service [Controller Server] volume lifecycle should be idempotentFailing for the past 2 builds (Since #580 )Took 23 ms.add descriptionStacktrace/usr/local/go/src/github.com/kubernetes-csi/csi-test/pkg/sanity/controller.go:1066
volume create failed
Unexpected error:
    <*status.Error | 0xc0001fe550>: {
        Code: 6,
        Message: "Volume was already created with different size.",
        Details: nil,
        XXX_NoUnkeyedLiteral: {},
        XXX_unrecognized: nil,
        XXX_sizecache: 0,
    }
    rpc error: code = AlreadyExists desc = Volume was already created with different size.
occurred
/usr/local/go/src/github.com/kubernetes-csi/csi-test/pkg/sanity/controller.go:1659Standard Output�[1mSTEP�[0m: reusing connection to CSI driver at /tmp/k8s_dir/nodecsi
�[1mSTEP�[0m: reusing connection to CSI driver controller at /tmp/k8s_dir/f
�[1mSTEP�[0m: creating mount and staging directories
�[1mSTEP�[0m: getting node information
�[1mSTEP�[0m: creating a single node writer volume

[pohly]

@timoreimann is this perhaps related to your cleanup changes?

/help wanted

[pohly]

/help

[k8s-ci-robot]

@pohly:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[timoreimann]

I wonder if #297 (which I thought was long merged but I just realized I still have to follow up on) fixes this problem.

[ArbelNathan]

@timoreimann even after the branch was merged, we are still getting that error

[timoreimann]

@ArbelNathan is your driver possibly affected by the fact that cleanup currently only happens in LIFO order? Some drivers may require FIFO order which #319 is meant to address (and that I should find some time to work on soon).

If that's not the case, then it might be a different problem. In that case, a dedicated issue would be appreciated if yet missing.

[ArbelNathan]

this issue is about a simpler scenario than snapshots (though our driver requires LIFO order) - DeleteVolume is called before controller unpublish.
so it is indeed a different problem, and this issue is dedicated to that problem.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[timoreimann]

/remove-lifecycle stale

[cospotato]

Hello @timoreimann ! according to spec. DeleteVolume RPC will error when volume in_use. So i wonder that what is the interpretation of in_use in Kubernetes-csi ? For example, if i ControllerPublishVolume to the specific node, does it mean the Volume in_use ?

[ArbelNathan]

we are still getting the same error, is there any reason why the unpublish is not called before trying deleting the volume?

[llabordehpe]

I have been dealing with this same problem for the last couple days where my CSI driver fails the tests "volume lifecycle should work" and "volume lifecycle should be idempotent" because the csi-sanity tool calls DeleteVolume before ControllerUnpublishVolume in the cleanup path.
I traced back the problem to this commit from last year, which unsurprisingly dealt with the cleanup of created resources:
e8a621c
Is @timoreimann still around to fix it?

[ArbelNathan]

@llabordehpe can you be more specific on the cause of the problem?
maybe we could donate to the project and open a PR with a fix.

[ArbelNathan]

Do we even know that unpublish (in cleanup only) is working?
do we have a job that its cleanup that contains unpublish? (besides volume LifeCycle)