Race between pvc mount and multipathd path addition after upgrading to kubernetes 1.20.2 #511
Comments
|
Hi @ffilippopoulos, We've tried to reproduce this issue internally and haven't been able to do so. There may be something specific to your environment that is causing this issue. Please contact NetApp Support so that we can investigate this issue further. To open a case with NetApp, please go to https://www.netapp.com/company/contact-us/support/ |
|
@ffilippopoulos Hey we have been experimenting with Trident on top of of flatcar Linux and we are observing periodic core dumps from iscsiadm. Have you seen anything like that in your logs? Something like this is what we're seeing: time="2021-01-29T21:11:22Z" level=debug msg="<<<< osutils.execCommand." command=iscsiadm error="signal: segmentation fault (core dumped)" output= It's possible that we've screwed up our Flatcar Linux installation somehow, but it's also possible that Flatcar Linux has a screwed up Open iSCSI userspace. As you know Flatcar Linux isn't a supported Linux distro for Trident, but there's every reason that it should probably work as long as the kernel and userspace tools are high enough versions and not buggy. The coredumps make me suspect there could be bugs, though. |
|
hi @bswartz, tyvm for looking at this issue! We do not see any core dumps as you posted, but we see the following logs when the mount gets stuck: Please note that this appears only on nodes of our cluster that run kubernetes 1.20.2, so we believe that the kube version upgrade is significant here. As stated in the description the issue we have is probably because multipathd does not add the path under the respective dm in time and trident decides to proceed by mounting directly one of the paths (
Also note that we tried updating flatcar to the latest version today (2605.11.0 which comes with a iscsi bump to version 2.1.3) and we ran into the core dumps you posted above. |
|
Digging a little bit more into that, I believe that our problem isn't related with multipath, but with iscsi scans failing to find all the devices in time: In the logs above, it seems like trident finds only one device (sdc) and thus decides to skip multipath discovery and proceed mounting the device scanned. That is when the race with multipath occurs. |
|
@bswartz I've run into on old comment of yours for a relevant situation: kubernetes/kubernetes#60894 (comment) Do you think trident could also follow a similar approach, where a delay is introduced based on the number of portals in case a smaller amount of volumes is discovered? |
|
based on the above ^ I've tried this: https://github.com/ffilippopoulos/trident/pull/1/files and looks to do the job. This is basically checking If you guys think that this approach will not mess with anything else we could even PR it directly. |
|
Sorry I haven't been looking at this issue the last few days @ffilippopoulos. I think your analysis is on to something. Trident's current multipath behavior is a little fragile because we don't allow users anywhere to tell Trident if they want multipathing enabled or disabled -- we simply try to detect it, and in your case we may be detecting it wrong. It isn't the first time I've seen a problem like this, and my first instinct to address the problem is to add some sort of control knob that users can use to explicitly turn multipathing on or off. In the on case, we could be more aggressive about waiting for the multipath devices to show up. I'm worried that changing it to wait in ALL cases will introduce problems in situations where multipathing either isn't enabled, or it's in a degraded state. Let me know if you disagree or have an alternative suggestion. I'm going to take a look at adding a top level option to the Trident backend to control this behavior. |
it will at least create a not needed delay, depending on how much time is allowed waiting for multipath devices, that is why in the code changes I linked above after forking trident, I only allow it to try for 5 seconds. |
|
The most obvious risk is when multipathing actually is enabled, and you have a path down at the moment you try to attach a pod. In that case, it's not the best policy to wait for the down path to come back up to allow the pod to start up. |
|
yup, I agree, that is why we are talking about a small wait window (in my case I added 5 seconds but I think it could be less). After that we'd have to proceed with what is discovered. |
|
Also, note that the 5 seconds period might not be enough (observed that loaded nodes in a cluster rollout might need more). It would be worthy to make the wait period configurable imo. |
I strongly recommend setting |
|
Please refer to open-iscsi/open-iscsi#252 for the open-iscsi bug. |
|
Is there any update regarding this one? |
|
I think I've been seeing a related problem with an older stack. iSCSI mounts fail (sometimes also to SolidFire, which I briefly set up to check if it's only ONTAP-related) although PVC/PV are both bound - it seems that iSCSI and/or multipath aren't ready and I in the pod I get I'll try to use @ffilippopoulos' patch and do more testing, but if anyone has any other patches or ideas (I'll try w/o find_multipath as well), let us know what else can we try. Tangential to this: ONTAP best practices for Linux recommend to enable |
|
Hello, we have been running with utilitywarehouse#3 patch, which implements @bswartz 's suggestion of having an explicit We can confirm, this also mitigates our issue, and is an alternative to the #568 solution. In the conversation on the case with NetAPP support (https://www.netapp.com/company/contact-us/support/), we have been told that NetAPP cannot support this change, are working on their own fix, which will "probably be available in the next release". After which our case was closed, and we are told to "follow our newsletter". |
|
Hi @george-angel, While investigating this issue further it was discovered that the mpath flag isn't necessary. It was also decided that additional improvements and refactoring of the osutils.go iSCSI code are needed. In order to complete these code changes there are also changes needed to our functional test suite (i.e. multipathd manipulation, network connectivity, etc...). Once we are further along with the testing and are convinced that this issue is solved and backwards compatibility is maintained we will be able to share more information with you. Thanks for your collaboration so far. |
|
Thanks for the update @gnarl 👍 |
|
This issue was addressed with commit 2eee92b and will be included in the Trident 21.07 release. |
Describe the bug
After upgrading to kubernetes 1.20.2 we see that occasionally pvc mounts fail because the path (etc ./dev/sdc) that trident is trying to mount is already locked via multipathd.
trident-csipods error that the device is/dev/sdc already mounted or mount point busy.and keep looping over and over.Environment
Provide accurate information about the environment to help us reproduce the issue.
To Reproduce
Use the latest trident version on a kubernetes 1.20.2 cluster and kick pods with a pvc a few times until the error occurs.
Expected behavior
Trident should wait for multipath devices to be present or be able to recover and look again after a mount failure.
Additional context
Describing kube pods look like:
From trident-csi debug logs we see:
The reason that this path is busy on the host seems to be multipath:
also sdc is now pointing to the correct dm:
Please note the intermittent mount failures appeared only after upgrading kubernetes from 1.19.2 to 1.20.2 and we are able to reproduce them quite consistently.
Any advice on how to continue debugging that from our side would be much appreciated.
The text was updated successfully, but these errors were encountered: