[Bug]: Files deleted from local client when external local storage unavailable.

[basos9]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• Nextcloud Server is running on 64bit capable CPU, PHP and OS.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Files synced on a local client that are hosted on external local storage, get deleted when the storage is unmounted. The external storage is a subfolder of mountpoint. Also the https://ncloud.xxx.xx/remote.php/dav/files/xxxx/ seems to return 200 status code for erroneus external local storages while the nextcloud gui shows them red.

Steps to reproduce

1. Add an exernal storage with a subfolder under a mount point e.g. mp /data, ext /data/ext
2. sync files with nextcloud client
3. unmount the mp /data
4. force sync on Nextcloud client
5. Folder ext is deleted from nextcloud client

Expected behavior

The forlder should not be deleted for error conditions.

Maybe an error status code should be responded for these cases see nextcloud/desktop#2454 (comment)

Installation method

Community Manual installation with Archive

Nextcloud Server version

24 and 25.0.6

Operating system

Debian/Ubuntu

PHP engine version

PHP 7.4

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 22.2.3 to 23.0.1)

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "ncloud.xxx.xxx"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/ncloud.xxx.xxx",
        "htaccess.RewriteBase": "\/",
        "dbtype": "mysql",
        "version": "24.0.9.2",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "theme": "",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "loglevel": 2,
        "maintenance": false,
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "mysql.utf8mb4": true,
        "mail_smtpmode": "sendmail",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "trashbin_retention_obligation": "40, 200",
        "versions_retention_obligation": "auto, 30",
        "updater.release.channel": "stable"
    }
}

List of activated Apps

Enabled:
  - accessibility: 1.10.0
  - activity: 2.16.0
  - calendar: 3.5.5
  - circles: 24.0.1
  - cloud_federation_api: 1.7.0
  - comments: 1.14.0
  - contacts: 4.2.4
  - contactsinteraction: 1.5.0
  - dashboard: 7.4.0
  - dav: 1.22.0
  - federatedfilesharing: 1.14.0
  - federation: 1.14.0
  - files: 1.19.0
  - files_external: 1.16.1
  - files_pdfviewer: 2.5.0
  - files_rightclick: 1.3.0
  - files_sharing: 1.16.2
  - files_trashbin: 1.14.0
  - files_versions: 1.17.0
  - files_videoplayer: 1.13.0
  - firstrunwizard: 2.13.0
  - logreader: 2.9.0
  - lookup_server_connector: 1.12.0
  - nextcloud_announcements: 1.13.0
  - notes: 4.5.1
  - notifications: 2.12.1
  - notify_push: 0.5.2
  - oauth2: 1.12.0
  - password_policy: 1.14.0
  - photos: 1.6.0
  - privacy: 1.8.0
  - provisioning_api: 1.14.0
  - recommendations: 1.3.0
  - serverinfo: 1.14.0
  - settings: 1.6.0
  - sharebymail: 1.14.0
  - support: 1.7.0
  - survey_client: 1.12.0
  - systemtags: 1.14.0
  - text: 3.5.1
  - theming: 1.15.0
  - twofactor_backupcodes: 1.13.0
  - updatenotification: 1.14.0
  - user_status: 1.4.0
  - viewer: 1.8.0
  - weather_status: 1.4.0
  - workflowengine: 2.6.0
Disabled:
  - admin_audit
  - encryption
  - user_ldap

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

[kesselb]

Hi, can you share screenshots for the external storage configuration?

[joshtrichards]

We actually do start returning a 503, but not... until it's too late. In the interim we start returning the 404 instead. Obviously that's not good. As far as I can tell the 503 (what we want) doesn't kick in until something manages to trigger the availability check [1] like a user recheck request on the external storage page (simply reloading the external storage settings page also works I believe).

Easy enough to reproduce. Here's how:

• Create a fake drive to mount
  • dd if=/dev/zero of=fakedrive bs=1024 count=10240
  • mkfs -t ext3 fakedrive
• Mount it on the host
  • sudo mount -t ext3 fakedrive /media/fakedrive/
• Put a hello world file in it
• Add the external Local mount in NC
• Do a PROPFIND via cURL on the folder or file[2]
  • 200 aka HTTP/1.1 207 Multi-Status (GOOD)
• Unmount the fake drive on the host (i.e. make it disappear)
• Do a PROPFIND via cURL on it (without doing anything else - i.e. do NOT refresh the settings page)[3]
  • 404 Sabre\DAV\Exception\NotFound (No! After all, we know it's only a StorageNotAvailable / ServiceUnavailable exception since the file exists in the db just not in the filesystem )
• Force a recheck of external storage availability by click the still green (but wrong) icon (I believe reloading the external storage settings page also will suffice)
• Do a PROPFIND via cURL on it [4]
  • HTTP/1.1 503 Service Unavailable / Sabre\DAV\Exception\ServiceUnavailable
• A variation happens in reverse (i.e. when mount is re-added it'll return 503 until an availability check is triggered)

If testing in Docker containers you'll also have to add the bind mount in the app container... then remove it to simulate the removal.

I think the issue is around here:

server/apps/dav/lib/Connector/Sabre/Directory.php

Line 213 in 4df145a

if (is_null($info)) {

and

server/apps/dav/lib/Connector/Sabre/ObjectTree.php

Line 155 in 4df145a

$info = $this->fileView->getFileInfo($path);

Instead of the ServiceUnavailable (5xx) we start returning the NotFound (404) until the availability gets rechecked. But we should be able to use the fact we know the file/folder still exists in the db to assume this is the former rather than the latter, no?

Should also confirm this behaves the same with S3/B2/etc - I assume it does but dunno.

[1]

server/apps/files_external/lib/Config/ConfigAdapter.php

Line 139 in 4df145a

$availability = $storage->getAvailability();

[2] Good (no issues) state

curl  -s -u ncadmin:ncadminpass "http://nc-test.local:3680/remote.php/dav/files/ncadmin/localdrive/helloworld.txt" -X PROPFIND | xml_pp
<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:nc="http://nextcloud.org/ns" xmlns:oc="http://owncloud.org/ns" xmlns:s="http://sabredav.org/ns">
  <d:response>
    <d:href>/remote.php/dav/files/ncadmin/localdrive/helloworld.txt</d:href>
    <d:propstat>
      <d:prop>
        <d:getlastmodified>Wed, 26 Apr 2023 15:46:33 GMT</d:getlastmodified>
        <d:getcontentlength>12</d:getcontentlength>
        <d:resourcetype/>
        <d:getetag>&quot;2bc3404c3e65061e6d4c65aa0e699515&quot;</d:getetag>
        <d:getcontenttype>text/plain</d:getcontenttype>
      </d:prop>
      <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
  </d:response>
</d:multistatus>

[3] Bad (intermediate) state where we start 404ing

 curl  -s -u ncadmin:ncadminpass "http://nc-test.local:3680/remote.php/dav/files/ncadmin/localdrive/helloworld.txt" -X PROPFIND | xml_pp
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\NotFound</s:exception>
  <s:message>File with name //localdrive could not be located</s:message>
</d:error>

[4] Expected (after manual intervention triggering a storage availability check) we start 503ing like we want

curl -s -u ncadmin:ncadminpass "http://nc-test.local:3680/remote.php/dav/files/ncadmi
n/localdrive/helloworld.txt" -X PROPFIND | xml_pp
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\ServiceUnavailable</s:exception>
  <s:message>Storage with mount id 1 is not available</s:message>
</d:error>

[basos9]

Hello,
I since upgraded to 25.0.6 but I reproduced the issue as well.

[joshtrichards]

Fixed by #39707. Similar to #39706.