Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ncp-backup-auto fails: tar --exclude does not seem to work (nextcloud.log has changed while processing) #1791

Closed
CarlosCastaneda opened this issue Jun 29, 2023 · 2 comments
Closed

ncp-backup-auto fails: tar --exclude does not seem to work (nextcloud.log has changed while processing) #1791

CarlosCastaneda opened this issue Jun 29, 2023 · 2 comments
Labels
has-updates

Comments

@CarlosCastaneda
Copy link

CarlosCastaneda commented Jun 29, 2023 •
edited
Loading

ncp-backup-auto runs /usr/local/bin/ncp-backup with appropriate parameters which calls tar to backup data.
Despite the fact that tar is called with --exclude "$data/{access,error,nextcloud}.log" I can find nextcloud.log within the backup.
If nextcloud.log has changed during backup tar returns with an error (ncp.log)

Maintenance mode already enabled
backup database...
backup files...
tar: data/nextcloud.log: file changed as we read it
error generating backup

and the commands following tar

rm "$dbbackup"
chmod 640 "$destfile"
chown :www-data "$destfile"

echo "backup $destfile generated"

are not executed resulting in incorrect permissions.
As far as I can see the backup itself is complete. I think that the exclude parameter mentioned above is not expanded as expected.
I could modify the code, e.g.

--exclude "$data/access.log" \
--exclude "$data/error.log" \
--exclude "$data/nextcloud.log" \

but as soon as there will be an update those modification will be overwritten.

NextcloudPi diagnostics 

Cannot load Zend OPcache - it was already loaded
NextcloudPi version  v1.52.0
NextcloudPi image    NextCloudPi_RaspberryPi_v1.51.0.img
OS                   Debian GNU/Linux 11. 6.1.21-v8+ (aarch64)
automount            yes
USB devices          sda sdb
datadir              /media/myCloudDrive1/ncdata/data
data in SD           no
data filesystem      btrfs
data disk usage      53G/1.9T
rootfs usage         6.5G/31G
swapfile             /var/swap
dbdir                /var/lib/mysql
Nextcloud check      ok
Nextcloud version    26.0.3.2
HTTPD service        up
PHP service          up
MariaDB service      up
Redis service        up
HPB service          up
Postfix service      up
Internet check       no
Public IPv4          ***REMOVED SENSITIVE VALUE***
Public IPv6          ***REMOVED SENSITIVE VALUE***
Port 80              [N/A]
Port 443             [N/A]
IP                   ***REMOVED SENSITIVE VALUE***
Gateway              ***REMOVED SENSITIVE VALUE***
Interface            eth0
Certificates         ***REMOVED SENSITIVE VALUE***
NAT loopback         no
Uptime               5days
Nextcloud configuration 

Cannot load Zend OPcache - it was already loaded
{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": {
            "0": "localhost",
            "3": "nextcloudpi",
            "1": "192.168.10.33",
            "2": "nextcloudpi.31fqey8qafjrkyf6.myfritz.net",
            "11": "2a01:c22:c013:b800:dea6:32ff:fe5b:e2de",
            "14": "nextcloudpi"
        },
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "26.0.3.2",
        "overwrite.cli.url": "https:\/\/nextcloudpi\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "tempdirectory": "\/media\/myCloudDrive1\/ncdata\/data\/tmp",
        "mail_smtpmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "overwriteprotocol": "https",
        "debug": false,
        "loglevel": 0,
        "loglevel_frontend": 2,
        "log_type": "file",
        "log_query": false,
        "default_phone_region": "de",
        "maintenance": false,
        "theme": "",
        "logfile": "\/media\/myCloudDrive1\/ncdata\/data\/nextcloud.log",
        "mail_sendmailmode": "smtp",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpsecure": "ssl",
        "app_install_overwrite": [
            "files_texteditor",
            "gallery"
        ],
        "data-fingerprint": "85debc01f804ff063236f876f2563a0d",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [
            "2FA"
        ],
        "twofactor_enforced_excluded_groups": [],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***"
    }
}
HTTPd logs 

[Thu Jun 29 00:00:05.943516 2023] [ssl:warn] [pid 1076:tid 547612738624] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Thu Jun 29 00:00:05.951196 2023] [mpm_event:notice] [pid 1076:tid 547612738624] AH00489: Apache/2.4.56 (Debian) OpenSSL/1.1.1n configured -- resuming normal operations
[Thu Jun 29 00:00:05.951250 2023] [core:notice] [pid 1076:tid 547612738624] AH00094: Command line: '/usr/sbin/apache2'
Database logs 

tail: cannot open '/var/log/mysql/*.log' for reading: No such file or directory
Nextcloud logs 

{"reqId":"TG3Y34MIfw7rYYDHx3Gv","level":0,"time":"2023-06-29T11:55:29+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Json","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"TG3Y34MIfw7rYYDHx3Gv","level":0,"time":"2023-06-29T11:55:29+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Atom","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"TG3Y34MIfw7rYYDHx3Gv","level":0,"time":"2023-06-29T11:55:29+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Rss","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"TG3Y34MIfw7rYYDHx3Gv","level":0,"time":"2023-06-29T11:55:29+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Rdf","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"TG3Y34MIfw7rYYDHx3Gv","level":0,"time":"2023-06-29T11:55:29+00:00","remoteAddr":"","user":"--","app":"serverDI","method":"","url":"--","message":"The requested alias \"SystemConfig\" is deprecated. Please request \"OC\\SystemConfig\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"--","version":"26.0.3.2","data":{"app":"serverDI"}}
{"reqId":"UBWuGZ6c1uynI4DhZXq9","level":0,"time":"2023-06-29T11:55:32+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Json","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"UBWuGZ6c1uynI4DhZXq9","level":0,"time":"2023-06-29T11:55:32+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Atom","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"UBWuGZ6c1uynI4DhZXq9","level":0,"time":"2023-06-29T11:55:32+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Rss","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"UBWuGZ6c1uynI4DhZXq9","level":0,"time":"2023-06-29T11:55:32+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Rdf","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"UBWuGZ6c1uynI4DhZXq9","level":0,"time":"2023-06-29T11:55:32+00:00","remoteAddr":"","user":"--","app":"serverDI","method":"","url":"--","message":"The requested alias \"SystemConfig\" is deprecated. Please request \"OC\\SystemConfig\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"--","version":"26.0.3.2","data":{"app":"serverDI"}}
{"reqId":"SmXz01NvHeU1dbMG6gsv","level":0,"time":"2023-06-29T11:55:33+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Json","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"SmXz01NvHeU1dbMG6gsv","level":0,"time":"2023-06-29T11:55:33+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Atom","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"SmXz01NvHeU1dbMG6gsv","level":0,"time":"2023-06-29T11:55:33+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Rss","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"SmXz01NvHeU1dbMG6gsv","level":0,"time":"2023-06-29T11:55:33+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Rdf","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"SmXz01NvHeU1dbMG6gsv","level":0,"time":"2023-06-29T11:55:33+00:00","remoteAddr":"","user":"--","app":"serverDI","method":"","url":"--","message":"The requested alias \"SystemConfig\" is deprecated. Please request \"OC\\SystemConfig\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"--","version":"26.0.3.2","data":{"app":"serverDI"}}
{"reqId":"uL74TC9LscrdfZXPd6Bm","level":0,"time":"2023-06-29T11:56:26+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Json","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"uL74TC9LscrdfZXPd6Bm","level":0,"time":"2023-06-29T11:56:26+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Atom","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"uL74TC9LscrdfZXPd6Bm","level":0,"time":"2023-06-29T11:56:26+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Rss","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"uL74TC9LscrdfZXPd6Bm","level":0,"time":"2023-06-29T11:56:26+00:00","remoteAddr":"","user":"--","app":"news","method":"","url":"--","message":"new parser added : FeedIo\\Standard\\Rdf","userAgent":"--","version":"26.0.3.2","data":{"app":"news"}}
{"reqId":"uL74TC9LscrdfZXPd6Bm","level":0,"time":"2023-06-29T11:56:26+00:00","remoteAddr":"","user":"--","app":"serverDI","method":"","url":"--","message":"The requested alias \"SystemConfig\" is deprecated. Please request \"OC\\SystemConfig\" directly. This alias will be removed in a future Nextcloud version.","userAgent":"--","version":"26.0.3.2","data":{"app":"serverDI"}}
@CarlosCastaneda CarlosCastaneda changed the title ncp-backup-auto fails because nextcloud.log has changed while processing ncp-backup-auto fails: tar --exclude does not seem to work (nextcloud.log has changed while processing) Jul 3, 2023
@CarlosCastaneda
Copy link
Author

These two lines in /usr/local/bin/ncp-backup have to be changed in order to work

--exclude "$data/{access,error,nextcloud}.log" \ => --exclude="$data"/{access,error,nextcloud}.log \ 

--exclude "nextcloud/data/{access,error,nextcloud}.log" \ => --exclude=nextcloud/data/{access,error,nextcloud}.log \

Here is a small repro:

mkdir -p /tmp/tar_backup/data
touch /tmp/tar_backup/data/{access,error,nextcloud}.log
touch /tmp/tar_backup/data/{text1,text2}.txt
data="data"

Exclude has no effect, all files are backuped

tar cvzf /tmp/tar_backup.tgz --exclude "$data/{access,error,nextcloud}.log" -C /tmp/tar_backup data
data/
data/error.log
data/access.log
data/nextcloud.log
data/text1.txt
data/text2.txt

An error occurs due to file expansion, which results in --exclude data/access.log data/error.log data/nextcloud.log and having access.log excluded only

tar cvzf /tmp/tar_backup.tgz --exclude "$data"/{access,error,nextcloud}.log -C /tmp/tar_backup data
tar: data/error.log: Cannot stat: No such file or directory
tar: data/nextcloud.log: Cannot stat: No such file or directory
data/
data/error.log
data/nextcloud.log
data/text1.txt
data/text2.txt
tar: Exiting with failure status due to previous errors

Works as expected --exclude="$data"/{access,error,nextcloud}.log

tar cvzf /tmp/tar_backup.tgz --exclude="$data"/{access,error,nextcloud}.log -C /tmp/tar_backup data
data/
data/text1.txt
data/text2.txt

theCalcaholic added a commit that referenced this issue Jul 17, 2023
@theCalcaholic
Merge pull request #1796 from nextcloud/devel
8c068cf 
Add support for Nextcloud 26.0.3

### Changes

- Adds support for NC 26.0.3
- Add ZFS support for the data directory (#1767)

### Fixes

- Fix bug in ncp-backup (#1791)
- Fix missing configuration field for nc-backup-auto (#1787)
@theCalcaholic
Copy link
Collaborator

Thank you for the extensive debugging! This should now be fixed in v1.52.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
has-updates
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@theCalcaholic @CarlosCastaneda