backintime gets slower with new incremental backups

[KUGA2]

Should it not get faster?

myuser@homeserver:~$ time sudo -i backintime --config /home/myuser/.backintime backup

Back In Time
Version: 1.1.12

Back In Time comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type `backintime --license' for details.

INFO: Lock
INFO: Take a new snapshot. Profile: 1 Main profile
INFO: Call rsync to take the snapshot
INFO: [qt4systrayicon] begin loop
INFO: Save config file
INFO: Save permissions
INFO: Create info file
INFO: Keep min free disk space: 51200 MiB
INFO: Keep min 2% free inodes
INFO: Unlock

real	61m28,234s
user	26m34,136s
sys	12m38,644s
myuser@homeserver:~$ INFO: [qt4systrayicon] end loop

myuser@homeserver:~$ time sudo -i backintime --config /home/myuser/.backintime backup
[sudo] password for myuser: 

Back In Time
Version: 1.1.12

Back In Time comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type `backintime --license' for details.

INFO: Lock
INFO: Take a new snapshot. Profile: 1 Main profile
INFO: Compare with old snapshot: 20210210-084908-645
INFO: [qt4systrayicon] begin loop
INFO: Nothing changed, no back needed
WARNING: No new snapshot
INFO: Keep min free disk space: 51200 MiB
INFO: Keep min 2% free inodes
INFO: Unlock

real	11m43,800s
user	0m53,257s
sys	1m42,319s
myuser@homeserver:~$ INFO: [qt4systrayicon] end loop

myuser@homeserver:~$ time sudo -i backintime --config /home/myuser/.backintime backup

Back In Time
Version: 1.1.12

Back In Time comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type `backintime --license' for details.

INFO: Lock
INFO: Take a new snapshot. Profile: 1 Main profile
INFO: Compare with old snapshot: 20210210-124522-910
INFO: [qt4systrayicon] begin loop
INFO: Create hard-links

INFO: Call rsync to take the snapshot

INFO: Save config file
INFO: Save permissions

INFO: Create info file

INFO: Keep min free disk space: 51200 MiB
INFO: Keep min 2% free inodes
INFO: Unlock

real	95m5,678s
user	6m17,439s
sys	10m43,666s
myuser@homeserver:~$ time sudo -i backintime --config /home/myuser/.backintime backup

Back In Time
Version: 1.1.12

Back In Time comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type `backintime --license' for details.

INFO: Lock
INFO: Take a new snapshot. Profile: 1 Main profile
INFO: Compare with old snapshot: 20210210-142120-416
INFO: [qt4systrayicon] begin loop

INFO: Create hard-links

INFO: Call rsync to take the snapshot

INFO: Save config file
INFO: Save permissions

INFO: Create info file

INFO: Keep min free disk space: 51200 MiB
INFO: Keep min 2% free inodes
INFO: Unlock

real	87m9,941s
user	6m7,641s
sys	10m29,359s

myuser@homeserver:~$ time sudo du -h --max-depth=1 /media/backup/backintime/homeserver/root/1/
132G	/media/backup/backintime/homeserver/root/1/20210210-142120-416
2,1G	/media/backup/backintime/homeserver/root/1/20210210-161749-696
1,9G	/media/backup/backintime/homeserver/root/1/20210210-084908-645
1,5G	/media/backup/backintime/homeserver/root/1/20210210-124522-910
138G	/media/backup/backintime/homeserver/root/1/

real	55m36,073s
user	0m44,389s
sys	4m0,430s

[emtiu]

Regardless of whether there is a lot of data to copy, rsync has to check every existing file against its backup copy. My guess is that what you're seeing comes from an extreme number of small files, a slow source or destination drive, or a combination of all of those.

Is this problem still relevant to you?

[KUGA2]

No longer relevant to me. But maybe others...

[emtiu]

Closing, since the problem can no longer be diagnosed (old version, no longer in use by reporter).

[nepfaff]

I'm having the same problem with version 1.3.3-3.
My initial backup was around 100 MB/s, and now I'm getting speeds of 11 kB/s. It is basically making zero progress. I'm tempted to just delete the old snapshot to allow me to create a new one. I do have a ton of small files. Is there anything that I could do to speed this up?

[buhtz]

Dear nepfaff,
thanks for reporting.

• Where does your 1.3.3-3 comes from; debian, Ubuntu PPA, Arch, or somewhere else?
• What was your previous version?
• Can you please post the output of backintime --diagnostics.
• Can you give us a round about number of how many files there are and their size?
• If possible and you know how: Can you try to roll back to 1.3.2 (e.g. from our upstream repo) and tell us if the problem is reproducible.

[nepfaff]

Hi @buhtz,

• I installed it via Ubuntu PPA.
• This is my first time using it (never used a different version). My problem is regarding the difference in backup speed between the first snapshot and future ones.
• My first backup had the following stats form the logs: sent 321.36G bytes received 26.77M bytes 72.33M bytes/sec total size is 324.27G
• I found 1236737 files (size varies heavily from multiple GB to kb).

backintime --diagnostics output:

{
    "backintime": {
        "name": "Back In Time",
        "version": "1.3.3-3",
        "latest-config-version": 6,
        "local-config-file": "/home/UsernameReplaced/.config/backintime/config",
        "local-config-file-found": true,
        "global-config-file": "/etc/backintime/config",
        "global-config-file-found": false,
        "started-from": "/usr/share/backintime/common",
        "running-as-root": false,
        "user-callback": "/home/UsernameReplaced/.config/backintime/user-callback",
        "keyring-supported": true
    },
    "host-setup": {
        "platform": "Linux-5.14.0-1058-oem-x86_64-with-glibc2.29",
        "system": "Linux #66-Ubuntu SMP Fri Feb 10 09:46:18 UTC 2023",
        "os-release": "Ubuntu 20.04.5 LTS",
        "display-system": "x11",
        "locale": "en_US, UTF-8",
        "PATH": "/opt/gurobi952/linux64//bin:/home/UsernameReplaced/anaconda3/condabin:/usr/local/cuda-11.8/bin:/opt/drake/bin:/home/UsernameReplaced/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/UsernameReplaced/.local/bin",
        "RSYNC_OLD_ARGS": "(not set)",
        "RSYNC_PROTECT_ARGS": "(not set)"
    },
    "python-setup": {
        "python": "3.8.10 default Nov 14 2022 12:59:47 CPython GCC 9.4.0",
        "python-executable": "/usr/bin/python3",
        "python-executable-symlink": true,
        "python-executable-resolved": "/usr/bin/python3.8",
        "sys.path": [
            "/usr/share/backintime/qt/plugins",
            "/usr/share/backintime/common/plugins",
            "/usr/share/backintime/plugins",
            "/usr/share/backintime/common",
            "/usr/lib/python38.zip",
            "/usr/lib/python3.8",
            "/usr/lib/python3.8/lib-dynload",
            "/usr/local/lib/python3.8/dist-packages",
            "/usr/lib/python3/dist-packages"
        ],
        "qt": "PyQt 5.14.1 / Qt 5.12.8"
    },
    "external-programs": {
        "rsync": "3.1.3",
        "ssh": "OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020",
        "sshfs": "2.10.0",
        "encfs": "1.9.5",
        "shell": "/bin/bash",
        "shell-version": "GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)"
    }
}

[buhtz]

Thanks for your feedback.

• This is my first time using it (never used a different version). My problem is regarding the difference in backup speed between the first snapshot and future ones.
• My first backup had the following stats form the logs: sent 321.36G bytes received 26.77M bytes 72.33M bytes/sec total size is 324.27G
• I found 1236737 files (size varies heavily from multiple GB to kb).

With 1.2 Mio files in 320 GB I would say it is a semi-big backup. 😄
Hard to say from my side if this is a problem or usual behavior.

How long does such a backup take in minutes and hours? How long was the initial backup?

Can you tell me something more about your snapshot profile?

• Which kind is it: local, SSH (encrytped yes/no)?
• Do you run it as root?
• What is the source of the backup? Filesystem?
• Where is the destination of the backup? Is it mounted somehow (NFS, Samba, ...)?

Do you have an idea how many of the files are modified between a backup?

[nepfaff]

The initial backup was 1-2h. The repeated backup has been running for over 4h (I never had the time to let it complete so far).

• local
• running as root
• source is ubuntu root (/)
• destination is an external drive (WD Elements)

Since last time, it should be <2000 files modified (most of them should be small text or image files, some large binary files).

[buhtz]

The initial backup was 1-2h.

Wow this is "slow" and points to kind of a slow connection between source and destination.

The repeated backup has been running for over 4h (I never had the time to let it complete so far).

Taking the initial duration into account I don't wonder.

• source is ubuntu root (/)

You backup the whole file system? It might be possible but maybe you are better with a system image software. You should think about your backup strategy.

• destination is an external drive (WD Elements)

USB? Here you have the point. Does its filesystem support hard links?

I'm not to much into rsync and can not explain all details. But keep in mind that rsync need to "read" the files from the initial backup and compare them with the latest version of the files in the source location to decide if they need a backup or just a hard link. Of course rsync doesn't read the whole file but timestamps and things like that. I'm not 100% sure about the details. But there is read activity on an USB device.

How you describe your situation it doesn't wonder me that the backups take their time.

As a test and if you have a enough space left please try the same hard drive as source and destination.

[aryoda]

The first backup may be faster than later ones in case of many files since the first backup does not need to compare the files of an existing snapshot (backup) but can copy the files directly.

This depends on the rsync settings to compare the files (please start the backup with the additional --debug argument and append the output so to give us an indication of the used rsync switches...

Furthermore:

If your backup source is / and the backup target (usb drive?) is mounted under / I am wondering if this causes a "backup endless loop" in rsync (I have never tested this)... At least I think it could try to backup the existing snapshots of the backup target too ;-)

Possible solution: Exclude the backup target folder from the backup in your BiT backup profile or even better do not backup / but explicitly only the sub folder you really need (BiT is not meant to backup your whole OS/system, e.g. due to locked files or missing permissions - this is the better done with imaging software).

[aryoda]

@buhtz Shall we re-open this issue or open a new issue? Continuing discussions on an already closed issue makes it different to find it again.

[nepfaff]

Thank you for your suggestions!
I already excluded the drive to prevent a loop. I also played with most of the exposed rsync settings and it didn't help.
However, I now tried Timeshift (an rsync wrapper designed for root backups) and it works perfectly (repeated backups finish in a few minutes). Maybe there are some rsync settings that are more suitable for root backups but I didn't spend the time comparing the two tools.

[buhtz]

@buhtz Shall we re-open this issue or open a new issue? Continuing discussions on an already closed issue makes it different to find it again.

It is IMHO to foggy to make an issue out of it. Lets keep it closed.

[aryoda]

@nepfaff

However, I now tried Timeshift (an rsync wrapper designed for root backups) and it works perfectly (repeated backups finish in a few minutes).
Maybe there are some rsync settings that are more suitable for root backups but I didn't spend the time comparing the two tools.

Good to know that there is a way make it work fast enough.

It would be great if you could invest some time to help us (and our BiT users) to find out the reason for the slow-down by publishing the (anonymized) command line of the rsync call created by timeshift for the repeated backup (even though you are using timeshift now).

It may be just a small switch/option that may e.g. cause a full file compare...

Edit: I tried timeshift on my computer and I could find out the rsync cmd line with sudo timeshift --create --scripted --verbose --debug:

rsync -aii --recursive --verbose --delete --force --stats --sparse --delete-excluded --link-dest='/run/timeshift/backup/timeshift/snapshots/2023-03-17_23-37-12/localhost/' --log-file='/run/timeshift/backup/timeshift/snapshots/2023-03-17_23-52-31/rsync-log' --exclude-from='/run/timeshift/backup/timeshift/snapshots/2023-03-17_23-52-31/exclude.list' --delete-excluded '/' '/run/timeshift/backup/timeshift/snapshots/2023-03-17_23-52-31/localhost/'

It is indeed quite fast but maybe also because it excludes some folder like your complete home folder:

/dev/*
/proc/*
/sys/*
/media/*
/mnt/*
/tmp/*
/run/*
/var/run/*
/var/lock/*
/var/lib/docker/*
/var/lib/schroot/*
/lost+found
/timeshift/*
/timeshift-btrfs/*
/data/*
/DATA/*
/cdrom/*
/sdcard/*
/system/*
/etc/timeshift.json
/var/log/timeshift/*
/var/log/timeshift-btrfs/*
/swapfile
/snap/*
/root/.thumbnails
/root/.cache
/root/.dbus
/root/.gvfs
/root/.local/share/[Tt]rash
/home/*/.thumbnails
/home/*/.cache
/home/*/.dbus
/home/*/.gvfs
/home/*/.local/share/[Tt]rash
/root/.mozilla/firefox/*.default/Cache
/root/.mozilla/firefox/*.default/OfflineCache
/root/.opera/cache
/root/.kde/share/apps/kio_http/cache
/root/.kde/share/cache/http
/home/*/.mozilla/firefox/*.default/Cache
/home/*/.mozilla/firefox/*.default/OfflineCache
/home/*/.opera/cache
/home/*/.kde/share/apps/kio_http/cache
/home/*/.kde/share/cache/http
/var/cache/apt/archives/*
/var/cache/pacman/pkg/*
/var/cache/yum/*
/var/cache/dnf/*
/var/cache/eopkg/*
/var/cache/xbps/*
/var/cache/zypp/*
/var/cache/edb/*
/var/lib/libvirt/**
/home/aryoda/**
/root/**
/home/*/**

[aryoda]

BTW: Looks like the author gave up timeshift but at least a fork seems to be maintained by a Linux Mint team:

teejee2008/timeshift#913