[Bug]: atuin does not sync correctly

[jankatins]

What did you expect to happen?

I wanted to setup a new laptop and expected that after atuin login, all old shell history from other computeres were available. It was not. Also, when I checked on other computers, they did not have the same amount of history, even after atuin sync.

What happened?

This is the session of an old computer running for days. As far as I can see, it shows that it didn't download history which was uploaded by another computer. I had to run atuin sync -f to get it to download additional history entries. I then ran it as well on a new computer (with ~40 entries) and it also started downloading entries there. Still, the new computer now shows 20425 entries (after about five atuin sync -f runs over 20min, similar increasing items like the below session), the old computer shows now 20728 (see latest run below):

[18:39:03] λ  atuin status
Atuin v18.3.0 - Build rev 9be49e434e549d6b13928b530139b66fef6171f1

[Local]
Sync frequency: 10m
Last sync: 2024-10-03 16:38:58.260977479 +00:00:00
History count: 10041
Deleted history count: 0

[Remote]
Address: https://api.atuin.sh
Username: jankatins

~
[18:39:09] λ  atuin sync
Sync complete! 10042 items in history database, force: false

~
[18:39:26] λ  atuin sync -f
Sync complete! 17615 items in history database, force: true

~ took 5s
[18:40:56] λ  atuin sync -f
Sync complete! 17926 items in history database, force: true

~ took 48s
[18:41:49] λ  atuin sync -f
Sync complete! 19328 items in history database, force: true

[18:45:52] λ  atuin sync -f
Sync complete! 20728 items in history database, force: true

I then checked for the history of the computer I decommissioned a few days ago (and have not available anymore!) and the latest data I have from that laptop is from "1mo ago" (I just checked for a command which run almost daily, no idea if there is a better way to search for by hostname?). So apparently, atuin did NOT sync for about a month? Thankfully I didn't do a lot of new stuff during the last month, so that's not the end of the world, but it is still scary that sync doesn't work correctly. :-(

Atuin doctor output

# Old computer
{
  "atuin": {
    "version": "18.3.0",
    "sync": {
      "cloud": true,
      "records": false,
      "auto_sync": true,
      "last_sync": "2024-10-03 16:42:01.426665248 +00:00:00"
    },
    "sqlite_version": "3.44.0"
  },
  "shell": {
    "name": "zsh",
    "default": "zsh",
    "plugins": [
      "atuin"
    ],
    "preexec": "built-in"
  },
  "system": {
    "os": "Debian GNU/Linux",
    "arch": "x86_64",
    "version": "unknown",
    "disks": [
      {
        "name": "/dev/mapper/bloodaxe--vg-root",
        "filesystem": "ext4"
      },
      {
        "name": "/dev/nvme0n1p2",
        "filesystem": "ext2"
      },
      {
        "name": "/dev/nvme0n1p1",
        "filesystem": "vfat"
      }
    ]
  }
}

Code of Conduct

• I agree to follow this project's Code of Conduct

[ellie]

This is because new installations are automatically migrated to sync v2, while the old are still on sync v1. So your old machine will still have synced OK, it's just not stored in a place that the new machine can find it.

Sync v1 had a whole bunch of issues which have been resolved in the new version. The next release will auto-migrate users, however making the process automatic has been tricky.

Sync v2 is enabled by the following

[sync]
records = true

Which you will find in your new machine's config. Disable it to download from the old store, and then enable it again to migrate to the new.

[mou]

Hi, @ellie. I think I faced the same issue. I have an old laptop that runs Atuin, which was installed by Nix over a year ago. Now, I have a new computer and want to sync history, but I was unable to. Your comment describes the issue I most likely face. So, what is the proper sequence of actions to make everything work, considering I have already tried to log in from the new computer and pushed some data to sync V2? Are there any migration guides for situations like this?

[ellie]

There's not much of a process. They just need to be on the same version of sync, in order to sync

If you enable

[sync]
records = true

On your old machine, it'll work. Run atuin sync after if you want to be extra sure.

(what I wrote previously is because OP no longer has access to the old machine, so needs to flip it off/on again)

considering I have already tried to log in from the new computer and pushed some data to sync V2? Are there any migration guides for situations like this?

Addressing this explicitly, there's no need to be concerned about conflicts/etc there 😊

[mou]

Thank you, Ellie. I will try to upgrade my old laptop with a newer version of Atuin because the version I'm using does not support Sync v2.

[jankatins]

Ok, it seems now the data is synced and after looking at the sqlite db itself, I see recent data from the decommissioned laptop. So no data was lost after all. Yay and sorry for the scare!

What this still cannot explain is why the data was not synced (down!) in one go but only across multiple atuin sync commands and not even that downloaded everything? E.g. I opened .local/share/atuin/history.db directly and if I understand the below SQL output right, it downloaded items even after I sent this issue: 21967 (now in the DB) - 20728 (items in DB 3 days ago, according to my initial report) =~1200 new items. But the number of items since a week ago when i decomissioned my old work laptop is ONLY about 900. So there are at minimum 300 items which were downloaded after multiple atuin sync -f runs.

I do share configs between all my computers, so all atuin instances ran with the old sync method, even the computer I just installed. I just switched the new computer to the new method and it uploaded all the items. Is it in general better to always use the new sync version?

SELECT count(*) from history;
-- 21967 entries

SELECT datetime(timestamp/1000000000 , 'unixepoch', 'localtime'), * from history
where hostname like '%old-hostname%' and "timestamp" > 1727165683559360000
order by timestamp desc;
-- Shows the last entry at 1727165683559360000, which is 2024-09-24 10:14:43 and looks about right

SELECT count(*) from history
where "timestamp" > 1727165683559360000;
-- Returns about 900 entries

UPDATE: I just turned it on on my private laptop as well and it has now more or less (+/- 1 entry for the latets sync run) the same number. So this feels much more reassuring that it has all the data. :-)