state sync failures observed with atomic root mismatch

[darioush]

Describe the bug
We have received 2 instances of reports with logs such as:

FATAL <C Chain> handler/handler.go:335 shutting down chain {"reason": "received an unexpected error", "error": "failed to notify VM that bootstrapping has started: synced root (0x35ae5c47aeab8d06a088b70034833af6414dfdb228f1f6e2a3cf65e94f4f04e5) does not match expected (0x48d9b80aa254dad0cbd11c36414eb36af46a595bbad975fefe5462a70eea9bd6) for atomic trie while processing async message:

To Reproduce
Unknown.

Expected behavior
No error during state sync.

[michaelbu-zerohash]

Attached C.log
clogs.tar.gz

[michaelbu-zerohash]

C-2023-07-09T01-46-58.180.log
C-2023-07-09T09-42-37.460.log
C-2023-07-09T17-32-05.627.log

[michaelbu-zerohash]

C-2023-07-07T14-56-09.843.log
C-2023-07-07T23-30-45.430.log
C-2023-07-08T08-10-21.166.log
C-2023-07-08T16-45-23.689.log

[ceyonur]

@darioush Could this be already fixed?

[darioush]

I don't think we found any reason that this would be fixed. Although I haven't seen any reproduction for it either.