5.0.0-rc3 WAX: Error on shutdown: corrupted size vs. prev_size

[matthewdarwin]

This issue still exists: corrupted size vs. prev_size. Has been seen in many previous versions. It is triggered on a normal clean shutdown. I don't know how often it happens because the state is written correctly and nodeos starts up. So it's just a log message, but nothing bad happens....

Dec 29 00:13:41 wax-snap71c nodeos[2575092]: info  2023-12-29T00:13:41.514 nodeos    http_plugin.cpp:515           plugin_shutdown      ] exit shutdown
Dec 29 00:13:41 wax-snap71c nodeos[2575092]: info  2023-12-29T00:13:41.515 nodeos    main.cpp:155                  operator()           ] nodeos version v5.0.0wax01-rc3 v5.0.0wax01-rc3-10c04d06485c8a1fdb52f6036dfa90b93813cb3a-dirty
Dec 29 00:13:41 wax-snap71c nodeos[2575092]: info  2023-12-29T00:13:41.515 nodeos    main.cpp:62                   log_non_default_opti ] Non-default options:  [.........]
Dec 29 00:13:41 wax-snap71c nodeos[2575092]: corrupted size vs. prev_size
Dec 29 00:13:45 wax-snap71c nodeos[2593285]: nodeos still running
Dec 29 00:13:46 wax-snap71c snapshot[2554292]: get_info failed
Dec 29 00:13:46 wax-snap71c systemd[1]: nodeos.service: Main process exited, code=killed, status=6/ABRT
Dec 29 00:13:49 wax-snap71c snapshot[2554292]: get_info failed
Dec 29 00:13:50 wax-snap71c nodeos[2593285]: nodeos stopped
Dec 29 00:13:51 wax-snap71c systemd[1]: nodeos.service: Failed with result 'signal'.
Dec 29 00:13:51 wax-snap71c systemd[1]: Stopped nodeos.service - EOS Nation Nodeos.
Dec 29 00:13:51 wax-snap71c systemd[1]: nodeos.service: Consumed 8h 6min 43.460s CPU time.

[matthewdarwin]

Today I got

nodeos[999210]: corrupted size vs. prev_size while consolidating

[spoonincode]

Can you provide the core dump for these crashes?

[matthewdarwin]

There is no core dump.. Core dumps are turned off in production systems.

[bhazzard]

Matthew says he thinks this issue has been seen since v2. Previous research found an issue when API calls were in progress while a node was shutting down.

Not a blocker for 5.0.0 stable.

[spoonincode]

Without core dumps provided it's hard to say for sure, but let's consider #2042 fixing this until knowing otherwise

[matthewdarwin]

If I see it again (after 5.0.1 release) we can do more debugging on it.