Database error: Corruption: block checksum mismatch

[nprt]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

A node unexpectedly failed with what seems like a corrupt database. Here are the logs at the time of the occurrence:

2024-12-05 22:54:11.745 2024-12-05 21:54:11.731 INFO tokio-runtime-worker substrate: 💤 Idle (379 peers), best: #23714680 (0xcd1a…e524), finalized #23714677 (0x4076…8237), ⬇ 1.5MiB/s ⬆ 3.1MiB/s
2024-12-05 22:54:12.395 2024-12-05 21:54:12.347 INFO tokio-runtime-worker substrate: 🏆 Imported #23714681 (0xcd1a…e524 → 0x1e30…6bdb)
2024-12-05 22:54:16.752 2024-12-05 21:54:16.732 INFO tokio-runtime-worker substrate: 💤 Idle (380 peers), best: #23714681 (0x1e30…6bdb), finalized #23714678 (0x325d…956f), ⬇ 2.0MiB/s ⬆ 3.4MiB/s
2024-12-05 22:54:18.405 2024-12-05 21:54:18.379 INFO tokio-runtime-worker substrate: 🏆 Imported #23714682 (0x1e30…6bdb → 0xcb4e…d339)
2024-12-05 22:54:18.522 ts=2024-12-05T21:54:18.522Z caller=collector.go:169 level=error msg="collector failed" name=arp duration_seconds=9.0534e-05 err="could not get ARP entries: rtnetlink NeighMessage has a wrong attribute data length"
2024-12-05 22:54:20.712 2024-12-05 21:54:20.712 ERROR tokio-runtime-worker grandpa: GRANDPA voter error: could not complete a round on disk: Database error: Corruption: block checksum mismatch: stored = 752577466, computed = 2596249044, type = 4 in /opt/chain-data/chains/polkadot/db/full/628201.sst offset 32019554 size 101163
2024-12-05 22:54:20.713 2024-12-05 21:54:20.713 ERROR tokio-runtime-worker sc_service::task_manager: Essential task grandpa-voter failed. Shutting down service.
Error:
0: Other: Essential task failed.
Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.

Node details:

2024-12-05 22:55:50.966 2024-12-05 21:55:50.915 INFO main sc_cli::runner: 💾 Database: RocksDb at /opt/chain-data/chains/polkadot/db/full
2024-12-05 22:55:50.966 2024-12-05 21:55:50.915 INFO main sc_cli::runner: 👤 Role: FULL
2024-12-05 22:55:50.966 2024-12-05 21:55:50.915 INFO main sc_cli::runner: 🏷 Node name: REDACTED
2024-12-05 22:55:50.966 2024-12-05 21:55:50.915 INFO main sc_cli::runner: 📋 Chain specification: Polkadot
2024-12-05 22:55:50.966 2024-12-05 21:55:50.915 INFO main sc_cli::runner: ❤️ by Parity Technologies [email protected], 2017-2024
2024-12-05 22:55:50.966 2024-12-05 21:55:50.915 INFO main sc_cli::runner: ✌️ version 1.16.2-dba2dd59101
2024-12-05 22:55:50.966 2024-12-05 21:55:50.915 INFO main sc_cli::runner: Parity Polkadot

Steps to reproduce

No response

[bkchr]

Looks like a disk corruption.