RocksDB database corruption on power loss

[CNazar]

in the event of an emergency shutdown of the virtual disk on the home computer (when the light turns off) in most cases there is a drop in the base (an attachment error is added). I understand that running the node on a home computer does not meet the requirements of the data center where uninterruptible power supply, but for statistics I send a report
https://ibb.co/yFhF8gq

vmware 14.1.4
ubuntu 18,04 lts
2cpu
3 GB
Hdd 80 gb

[bkchr]

@arkpar Could we not run rocksdb in a much more fail safe way? That is slower, but probably survives such crashes?

[arkpar]

It should be crash tolerant with current setup I believe. @ordian @NikVolf Any ideas?

[ordian]

It might be the same as https://github.com/openethereum/openethereum/issues/11586, my hope is that it is related to facebook/rocksdb#6316 (although this one happens when no space left), which was fixed in rocksdb 6.8.0. We'll have to wait for a new version of librocksdb-sys published to test that claim. Also there has been some fixes related to manifest corruption and fail recovery in 6.11.1 and 6.11.3: https://github.com/facebook/rocksdb/releases.

[jacogr]

So I have seen this as recently as .15 with the fd issues. A crash due to this issue lead to corruption. (On a singular instance, was not consistent nor widespread)

Since we have addressed the file descriptor issue lately, I don’t think the specific cause is relevant anymore in the one I’ve seen.