rocksdb may lose data after shutting down the machine/container directly

[qizhou]

Default rocksdb options fsync is disabled, which means that the data are not written to disk (in os memory) after db put operation. This is acceptable for performance optimization. However, this also implies that if a master/slave's machine/container is shutdown, and the cluster crashes, restarting the cluster may lead to an inconsistent state, which should be detected and recovered.

import rocksdb
a = rocksdb.Options()
a.use_fsync
False

[qizhou]

@ninjaahhh This may relate to the inconsistent state you found.

[ninjaahhh]

but from this updated comment, use_fsync doesn't seem like a fix?

note in our production env, kernel version is 4.4.0 and file system is ext4

[qizhou]

Interesting. Oh, btw, a lot of system may not implement fsync and fdatasync well, especially in container env (I remember vmware vm may ignore fsync and fdatasync for performance reason).

[qizhou]

close it at the moment. Will re-open it if we discover more clue from rocksdb