Skip to content

Conversation

@rjl493456442
Copy link
Member

@rjl493456442 rjl493456442 commented Sep 9, 2025

This pull request addresses the corrupted path database with log indicating:
history head truncation out of range, tail: 122557, head: 212208, target: 212557

This is a rare edge case where the in-memory layers, including the write buffer
in the disk layer, are fully persisted (e.g., written to file), but the state history
freezer is not properly closed (e.g., Geth is terminated after journaling but
before freezer.Close). In this situation, the recent state history writes will be
truncated on the next startup, while the in-memory layers resolve correctly.
As a result, the state history falls behind the disk layer (including the write buffer).

In this pull request, the state history freezer is always synced before journal,
ensuring the state history writes are always persisted before the others.

Edit:
It's confirmed that devops team has 10s container termination setting. It
explains why Geth didn't finish the entire termination without state history
being closed.

https://github.com/ethpandaops/fusaka-devnets/pull/63/files

@rjl493456442
Copy link
Member Author

rjl493456442 commented Sep 9, 2025

1756430477401	{"container_created_at":"2025-08-28T15:05:45.718597600Z","container_id":"5d115501ae696c265f180fb6bd9ae61b767da8e27ec498b223541cd37b2c36a3","container_name":"execution","host":"583fd1f204af","image":"ethpandaops/geth:master","label":{"buildnum":"","commit":"","ethpandaops.io.commitRef":"master","ethpandaops.io.repo":"ethereum/go-ethereum","version":""},"message":"INFO [08-29|01:21:17.400] Ethereum protocol stopped","source_type":"docker_logs","stream":"stderr"}
1756430477402	{"container_created_at":"2025-08-28T15:05:45.718597600Z","container_id":"5d115501ae696c265f180fb6bd9ae61b767da8e27ec498b223541cd37b2c36a3","container_name":"execution","host":"583fd1f204af","image":"ethpandaops/geth:master","label":{"buildnum":"","commit":"","ethpandaops.io.commitRef":"master","ethpandaops.io.repo":"ethereum/go-ethereum","version":""},"message":"INFO [08-29|01:21:17.402] Transaction pool stopped","source_type":"docker_logs","stream":"stderr"}
1756430477461	{"container_created_at":"2025-08-28T15:05:45.718597600Z","container_id":"5d115501ae696c265f180fb6bd9ae61b767da8e27ec498b223541cd37b2c36a3","container_name":"execution","host":"583fd1f204af","image":"ethpandaops/geth:master","label":{"buildnum":"","commit":"","ethpandaops.io.commitRef":"master","ethpandaops.io.repo":"ethereum/go-ethereum","version":""},"message":"INFO [08-29|01:21:17.460] Persisting dirty state                   head=212,684 root=73d573..ce710e layers=32684","source_type":"docker_logs","stream":"stderr"}
1756430485432	{"container_created_at":"2025-08-28T15:05:45.718597600Z","container_id":"5d115501ae696c265f180fb6bd9ae61b767da8e27ec498b223541cd37b2c36a3","container_name":"execution","host":"583fd1f204af","image":"ethpandaops/geth:master","label":{"buildnum":"","commit":"","ethpandaops.io.commitRef":"master","ethpandaops.io.repo":"ethereum/go-ethereum","version":""},"message":"INFO [08-29|01:21:25.432] Persisted dirty state to file            path=/data/geth/triedb/merkle.journal size=193.46MiB elapsed=7.971s","source_type":"docker_logs","stream":"stderr"}
1756430497086	{"container_created_at":"2025-08-29T01:21:36.220525333Z","container_id":"c818449655aba98e7e1df04ea7ffb395b7f61948c93de42f7da9066d8d1a159a","container_name":"execution","host":"583fd1f204af","image":"ethpandaops/geth:master","label":{"buildnum":"","commit":"","ethpandaops.io.commitRef":"master","ethpandaops.io.repo":"ethereum/go-ethereum","version":""},"message":"INFO [08-29|01:21:37.086] Maximum peer count                       ETH=50 total=50","source_type":"docker_logs","stream":"stderr"}
1756430497089	{"container_created_at":"2025-08-29T01:21:36.220525333Z","container_id":"c818449655aba98e7e1df04ea7ffb395b7f61948c93de42f7da9066d8d1a159a","container_name":"execution","host":"583fd1f204af","image":"ethpandaops/geth:master","label":{"buildnum":"","commit":"","ethpandaops.io.commitRef":"master","ethpandaops.io.repo":"ethereum/go-ethereum","version":""},"message":"INFO [08-29|01:21:37.089] Smartcard socket not found, disabling    err=\"stat /run/pcscd/pcscd.comm: no such file or directory\"","source_type":"docker_logs","stream":"stderr"}
1756430497097	{"container_created_at":"2025-08-29T01:21:36.220525333Z","container_id":"c818449655aba98e7e1df04ea7ffb395b7f61948c93de42f7da9066d8d1a159a","container_name":"execution","host":"583fd1f204af","image":"ethpandaops/geth:master","label":{"buildnum":"","commit":"","ethpandaops.io.commitRef":"master","ethpandaops.io.repo":"ethereum/go-ethereum","version":""},"message":"INFO [08-29|01:21:37.095] Set global gas cap                       cap=50,000,000","source_type":"docker_logs","stream":"stderr"}

The log with database corruption, provided by @barnabasbusa. The lacking of log Blockchain stopped indicates
it's possible the pathdb is journalled while state history is not closed.

The normal termination logs should be

^CINFO [09-08|22:32:19.672] Got interrupt, shutting down...
INFO [09-08|22:32:19.672] HTTP server stopped                      endpoint=127.0.0.1:5758
INFO [09-08|22:32:19.672] IPC endpoint closed                      url=/home/gary/mount/eth-hoodi/geth.ipc
ERROR[09-08|22:32:22.498] Beacon backfilling failed                err="syncing canceled (requested)"
INFO [09-08|22:32:22.498] Ethereum protocol stopped
INFO [09-08|22:32:22.506] Transaction pool stopped
INFO [09-08|22:32:22.525] Persisting dirty state                   head=799,570 root=a2879e..8ba337 layers=81
INFO [09-08|22:32:22.653] Persisted dirty state to file            path=/home/gary/mount/eth-hoodi/geth/triedb/merkle.journal size=35.22MiB elapsed=127.894ms
INFO [09-08|22:32:22.689] Blockchain stopped

@rjl493456442 rjl493456442 added this to the 1.16.4 milestone Sep 9, 2025
@fjl fjl merged commit ca6e2d1 into ethereum:master Sep 9, 2025
6 of 7 checks passed
Sahil-4555 pushed a commit to Sahil-4555/go-ethereum that referenced this pull request Oct 12, 2025
This pull request addresses the corrupted path database with log
indicating:
`history head truncation out of range, tail: 122557, head: 212208,
target: 212557`

This is a rare edge case where the in-memory layers, including the write
buffer
in the disk layer, are fully persisted (e.g., written to file), but the
state history
freezer is not properly closed (e.g., Geth is terminated after
journaling but
before freezer.Close). In this situation, the recent state history
writes will be
truncated on the next startup, while the in-memory layers resolve
correctly.
As a result, the state history falls behind the disk layer (including
the write buffer).

In this pull request, the state history freezer is always synced before
journal,
ensuring the state history writes are always persisted before the
others.

Edit: 
It's confirmed that devops team has 10s container termination setting.
It
explains why Geth didn't finish the entire termination without state
history
being closed.

https://github.com/ethpandaops/fusaka-devnets/pull/63/files
billettc pushed a commit to streamingfast/go-ethereum that referenced this pull request Nov 12, 2025
…hereum#718)

This pull request addresses the corrupted path database with log
indicating:
`history head truncation out of range, tail: 122557, head: 212208,
target: 212557`

This is a rare edge case where the in-memory layers, including the write
buffer
in the disk layer, are fully persisted (e.g., written to file), but the
state history
freezer is not properly closed (e.g., Geth is terminated after
journaling but
before freezer.Close). In this situation, the recent state history
writes will be
truncated on the next startup, while the in-memory layers resolve
correctly.
As a result, the state history falls behind the disk layer (including
the write buffer).

In this pull request, the state history freezer is always synced before
journal,
ensuring the state history writes are always persisted before the
others.

Edit: 
It's confirmed that devops team has 10s container termination setting.
It
explains why Geth didn't finish the entire termination without state
history
being closed.

https://github.com/ethpandaops/fusaka-devnets/pull/63/files

Co-authored-by: rjl493456442 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants