Skip to content

Conversation

@Jolly23
Copy link
Contributor

@Jolly23 Jolly23 commented Aug 30, 2025

Summary

Fixes long-standing ETA calculation errors in progress indicators that have been present since February 2021. The current implementation produces increasingly inaccurate estimates due to integer division precision loss.

Problem

// Occasionally report the indexing progress
if time.Since(logged) > time.Second*8 {
logged = time.Now()
var (
left = lastID - current + 1
done = current - beginID
speed = done/uint64(time.Since(start)/time.Millisecond+1) + 1 // +1s to avoid division by zero
)
// Override the ETA if larger than the largest until now
eta := time.Duration(left/speed) * time.Millisecond
log.Info("Indexing state history", "processed", done, "left", left, "elapsed", common.PrettyDuration(time.Since(start)), "eta", common.PrettyDuration(eta))
}

The ETA calculation has two critical issues:

  1. Integer division precision loss: speed is calculated as uint64
  2. Off-by-one: speed uses + 1(2 times) to avoid division by zero, however it makes mistake in the final calculation

This results in wildly inaccurate time estimates that don't improve as progress continues.

Example

Current output during state history indexing:

lvl=info msg="Indexing state history" processed=16858580 left=41802252 elapsed=18h22m59.848s eta=11h36m42.252s

Expected calculation:

  • Speed: 16858580 ÷ 66179848ms = 0.255 blocks/ms
  • ETA: 41802252 ÷ 0.255 = ~45.6 hours

Current buggy calculation:

  • Speed: rounds to 1 block/ms
  • ETA: 41802252 ÷ 1 = ~11.6 hours ❌

Solution

  • Created centralized CalculateETA() function in common package
  • Replaced all 8 duplicate code copies across the codebase

Testing

Verified accurate ETA calculations during archive node reindexing with significantly improved time estimates.

@Jolly23 Jolly23 requested a review from rjl493456442 as a code owner August 30, 2025 06:11
@rjl493456442 rjl493456442 self-assigned this Aug 30, 2025
@Jolly23
Copy link
Contributor Author

Jolly23 commented Aug 31, 2025

Seems like the appveyor check failure is not triggered by my changes

@rjl493456442 rjl493456442 changed the title fix: correct ETA calculation across all progress indicators all: improve ETA calculation across all progress indicators Sep 1, 2025
@rjl493456442 rjl493456442 merged commit 0e69530 into ethereum:master Sep 1, 2025
7 of 8 checks passed
@rjl493456442 rjl493456442 added this to the 1.16.3 milestone Sep 1, 2025
gballet pushed a commit to gballet/go-ethereum that referenced this pull request Sep 11, 2025
…#32521)

### Summary
Fixes long-standing ETA calculation errors in progress indicators that
have been present since February 2021. The current implementation
produces increasingly inaccurate estimates due to integer division
precision loss.

### Problem

https://github.com/ethereum/go-ethereum/blob/3aeccadd04aee2d18bdb77826f86b1ca000d3b67/triedb/pathdb/history_indexer.go#L541-L553
The ETA calculation has two critical issues:
1. **Integer division precision loss**: `speed` is calculated as
`uint64`
2. **Off-by-one**: `speed` uses `+ 1`(2 times) to avoid division by
zero, however it makes mistake in the final calculation

This results in wildly inaccurate time estimates that don't improve as
progress continues.

### Example
Current output during state history indexing:
```
lvl=info msg="Indexing state history" processed=16858580 left=41802252 elapsed=18h22m59.848s eta=11h36m42.252s
```

**Expected calculation:**
- Speed: 16858580 ÷ 66179848ms = 0.255 blocks/ms  
- ETA: 41802252 ÷ 0.255 = ~45.6 hours

**Current buggy calculation:**
- Speed: rounds to 1 block/ms
- ETA: 41802252 ÷ 1 = ~11.6 hours ❌

### Solution
- Created centralized `CalculateETA()` function in common package
- Replaced all 8 duplicate code copies across the codebase

### Testing
Verified accurate ETA calculations during archive node reindexing with
significantly improved time estimates.
Sahil-4555 pushed a commit to Sahil-4555/go-ethereum that referenced this pull request Oct 12, 2025
…#32521)

### Summary
Fixes long-standing ETA calculation errors in progress indicators that
have been present since February 2021. The current implementation
produces increasingly inaccurate estimates due to integer division
precision loss.

### Problem

https://github.com/ethereum/go-ethereum/blob/3aeccadd04aee2d18bdb77826f86b1ca000d3b67/triedb/pathdb/history_indexer.go#L541-L553
The ETA calculation has two critical issues:
1. **Integer division precision loss**: `speed` is calculated as
`uint64`
2. **Off-by-one**: `speed` uses `+ 1`(2 times) to avoid division by
zero, however it makes mistake in the final calculation

This results in wildly inaccurate time estimates that don't improve as
progress continues.

### Example
Current output during state history indexing:
```
lvl=info msg="Indexing state history" processed=16858580 left=41802252 elapsed=18h22m59.848s eta=11h36m42.252s
```

**Expected calculation:**
- Speed: 16858580 ÷ 66179848ms = 0.255 blocks/ms  
- ETA: 41802252 ÷ 0.255 = ~45.6 hours

**Current buggy calculation:**
- Speed: rounds to 1 block/ms
- ETA: 41802252 ÷ 1 = ~11.6 hours ❌

### Solution
- Created centralized `CalculateETA()` function in common package
- Replaced all 8 duplicate code copies across the codebase

### Testing
Verified accurate ETA calculations during archive node reindexing with
significantly improved time estimates.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants