ethclient: SubscribeFilterLogs() not respecting FilterQuery.FromBlock

[mcdee]

System information

Geth
Version: 1.6.7-stable
Git Commit: ab5646c
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.8.1
Operating System: linux

Expected behaviour

Setting up a simple log filter for a given contract address starting at block 1000000:

client, _ := ethclient.Dial("/home/foo/.ethereum/geth.ipc")

filter := ethereum.FilterQuery{}
filter.Addresses = make([]common.Address, 0)
filter.Addresses = append(filter.Addresses, common.HexToAddress("0x..."))
filter.FromBlock = big.NewInt(1000000)

Using this with filterLogs() provides a set of results:

logs, _ := client.FilterLogs(ctx, filter)
fmt.Println(len(logs)) // Ouptuts '143'

However using SubscribeFilterLogs() does not provide any immediate results:

ctx := context.Background()
ch := make(chan types.Log)
client.SubscribeFilterLogs(ctx, filter, ch)

for true {
    log := <-ch
    fmt.Println("Matching log encountered")
}

It is expected that SubscribeFilterLogs() start from the block as specified in the FilterQuery's FromBlock.

Actual behaviour

SubscribeFilterLogs() appears to start from the first new block received by the client regardless of the block as specified in the FilterQuery's FromBlock.

Steps to reproduce the behaviour

The above code provides the scenario to test (pick your favourite contract address and start block for testing purposes).

[rjl493456442]

The behaviour is correct.

Reason:

client.filterLogs function invokes eth_getLogs function ultimately, which traverses the whole database to retrieve all matched logs.

client.SubscribeFilterLogs function creates a subscription with specific args. Once a new log arrives, if it matches the filter criteria，the log will been returned via subscription, if not, it will been discarded. It will not traverse the past log.

[mcdee]

This is not at all obvious from the documentation.

It also begs the question: how would one obtain all log entries in the correct order? FilterLogs() followed by SubscribeFilterLogs() could miss log entries in new blocks between the calls.

The only thing that I can think of doing would be to SubscribeFilterLogs(), then FilterLogs(), work through the log entries returned by FilterLogs(), then start working through SubscribeFilterLogs() but that's hardly intuitive and requires handling duplicates. Is there a better way?

[nkbai]

first call FilterLogs ,write result to the channel which is for SubscribeFilterLogs
for example:

logs, err := c.caller.FilterLogs(ctx, q)
	if err != nil {
		return nil, nil, err
	}
	var ch = make(chan types.Log, len(logs))
	//subcribe logs that will happen.
	sub, err := c.caller.SubscribeFilterLogs(ctx, q, ch)
	for _, log := range logs {
		ch <- *log
	}
	return ch, sub, err

[mcdee]

@nkbai what happens if a block comes in between the calls to FilterLogs and SubscribeFilterLogs?

[nkbai]

i don't know , you may lost this block. you can listen pending event SubscribePendingTransactions.

[nkbai]

you can specify the query's toblock argument to rpc.PendingBlockNumber. if FilterLogs doesn't need too much time, it will get all the related blocks.

[mcdee]

It doesn't sound like there is a working solution to this at current, which is a shame given that reading logs is a very common requirement.

[fjl]

This is a valid feature request, and we're aware of the limitation. It would also be nice to support ToBlock in SubscribeFilterLogs for streaming queries.

[yondonfu]

Hi has there been any movement regarding implementing this feature? Support for fromBlock in SubscribeFilterLogs would be especially helpful for clients that rely on logs to update a local DB based on state changes in a contract such that if the client stops and restarts at a later point in time, the client can start receiving logs starting from the last block that it was online.

[kminevskiy]

Hello, any updates on this feature request?

[qustavo]

I struggled with the same issue, perhaps documentation could provide a good example on how to handle this