Race Between Initial Block Hashes can cause panic

[TheBlueMatt]

MaxFangX, on discord, reported:

Spent the last few hours pinning down the root cause of a channel manager panic. Turns out it was a race condition inherited from ldk-sample:

• (1) If a channel manager is starting for the first time, it gets its initial block hash by calling BlockSource::get_blockchain_info() and constructing a BestBlock out of that.
• (2) During the initial chain sync, if chain_tip.is_none() then the best chain_tip is retrieved using lightning_block_sync::init::validate_best_block_header() which similarly uses BlockSource::get_best_block()
  If blocks were mined between (1) and (2), the channel manager’s block hash will be at a lower height than that of the block hash contained inside the chain_tip fed to the SpvClient. Suppose that this results in the channel manager being at block 1 and the SpvClient being at block 2. Once block 3 comes in and the SpvClient detects it via poll_best_tip(), the SpvClient computes the ChainDifference between its saved chain_tip value and block 3, concluding that it just needs to connect block 3. However, when Listen::filtered_block_connected is called using the data in block 3, the channel manager panics with “Blocks must be connected in chain-order - the connected header must build on the last connected header” - because block 3’s prev block hash is block 2’s block hash, rather than block 1’s block hash which the channel manager expects.

It seems very unlikely to happen in practice but that also means it has the potential to be a superrrrr flaky bug for someone else in the future. Came up in our integration tests because we have separate "init" and "sync" stages, where "init" initializes the channel manager and "sync" calls into lightning_block_sync::init, and we were mining blocks in between.