Whisper cleanup, part 2

[karalabe]

Documentation and refinements for mostly the whole whisper codebase. Additionally:

• Fixed message dropping if no whisper identities were provided.
• Added known message expiration for peers, when the origin message expires.
• Wrote a few high level integration tests to catch breaking bugs.

[fjl]

As suggested previously, please don't use p2p.Server for protocol testing. You can wire protocols together like this instead:

rw1, rw2 := p2p.MsgPipe()
p1, p2 := p2p.NewPeer(...), p2p.NewPeer(...)
runWhisper(rw1, p2)
runWhisper(rw2, p1)

For testing a single protocol instance, you can use p2p.ExpectMsg:

rw1, rw2 := p2p.MsgPipe()
p := p2p.NewPeer(...)
runWhisper(rw1, p)

if err := p2p.Send(rw2, msgcode, ...); err != nil {
    t.Error(err)
}
if err := p2p.ExpectMsg(rw2, respcode, ...); err != nil {
    t.Error(err)
}

[karalabe]

K, will fix :)

[fjl]

Updated the comment above to mention p2p.ExpectMsg.

[fjl]

Please be aware that p2p.MsgPipe (just like net.Pipe and io.Pipe) has no buffer. This is supposed to be helpful when testing, but it also means that your tests will hang if you don't use it correctly.

[karalabe]

Hmmm, but in my integration tests, I was actually testing that the entire thing works as intended. They are not really wire protocol tests. p2p.Send and p2p.Expect would be useful as a lower level test, to verify the packets themselves. Those are the things I wanted to attend to after cleaning up the code.

Anyway, I am probably still missing a few things so I'll start digging into these constructs and get back with a proper reply after I know what I'm talking about :D

[fjl]

Style nit: I'd prefer DefaultTTL, DefaultPoW.

[karalabe]

Fixed in f6efdd8

[karalabe]

@fjl All traces of p2p.Server dumped, and switched over to manual wiring/mocking + added a few lower level protocol tests for whisper.peer-s directly. Whole tests significantly faster now, at around 3 secs for the full suite (cannot really reduce it beyond this, since whisper is based on cyclic messaging, so I do have to wait for the cycles to run).

Hope it's better this time :)

[fjl]

This comment is not necessary.

[karalabe]

I guess I get carried away sometimes :)), will fix.

[fjl]

As general feedback, this looks very good. I haven't looked at some of the changes to the core dispatch yet.

One of the general issues with Whisper is that it has not been proven whether the protocol actually works on a big network and what kind of topology it needs to work well. It might be an interesting exercise to test with bigger clusters and several topologies, now that we have the infrastructure for it. But that's for another time.

[karalabe]

My current issue with Whisper is that there isn't any (at least I haven't found) easy interface to live-test the thing. I can simulate some connections in the unit tests, or maybe even do some larger simulations by firing up some p2p.Servers, but it would be really nice if we could surface the whisper API into the console in some meaningful way.

I saw that there's a javascript implementation, but I got all kind of weird error messages with even a simple function call, and I've no clue yet whether I'm doing something wrong, the JavaScript implementation is buggy at the moment, or the interface between the two somewhere.

Do you by any chance have some code base that was based on Whisper and that could be run manually from a CLI without all kinds of fancy QML/Js/etc dependencies?

[karalabe]

Btw, I haven't really touched the core logic in whisper. I was mostly just polishing things up, and documenting/testing as I went by. During that I've found a few discrepancies with the spec, a few bugs and some missing but easily addressable things that I've fixed, but no major update yet. I wanted to have a solid base that's also well-ish tested before I start messing too deep with the code.

Now I have to figure out how to wire together standalone whisper processes so that I can test that it indeed works in a distributed environment too, as well as connect it to the C++ implementation to see if they actually speak the same protocol or not :)

[fjl]

I can simulate some connections in the unit tests, or maybe even do some larger simulations by firing up some p2p.Servers

The advantage of your current approach (using p2p.MsgPipe) is that you can try arbitrary network topologies without the overhead of creating many thousand TCP connections (each of them running encryption, multiplexing, etc).

Do you by any chance have some code base that was based on Whisper and that could be run manually from a CLI without all kinds of fancy QML/Js/etc dependencies?

No, but you can try the 'Whisper Chat' DApp in Mist. It's in the sidebar.

[obscuren]

This is a cool approach 👍

[karalabe]

@fjl PTAL