Infinite recursion in _load_metadata_for_topics is problematic when metadata is unstable in Kafka #68
Description
I found out during testing that the constructor makes a never ending recursive call with _load_metadata_for_topics. This has caused me a bit of trouble when a consumer was started while a topic had offline partitions for a long period.
The code assumes a situation like this will only be temporary and thus sleeps and retries forever.
A consumer is affected by an offline partitions for a topic even if it isn't even interested in reading from it. i.e. topic xyz had no master (broker down) and consumer was only expected to read from topic abc. Since the initial _load_metadata_for_topics is loading metadata for all topics, it affected even though it only needs the metadata from the one of the topic.
My suggestion (pull request will be coming) would be to simply avoid recursion altogether and rely on the fact the metadata will be loaded on the fly (lazy) in _get_leader_for_partition if a request is made for a specific topic which was available at first. If it's still unavailable, the request will fail, as it should.