cassandra:2.2 image won't start on an overlay network

[wfwheele]

I went through the following article to setup a docker swarm environment on my laptop:
https://medium.com/on-docker/docker-overlay-networks-that-was-easy-8f24baebb698#.8bbx8sc6h
I got through this fine and everything worked as expected.

Then made a new network with docker network create -d overlay cassNet
and then run docker run -d --name cass0 --net cassNet cassandra:2.2 and it exits immediately. Upon further inspect I found the following reason by running docker run -it --name cass1 --net cassNet cassandra:2.2

INFO  14:26:06 Loading settings from file:/etc/cassandra/cassandra.yaml
INFO  14:26:06 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.0.0.2 172.18.0.2; broadcast_rpc_address=10.0.0.2 172.18.0.2; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_user_defined_functions=false; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.0.0.2 172.18.0.2; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_validity_in_ms=2000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=10.0.0.2 172.18.0.2}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; windows_timer_interval=1; write_request_timeout_in_ms=2000]
INFO  14:26:06 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  14:26:06 Global memtable on-heap threshold is enabled at 122MB
INFO  14:26:06 Global memtable off-heap threshold is enabled at 122MB
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: Unknown listen_address '10.0.0.2 172.18.0.2'
Unknown listen_address '10.0.0.2 172.18.0.2'
ERROR 14:26:06 Exception encountered during startup: Unknown listen_address '10.0.0.2 172.18.0.2'

I've noticed that the 3.x images will run on an overlay network just fine.

[wfwheele]

So it looks like this can be worked around by making sure that the CASSANDRA_LISTEN_ADDRESS environment variable is set when running docker run. so if one ran docker network create -d overlay overlay-net then docker run -d --name some-cassandra --net overlay-net -e CASSANDRA_LISTEN_ADDRESS=some-cassandra cassandra:2.2 the container seems to run fine. Not sure if this is an "issue" really, but maybe an update to the documentation could help those running in this sort of environment?

[tianon]

Yeah, looks like this is caused by hostname --ip-address returning more than one result. 😞

Does this mean that 3.x handles multiple values for these IP address keys without problems? For 2.x, should we parse the hostname --ip-address output to scrape off just the first value, or simply document this quirk?

[tianon]

(If we handle this in some automatic fashion, how do we choose which IP address to use? Just use the first one without question?)

[tianon]

Example output, for reference:

root@540c71220fb3:/# hostname --ip-address
10.0.0.2 172.19.0.2

[kopachevsky]

Any workaround to that? I really don't want to create my own dockerfile to fix this issue