Ingester startup failed: JOINING -> JOINING is disallowed

[bboreham]

It seems Consul was having trouble for a bit, but the ingester should go back to the beginning and retry.
Similar to #2508

level=info ts=2020-04-30T08:56:13.148466874Z caller=server.go:147 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2020-04-30T08:56:13.223776466Z caller=main.go:119 msg="Starting Cortex" version="(version=1.0.0, branch=master, revision=a353bfac)"
level=info ts=2020-04-30T08:56:13.226347568Z caller=module_service.go:58 msg=initialising module=memberlist-kv
level=info ts=2020-04-30T08:56:13.227432787Z caller=module_service.go:58 msg=initialising module=runtime-config
level=info ts=2020-04-30T08:56:13.23660102Z caller=module_service.go:58 msg=initialising module=store
level=info ts=2020-04-30T08:56:13.236841065Z caller=module_service.go:58 msg=initialising module=ingester
level=info ts=2020-04-30T08:56:13.239179246Z caller=cortex.go:329 msg="Cortex started"
level=info ts=2020-04-30T08:56:13.239290396Z caller=lifecycler.go:483 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2020-04-30T08:56:13.252611744Z caller=lifecycler.go:507 msg="instance not found in ring, adding with no tokens" ring=ingester
level=info ts=2020-04-30T08:56:15.954326515Z caller=lifecycler.go:687 msg="changing instance state from" old_state=PENDING new_state=JOINING ring=ingester
level=error ts=2020-04-30T08:56:15.957579614Z caller=client.go:155 msg="error CASing" key=collectors/ring err="Unexpected response code: 500 (rpc error making call: stream closed)"
level=error ts=2020-04-30T08:56:15.979142491Z caller=client.go:155 msg="error CASing" key=collectors/ring err="Unexpected response code: 500 (rpc error getting client: failed to get conn: dial tcp <nil>->10.244.228.168:8300: getsockopt: connection refused)"
level=error ts=2020-04-30T08:56:15.984072334Z caller=client.go:155 msg="error CASing" key=collectors/ring err="Unexpected response code: 500 (rpc error getting client: failed to get conn: dial tcp <nil>->10.244.228.168:8300: getsockopt: connection refused)"
level=error ts=2020-04-30T08:56:15.993551293Z caller=client.go:155 msg="error CASing" key=collectors/ring err="Unexpected response code: 500 (rpc error getting client: failed to get conn: dial tcp <nil>->10.244.228.168:8300: getsockopt: connection refused)"
level=error ts=2020-04-30T08:56:16.002410843Z caller=client.go:155 msg="error CASing" key=collectors/ring err="Unexpected response code: 500 (rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection)"
level=error ts=2020-04-30T08:56:16.017335033Z caller=client.go:155 msg="error CASing" key=collectors/ring err="Unexpected response code: 500 (rpc error getting client: failed to get conn: dial tcp <nil>->10.244.228.168:8300: getsockopt: connection refused)"
level=error ts=2020-04-30T08:56:16.030388701Z caller=client.go:155 msg="error CASing" key=collectors/ring err="Unexpected response code: 500 (rpc error getting client: failed to get conn: dial tcp <nil>->10.244.228.168:8300: getsockopt: connection refused)"
level=error ts=2020-04-30T08:56:16.037535603Z caller=client.go:155 msg="error CASing" key=collectors/ring err="Unexpected response code: 500 (rpc error getting client: failed to get conn: dial tcp <nil>->10.244.228.168:8300: getsockopt: connection refused)"
level=error ts=2020-04-30T08:56:16.05936685Z caller=client.go:155 msg="error CASing" key=collectors/ring err="Unexpected response code: 500 (rpc error getting client: failed to get conn: dial tcp <nil>->10.244.228.168:8300: getsockopt: connection refused)"
level=error ts=2020-04-30T08:56:16.064021382Z caller=client.go:155 msg="error CASing" key=collectors/ring err="Unexpected response code: 500 (rpc error getting client: failed to get conn: dial tcp <nil>->10.244.228.168:8300: getsockopt: connection refused)"
level=warn ts=2020-04-30T08:56:16.064121929Z caller=grpc_logging.go:55 method=/cortex.Ingester/TransferChunks duration=109.912138ms err="failed to CAS collectors/ring" msg="gRPC\n"
level=error ts=2020-04-30T08:56:28.286815376Z caller=client.go:155 msg="error CASing" key=collectors/ring err="Unexpected response code: 500 (rpc error getting client: failed to get conn: dial tcp <nil>->10.244.228.168:8300: i/o timeout)"
level=info ts=2020-04-30T08:56:34.417672736Z caller=lifecycler.go:653 msg="found empty ring, inserting tokens" ring=ingester
level=info ts=2020-04-30T08:56:34.446224554Z caller=lifecycler.go:653 msg="found empty ring, inserting tokens" ring=ingester
level=warn ts=2020-04-30T08:56:34.480207872Z caller=grpc_logging.go:55 method=/cortex.Ingester/TransferChunks duration=14.841606424s err="Changing instance state from JOINING -> JOINING is disallowed" msg="gRPC\n"

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.