CollectiveAllReduceStrategy fails with CPU-only workers

[byronyi]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS High Sierra
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
• TensorFlow installed from (source or binary): tf-nightly
• TensorFlow version (use command below): ('v1.9.0-rc2-4081-g626bc997c2', '1.11.0-dev20180913')
• Python version: Python 2.7.15
• Bazel version (if compiling from source): N/A
• GCC/Compiler version (if compiling from source): N/A
• CUDA/cuDNN version: N/A
• GPU model and memory: N/A
• Exact command to reproduce: https://gist.github.com/df3df82f7ae8f47b6288fc42eb8c8b17

Describe the problem

Invoke tf.estimator.train_and_evaluate with CollectiveAllReduceStrategy fails on CPU-only worker nodes, with the following message:

InternalError: ScopedAllocatorMgr not supported on device /job:worker/replica:0/task:0/device:CPU:0

Source code / logs

from tensorflow.contrib.distribute import CollectiveAllReduceStrategy
from tensorflow.contrib.distribute import DistributeConfig

distribution = CollectiveAllReduceStrategy(num_gpus_per_worker=0)

config = tf.estimator.RunConfig(
    experimental_distribute=DistributeConfig(
        train_distribute=distribution,
        remote_cluster={
            'worker': ['localhost:5000', 'localhost:5001'],
        },
    )
)

estimator = tf.estimator.Estimator(model_fn=model_fn, config=config)

tf.estimator.train_and_evaluate(estimator=estimator, train_spec=train_spec, eval_spec=eval_spec)

INFO:tensorflow:CollectiveAllReduceStrategy with local_devices = ['/device:CPU:0']
INFO:tensorflow:Initializing RunConfig with distribution strategies.
INFO:tensorflow:RunConfig initialized for Distribute Coordinator with STANDALONE_CLIENT mode
WARNING:tensorflow:Using temporary folder as model directory: /var/folders/gn/sjntndrs1fs22kfr302697mr0000gn/T/tmpeIE_x6
INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_keep_checkpoint_max': 5, '_task_type': 'worker', '_global_id_in_cluster': 0, '_is_chief': True, '_cluster_spec': {'worker': ['localhost:5000', 'localhost:5001']}, '_model_dir': '/var/folders/gn/sjntndrs1fs22kfr302697mr0000gn/T/tmpeIE_x6', '_protocol': None, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_tf_random_seed': None, '_save_summary_steps': 100, '_device_fn': None, '_experimental_distribute': DistributeConfig(train_distribute=<tensorflow.contrib.distribute.python.collective_all_reduce_strategy.CollectiveAllReduceStrategy object at 0x123f09810>, eval_distribute=None, remote_cluster={'worker': ['localhost:5000', 'localhost:5001']}), '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_evaluation_master': '', '_eval_distribute': None, '_train_distribute': <tensorflow.contrib.distribute.python.collective_all_reduce_strategy.CollectiveAllReduceStrategy object at 0x123f09810>, '_master': '', '_distribute_coordinator_mode': 'standalone_client'}
INFO:tensorflow:Running `train_and_evaluate` with Distribute Coordinator.
INFO:tensorflow:Running Distribute Coordinator with mode = 'standalone_client', cluster_spec = {'worker': ['localhost:5000', 'localhost:5001']}, task_type = None, task_id = None, environment = None, rpc_layer = 'grpc'
WARNING:tensorflow:`eval_strategy` is not passed in. No distribution strategy will be used for evaluation.
INFO:tensorflow:Multi-worker CollectiveAllReduceStrategy with cluster_spec = {'worker': ['localhost:5000', 'localhost:5001']}, task_type = 'worker', task_id = 0, num_workers = 2, local_devices = ['/job:worker/task:0']
INFO:tensorflow:Multi-worker CollectiveAllReduceStrategy with cluster_spec = {'worker': ['localhost:5000', 'localhost:5001']}, task_type = 'worker', task_id = 1, num_workers = 2, local_devices = ['/job:worker/task:1']
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Collective All-reduce invoked with batches size = 2, num_workers = 2
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Creating chief session creator with config: device_filters: "/job:worker/task:0"
allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
    scoped_allocator_optimization: ON
    scoped_allocator_opts {
      enable_op: "CollectiveReduce"
    }
  }
}
isolate_session_state: true
experimental {
  collective_group_leader: "/job:worker/replica:0/task:0"
}

INFO:tensorflow:Collective All-reduce invoked with batches size = 2, num_workers = 2
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Creating chief session creator with config: device_filters: "/job:worker/task:1"
allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
    scoped_allocator_optimization: ON
    scoped_allocator_opts {
      enable_op: "CollectiveReduce"
    }
  }
}
isolate_session_state: true
experimental {
  collective_group_leader: "/job:worker/replica:0/task:0"
}

INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Initialize system
INFO:tensorflow:Initialize system
INFO:tensorflow:Saving checkpoints for 0 into /var/folders/gn/sjntndrs1fs22kfr302697mr0000gn/T/tmpeIE_x6/model.ckpt.

Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/distribute/distribute_coordinator.py", line 344, in _run_single_worker
    worker_fn(strategy)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/distribute/estimator_training.py", line 232, in _worker_fn
    hooks=list(train_spec.hooks))
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 355, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1178, in _train_model
    return self._train_model_distributed(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1325, in _train_model_distributed
    saving_listeners)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1408, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1148, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1239, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1224, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1296, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1076, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 887, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1110, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1286, in _do_run
    run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1306, in _do_call
    raise type(e)(node_def, op, message)
InternalError: ScopedAllocatorMgr not supported on device /job:worker/replica:0/task:1/device:CPU:0
	 [[{{node scoped_allocator_1}} = _ScopedAllocator[T=DT_FLOAT, expected_call_count=2, id=1, sa_name="scoped_allocator_1", shape=[17], shapes=[[1,1], [1]], _device="/job:worker/replica:0/task:1/device:CPU:0"]()]]

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/distribute/distribute_coordinator.py", line 344, in _run_single_worker
    worker_fn(strategy)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/distribute/estimator_training.py", line 232, in _worker_fn
    hooks=list(train_spec.hooks))
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 355, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1178, in _train_model
    return self._train_model_distributed(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1325, in _train_model_distributed
    saving_listeners)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 1408, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1148, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1239, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1224, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1296, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1076, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 887, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1110, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1286, in _do_run
    run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1306, in _do_call
    raise type(e)(node_def, op, message)
InternalError: ScopedAllocatorMgr not supported on device /job:worker/replica:0/task:0/device:CPU:0
	 [[{{node scoped_allocator_1}} = _ScopedAllocator[T=DT_FLOAT, expected_call_count=2, id=1, sa_name="scoped_allocator_1", shape=[17], shapes=[[1,1], [1]], _device="/job:worker/replica:0/task:0/device:CPU:0"]()]]

PS: thanks @yuefengz for today's introduction of multi-node distribution strategy in TF Roadshow 2018@Beijing 😆

[byronyi]

Ping @dubey @poxvoculi

[poxvoculi]

Looks like your CPU device is not overriding the

virtual ScopedAllocatorMgr* GetScopedAllocatorMgr() const { return nullptr; }

method on DeviceBase. It is overridden in common_runtime/threadpool_device.h, which is normally the only way of building a CPU device. I'm not sure whether this is an issue with the MacOS build or something specific to your build. Given some more time, I'll try to reproduce.

[yuefengz]

@byronyi really appreciate you came to our roadshow event!

For now, if there is not a simple workaround, you can try MirroredStrategy as well. But we still recommend CollectiveAllReducStrategy for multi-node training and we will fix it as soon as possible. Thanks!

[byronyi]

It seems I used SingleThreadedCpuDevice that was added in 9b18bd7, later than when ScopedAllocator was added in 282750f.

@skye Mind to fix this? Or I could help submit a patch if you're okay with it.

EDIT: turns out it's the problem of RenamedDevice.

[skye]

@byronyi if you'd like to submit a PR that'd be great!

[byronyi]

@poxvoculi @dubey PR submitted.

[byronyi]

@yuefengz I just tried MirroredStrategy but it seems to require NCCL even if it is a CPU-only build.

Is this going to be fixed soon? If not, I will open a separate issue for this.

INFO:tensorflow:Initializing RunConfig with distribution strategies.
INFO:tensorflow:RunConfig initialized for Distribute Coordinator with STANDALONE_CLIENT mode
WARNING:tensorflow:Using temporary folder as model directory: /var/folders/gn/sjntndrs1fs22kfr302697mr0000gn/T/tmpSuQoQy
INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_keep_checkpoint_max': 5, '_task_type': 'worker', '_global_id_in_cluster': 0, '_is_chief': True, '_cluster_spec': {'ps': ['localhost:5000'], 'worker': ['localhost:5001']}, '_model_dir': '/var/folders/gn/sjntndrs1fs22kfr302697mr0000gn/T/tmpSuQoQy', '_protocol': None, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_tf_random_seed': None, '_save_summary_steps': 100, '_device_fn': None, '_experimental_distribute': DistributeConfig(train_distribute=<tensorflow.contrib.distribute.python.mirrored_strategy.MirroredStrategy object at 0x12316fc90>, eval_distribute=None, remote_cluster={'ps': ['localhost:5000'], 'worker': ['localhost:5001']}), '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_evaluation_master': '', '_eval_distribute': None, '_train_distribute': <tensorflow.contrib.distribute.python.mirrored_strategy.MirroredStrategy object at 0x12316fc90>, '_master': '', '_distribute_coordinator_mode': 'standalone_client'}
INFO:tensorflow:Running `train_and_evaluate` with Distribute Coordinator.
INFO:tensorflow:Running Distribute Coordinator with mode = 'standalone_client', cluster_spec = {'ps': ['localhost:5000'], 'worker': ['localhost:5001']}, task_type = None, task_id = None, environment = None, rpc_layer = 'grpc'
WARNING:tensorflow:`eval_strategy` is not passed in. No distribution strategy will be used for evaluation.
WARNING:tensorflow:Could not find a standard reader in the input pipeline(one of TextLineDataset, TFRecordDataset, FixedLengthRecordDataset).So auto-sharding is not done. Please verify correctness of auto-sharding for your input.
WARNING:tensorflow:Could not find a standard reader in the input pipeline(one of TextLineDataset, TFRecordDataset, FixedLengthRecordDataset).So auto-sharding is not done. Please verify correctness of auto-sharding for your input.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:batch_all_reduce invoked for batches size = 2 with algorithm = nccl, num_packs = 1, agg_small_grads_max_bytes = 0 and agg_small_grads_max_group = 10
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Creating chief session creator with config: allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
isolate_session_state: true

INFO:tensorflow:Graph was finalized.
---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
<ipython-input-6-77bd9e8610db> in <module>()
     17 train_spec = tf.estimator.TrainSpec(input_fn=input_fn)
     18 eval_spec = tf.estimator.EvalSpec(input_fn=input_fn)
---> 19 tf.estimator.train_and_evaluate(estimator=estimator, train_spec=train_spec, eval_spec=eval_spec)

/usr/local/lib/python2.7/site-packages/tensorflow/python/estimator/training.pyc in train_and_evaluate(estimator, train_spec, eval_spec)
    460     logging.info('Running `train_and_evaluate` with Distribute Coordinator.')
    461     distribute_coordinator_training.train_and_evaluate(
--> 462         estimator, train_spec, eval_spec, _TrainingExecutor)
    463     return
    464 

/usr/local/lib/python2.7/site-packages/tensorflow/python/distribute/estimator_training.pyc in train_and_evaluate(estimator, train_spec, eval_spec, executor_cls)
    262       mode=run_config._distribute_coordinator_mode,
    263       cluster_spec=cluster_spec,
--> 264       session_config=run_config.session_config)

/usr/local/lib/python2.7/site-packages/tensorflow/python/distribute/distribute_coordinator.pyc in run_distribute_coordinator(worker_fn, strategy, eval_fn, eval_strategy, mode, cluster_spec, task_type, task_id, session_config, rpc_layer)
    747       else:
    748         _run_in_graph_client(worker_fn, strategy, eval_fn, eval_strategy,
--> 749                              cluster_spec, session_config, rpc_layer)
    750     else:
    751       # If not a client job, run the standard server.

/usr/local/lib/python2.7/site-packages/tensorflow/python/distribute/distribute_coordinator.pyc in _run_in_graph_client(worker_fn, strategy, eval_fn, eval_strategy, cluster_spec, session_config, rpc_layer)
    471       None,
    472       session_config,
--> 473       rpc_layer=rpc_layer)
    474   if eval_thread:
    475     eval_thread.join()

/usr/local/lib/python2.7/site-packages/tensorflow/python/distribute/distribute_coordinator.pyc in _run_single_worker(worker_fn, strategy, cluster_spec, task_type, task_id, session_config, rpc_layer, worker_barrier)
    342       worker_barrier=worker_barrier)
    343   with context:
--> 344     worker_fn(strategy)
    345 
    346 

/usr/local/lib/python2.7/site-packages/tensorflow/python/distribute/estimator_training.pyc in _worker_fn(strategy)
    230         input_fn=train_spec.input_fn,
    231         max_steps=train_spec.max_steps,
--> 232         hooks=list(train_spec.hooks))
    233 
    234   def _eval_fn(strategy):

/usr/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.pyc in train(self, input_fn, hooks, steps, max_steps, saving_listeners)
    353 
    354       saving_listeners = _check_listeners_type(saving_listeners)
--> 355       loss = self._train_model(input_fn, hooks, saving_listeners)
    356       logging.info('Loss for final step: %s.', loss)
    357       return self

/usr/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.pyc in _train_model(self, input_fn, hooks, saving_listeners)
   1176   def _train_model(self, input_fn, hooks, saving_listeners):
   1177     if self._train_distribution:
-> 1178       return self._train_model_distributed(input_fn, hooks, saving_listeners)
   1179     else:
   1180       return self._train_model_default(input_fn, hooks, saving_listeners)

/usr/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.pyc in _train_model_distributed(self, input_fn, hooks, saving_listeners)
   1323         return self._train_with_estimator_spec(estimator_spec, worker_hooks,
   1324                                                hooks, global_step_tensor,
-> 1325                                                saving_listeners)
   1326 
   1327   def _train_with_estimator_spec(self, estimator_spec, worker_hooks, hooks,

/usr/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.pyc in _train_with_estimator_spec(self, estimator_spec, worker_hooks, hooks, global_step_tensor, saving_listeners)
   1403         save_summaries_steps=self._config.save_summary_steps,
   1404         config=self._session_config,
-> 1405         log_step_count_steps=self._config.log_step_count_steps) as mon_sess:
   1406       loss = None
   1407       while not mon_sess.should_stop():

/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.pyc in MonitoredTrainingSession(master, is_chief, checkpoint_dir, scaffold, hooks, chief_only_hooks, save_checkpoint_secs, save_summaries_steps, save_summaries_secs, config, stop_grace_period_secs, log_step_count_steps, max_wait_secs, save_checkpoint_steps, summary_dir)
    449         max_wait_secs=max_wait_secs,
    450         save_checkpoint_steps=save_checkpoint_steps,
--> 451         summary_dir=summary_dir)
    452 
    453   if not is_chief:

/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.pyc in _create_monitored_session_with_worker_context(worker_context, scaffold, checkpoint_dir, hooks, chief_only_hooks, save_checkpoint_secs, save_summaries_steps, save_summaries_secs, config, stop_grace_period_secs, log_step_count_steps, max_wait_secs, save_checkpoint_steps, summary_dir)
    340       session_creator=session_creator,
    341       hooks=all_hooks,
--> 342       stop_grace_period_secs=stop_grace_period_secs)
    343 
    344 

/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.pyc in __init__(self, session_creator, hooks, stop_grace_period_secs)
    919     super(MonitoredSession, self).__init__(
    920         session_creator, hooks, should_recover=True,
--> 921         stop_grace_period_secs=stop_grace_period_secs)
    922 
    923 

/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.pyc in __init__(self, session_creator, hooks, should_recover, stop_grace_period_secs)
    641         stop_grace_period_secs=stop_grace_period_secs)
    642     if should_recover:
--> 643       self._sess = _RecoverableSession(self._coordinated_creator)
    644     else:
    645       self._sess = self._coordinated_creator.create_session()

/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.pyc in __init__(self, sess_creator)
   1105     """
   1106     self._sess_creator = sess_creator
-> 1107     _WrappedSession.__init__(self, self._create_session())
   1108 
   1109   def _create_session(self):

/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.pyc in _create_session(self)
   1110     while True:
   1111       try:
-> 1112         return self._sess_creator.create_session()
   1113       except _PREEMPTION_ERRORS as e:
   1114         logging.info('An error was raised while a session was being created. '

/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.pyc in create_session(self)
    798       """Creates a coordinated session."""
    799       # Keep the tf_sess for unit testing.
--> 800       self.tf_sess = self._session_creator.create_session()
    801       # We don't want coordinator to suppress any exception.
    802       self.coord = coordinator.Coordinator(clean_stop_exception_types=[])

/usr/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.pyc in create_session(self)
    564         init_op=self._scaffold.init_op,
    565         init_feed_dict=self._scaffold.init_feed_dict,
--> 566         init_fn=self._scaffold.init_fn)
    567 
    568 

/usr/local/lib/python2.7/site-packages/tensorflow/python/training/session_manager.pyc in prepare_session(self, master, init_op, saver, checkpoint_dir, checkpoint_filename_with_path, wait_for_checkpoint, max_wait_secs, config, init_feed_dict, init_fn)
    285                            "init_fn or local_init_op was given")
    286       if init_op is not None:
--> 287         sess.run(init_op, feed_dict=init_feed_dict)
    288       if init_fn:
    289         init_fn(sess)

/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
    885     try:
    886       result = self._run(None, fetches, feed_dict, options_ptr,
--> 887                          run_metadata_ptr)
    888       if run_metadata:
    889         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1108     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1109       results = self._do_run(handle, final_targets, final_fetches,
-> 1110                              feed_dict_tensor, options, run_metadata)
   1111     else:
   1112       results = []

/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1284     if handle is None:
   1285       return self._do_call(_run_fn, feeds, fetches, targets, options,
-> 1286                            run_metadata)
   1287     else:
   1288       return self._do_call(_prun_fn, handle, feeds, fetches)

/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args)
   1304           pass
   1305       message = error_interpolation.interpolate(message, self._graph)
-> 1306       raise type(e)(node_def, op, message)
   1307 
   1308   def _extend_graph(self):

NotFoundError: Op type not registered 'NcclAllReduce' in binary running on Bairens-MacBook.local. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.

[guptapriya]

@byronyi would be good to file a separate issue for MirroredStrategy, we should not default to nccl for CPU only builds.

[byronyi]

I guess #21470 would be enough for now? I saw multiple issues opened for dist-strat, let's not overload @yuefengz at this moment :)

[yuefengz]

@byronyi it uses nccl as default all-reduce operations. The simplest workaround it to install nccl, or specifiy cross_tower_ops=tf.contrib.distribute.ReductionToOneDeviceCrossTowerOps in MirroredStrategy. If you have multiple workers in your cluster, MirroredStrategy will switch to another cross_tower_ops. Would that work for you?

We probably can check whether nccl is installed and if not switch to ReductionToOneDeviceCrossTowerOps in the future.