Hadoop multi node setup

[at15]

Currently, single node hadoop is working #5 with Yarn and can run HiBench, now having the real multi node setup is the major issue

Hadoop

• properly setup host
• can't view resource manager Web UI ResourceManager - http://localhost:8088/
• use different config and script for single and cluster
• hadoop prefix

 HADOOP_PREFIX=/path/to/hadoop
 export HADOOP_PREFIX

• https://github.com/njvijay/vagrant-jilla-hadoop/blob/master/resources/hadoop/hadoop.sh
• job history server is not started by start-dfs and start-yarn I guess?

Vagrant

• manage host using vagrant host manager https://github.com/devopsgroup-io/vagrant-hostmanager use static ip and fixed hosts file
• use private network instead of simple port forwarding
• built a base box for multiple virtual machine Setting up base box #7
• start multiple vm in parallel instead of wait one by one can't do that
  • https://www.vagrantup.com/docs/cli/up.html#_no_parallel
  • use for loop https://github.com/njvijay/vagrant-jilla-hadoop/blob/master/Vagrantfile

Extra

• jvm option
  • Configuring Environment of Hadoop Daemons https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html
• may consider using Ansible instead of raw shell scripts
  • need to solve the grub update problem

[at15]

well, it's running, so ...