Centralized management of clusters

One 、 The current problems

Before reading this article , You should read ： Get to know namenode and datanode.

Before , We started Hadoop In clusters , The first is to start namenode, Then start datanode. Be careful ： Our previous practice is to manually start datanode The command of is sent to all datanode, Obviously, if the cluster is very large , This is not appropriate . We hope to pass start-dfs.sh Start all nodes . So we need to configure namenode On top of the machine slaves file , This file manages this namenode All of the following datanode. The location of this file is ：{hadoop_home}/etc/hadoop, among {hadoop_home} yes Hadoop Installation directory .

Two 、 To configure namenode On the machine slaves file

1. function cd  /usr/local/hadoop/hadoop-2.7.3/etc/hadoop Enter into slaves Directory of files

2. function vim slaves command , add to slave Machine name , Save and exit .

3、 ... and 、 Configuration free SSH Remote login

1. stay namenode On the machine , Get into /root/.ssh Below directory , Run the command : ssh-keygen -t rsa

 2. take namenode The public key on the machine copy To npfdev1( This machine ),npfdev2,npfdev3,npfdev4 above .

Run the following command ：

ssh-copy-id npfdev1

ssh-copy-id npfdev2

ssh-copy-id npfdev3

ssh-copy-id npfdev4

Four 、 stay namenode Machine running start-dfs.sh Start cluster

1. Once the boot is complete , see ：

Be careful ：start-dfs.sh Will start by default secondarynamenode

5、 ... and 、 stay namenode Machine running stop-dfs.sh Start cluster

1. After stopping , see ：

6、 ... and 、 Startup and shutdown Hadoop Summary of cluster command steps ：

1.  modify master On /etc/hadoop/slaves file , every last slave Occupy a line .

2.  Configuration free SSH Remote login .

3. start-dfs.sh Start cluster .

4. stop-dfs.sh Stop the cluster .

Be careful ： If you make similar mistakes ,

The solution is in hadoop-env.sh and yarn-env.sh Add the following two lines ：

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"

reference ：

1. Hadoop 2.2.0 Cluster installation