Detailed deployment and configuration of CEPH cluster (II)

​Ceph Detailed cluster deployment configuration

One 、

Deployment structure

Virtual machines create three servers ,CENTOS Version is 7.6, IP Network segment 192.168.116.2/24. The names of the three hosts are ：

CENTOS7-1: IP by 192.168.116.141,  It is also a management node ,  Make a child node again .

CENTOS7-2: IP by 192.168.116.142,  Child node .

CENTOS7-3: IP by 192.168.116.143,  Child node

Two 、 The system configuration

System configuration work ,  The three nodes execute in turn ：

1、

Modify host name

[[email protected] ~]# vi /etc/hostname
CENTOS7-1

2、

edit hosts file

192.168.116.141 CENTOS7-1 
192.168.116.142 CENTOS7-2 
192.168.116.143 CENTOS7-3

Be careful ,  The host name should be consistent with the node name ,  Otherwise, there will be problems during installation

3、

modify yum Source

vi /etc/yum.repos.d/ceph.repo,  To avoid the problem of slow network speed ,  Here is the source image ：

[ceph] 
name=Ceph packages for $basearch
baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/x86_64/ 
enabled=1 
gpgcheck=1 
type=rpm-md 
gpgkey=https://download.ceph.com/keys/release.asc
 
[ceph-noarch] 
name=Ceph noarch packages 
#  Official source  
#baseurl=http://download.ceph.com/rpm-mimic/el7/noarch 
#  Tsinghua source  
baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/noarch/ 
enabled=1 
gpgcheck=1 
type=rpm-md 
gpgkey=https://download.ceph.com/keys/release.asc
 
[ceph-source] 
name=Ceph source packages
baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/SRPMS/ 
enabled=1 
gpgcheck=1 
type=rpm-md 
gpgkey=https://download.ceph.com/keys/release.asc

4、

install ceph And ceph-deploy Components

yum update && yum -y install ceph ceph-deploy

installation is complete ,  If you execute ceph-deploy appear ImportError: No module named pkg_resources

install python2-pip：

yum install epel-release -y

yum -y install python2-pip

5、

install NTP Time synchronization tool

 

yum install ntp ntpdate ntp-doc -y

 

Make sure the time zone is correct ,  Set boot up ：

systemctl enable ntpd

And time every 1 Hour auto calibration synchronization . edit  vi /etc/rc.d/rc.local  Additional ：

/usr/sbin/ntpdate ntp1.aliyun.com > /dev/null 2>&1; /sbin/hwclock -w

Configure scheduled tasks ,  perform crontab -e  Join in ：

3、 ... and 、 Password free SSH land

1、

It's not recommended to use the built-in users of the system ,  Create a ceph_user user ,  The password is also set to ceph_user：

useradd -d /home/ceph_user -m ceph_user 
passwd ceph_user

2、

Set up sudo jurisdiction

echo "ceph_user ALL = (root) NOPASSWD:ALL" | sudo tee
/etc/sudoers.d/ceph_user 
sudo chmod 0440 /etc/sudoers.d/ceph_user

1、2 The two steps are performed on three machines in turn .

Next, on the master node ,  Carry on ：

3、 Generate the key ：

Switching users ： su ceph_user

perform ssh-keygen, Always click generate according to the default prompt RSA Key information .

4、

Distribute the key to each machine node

ssh-copy-id [email protected] 
ssh-copy-id [email protected] 
ssh-copy-id [email protected]

5、

Modify... On the management node  ~/.ssh/config  file ,  simplify SSH Input information for remote connection ：

The management node will have root and ceph_user Multiple users , ssh Remote connection will log in as the current user by default ,  If we were root Remote connection with identity ,  Still need to enter password ,  We want to simplify ,  What to do with ？

Switch root identity

su root

modify ~/.ssh/config  file

Host CENTOS7-1 
 Hostname CENTOS7-1
 User ceph_user
Host CENTOS7-2 
 Hostname CENTOS7-2
 User ceph_user
Host CENTOS7-3 
 Hostname CENTOS7-3
 User ceph_user

 

Pay attention to modify file permissions ,  Can't use 777 Maximum authority ：

chmod 600 ~/.ssh/config

Conduct ssh When connecting remotely , Host The host name of is case sensitive ,  So pay attention to the host name of the configuration file .

6、 Open ports ,  Non-production environment ,  You can disable the firewall directly ：

 

systemctl stop firewalld.service 
systemctl disable firewalld.service

7、

SELINUX Set up

SELinux Set to disabled ：

setenforce 0

permanent ：

edit  vi /etc/selinux/config modify ：

SELINUX=disabled

Four 、

Cluster setup configuration

use root Install as

1、 Create the cluster configuration directory on the management node ,cd /usr/local：

mkdir ceph-cluster 
cd ceph-cluster

Be careful ： 

This directory serves as ceph The base directory of operation commands ,  Will store and process configuration information .

2、 Create clusters ,  Contains three machine nodes ：

ceph-deploy new CENTOS7-1 CENTOS7-2 CENTOS7-3

Once created ,  A configuration file will be generated .

3、 

If there is a problem with the installation and configuration of the next cluster ,  You can execute the following command to clear ,  reinstall ：

ceph-deploy purge CENTOS7-1 CENTOS7-2 CENTOS7-3 
ceph-deploy purgedata CENTOS7-1 CENTOS7-2 CENTOS7-3 
ceph-deploy forgetkeys

Put the three nodes of mon Information is also deleted

rm -rf /var/run/ceph/

4、 

Modify the configuration file ,  Some configurations need to be used later ：

vi /usr/local/ceph-cluster/ceph.conf

Join in ：

[global] 
#  The public network  
public network = 192.168.88.0/24 
#  Set up pool Pool default allocation quantity   The default number of copies is 3 
osd pool default size = 2 
#  Tolerate more clock errors  
mon clock drift allowed = 2 
mon clock drift warn backoff = 30 
#  Allow deletion of pool 
mon_allow_pool_delete = true
[mgr]
#  Turn on WEB The dashboard  mgr modules = dashboard

The first is the number of copies ,  Set to 2 Share .

The second is external IP Access network segment , Pay attention to the fact IP Modify the network segment .

Third 、 The four terms are drift errors allowed for a certain period of time .

5、

Execution and installation ：

ceph-deploy install CENTOS7-1 CENTOS7-2 CENTOS7-3

If something goes wrong ：

ceph_deploy][ERROR ] RuntimeError: Failed to execute command: ceph --version

It can be installed separately on each node ：

yum -y install ceph 

If there are no warehouse files ceph.repo,  Follow the steps above to create .

6、 initial monitor Information ：

ceph-deploy mon create-initial ## ceph-deploy --overwrite-conf mon create-initial

After execution ,  The following files will be generated ：

7、

Synchronize management information ：

 

Distribute configuration file and management information to each node ：

ceph-deploy admin CENTOS7-1 CENTOS7-2 CENTOS7-3

8、

install mgr( Manage daemons ),  Greater than 12.x Version needs to be installed ,  We have the latest version , To be implemented ：

 

ceph-deploy mgr create CENTOS7-1 CENTOS7-2 CENTOS7-3

9、

install OSD( Object storage device )

 

Be careful ：  The new version of the OSD No, prepare And activate command .

A new hard disk is needed here for OSD The storage device ,  Shut down the virtual machine ,  Add a hard disk ,  Don't format .

restart , fdisk -l  View the new disk name ：

Execution creation OSD command ：

ceph-deploy osd create --data /dev/sdb CENTOS7-1

All three nodes need to be executed in turn .

ceph-deploy gatherkeys CENTOS7-1

10、

Verify nodes ：

 

Input ceph health  or  ceph -s see ,  appear HEALTH_OK For normal .

Starting from a virtual machine ,  If something goes wrong :

 

 

Execute commands at each node ,  Make sure the time is synchronized :

ntpdate ntp1.aliyun.com

5、 ... and 、

Installation management background

1、  Turn on dashboard modular

ceph mgr module enable dashboard

2、

Generate signature  

ceph dashboard create-self-signed-cert

3、

Create directory  

mkdir mgr-dashboard

[[email protected] mgr-dashboard]# pwd 
/usr/local/ceph-cluster/mgr-dashboard

4、

Generate key pair

cd /usr/local/ceph-cluster/mgr-dashboard
openssl req -new -nodes -x509 -subj "/O=IT/CN=ceph-mgr-dashboard" -days 3650 -keyout dashboard.key -out dashboard.crt -extensions v3_ca

 

5、

start-up dashboard 

ceph mgr module disable dashboard 
ceph mgr module enable dashboard

6、

Set up IP And PORT 

 

ceph config set mgr mgr/dashboard/server_addr 192.168.88.161
ceph config set mgr mgr/dashboard/server_port 18843

7、

close HTTPS

ceph config set mgr mgr/dashboard/ssl false

8、

View service information  

[[email protected] mgr-dashboard]# ceph mgr services
{
 "dashboard": "https://node1:8443/"
}

9、  Set management user and password

ceph dashboard set-login-credentials admin admin

10、

visit

6、 ... and 、

establish Cephfs

After the cluster is created ,  There is no file system by default ,  We create a Cephfs A file system that supports external access .

ceph-deploy --overwrite-conf mds create CENTOS7-1 CENTOS7-2 CENTOS7-3

1、 Create two storage pools ,  Execute two commands ：

ceph osd pool create cephfs_data 128 
ceph osd pool create cephfs_metadata 64

Less than 5 individual OSD Can handle pg_num Set to 128

OSD Quantity in 5 To 10, You can set pg_num by 512

OSD Quantity in 10 To 50, You can set pg_num by 4096

OSD The number is larger than 50, Need to compute pg_num Value

You can list the currently created storage pools with the following command ：

ceph osd lspools

2.、 establish fs,  The name is fs_test:

ceph fs new fs_test cephfs_metadata cephfs_data

3、

Status view ,  The following information represents normal ：

 

[[email protected] mgr-dashboard]# ceph fs ls
name: fs_test, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
[[email protected] ceph-cluster]# ceph mds stat
fs_test-1/1/1 up {0=CENTOS7-2=up:active}, 2 up:standby

attach ：  If the creation error ,  You need to remove ,  perform ：

ceph fs rm fs_test --yes-i-really-mean-it 
ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it

Make sure that the ceph.conf Open the following configuration in ：

[mon] mon allow pool delete = true

4、

use fuse mount

Make sure the ceph-fuse Orders can be executed ,  without ,  Install ：

yum -y install ceph-fuse

5、 Create mount directory

mkdir -p /usr/local/cephfs_directory

6、

mount cephfs

ceph-fuse -k /etc/ceph/ceph.client.admin.keyring -m 192.168.88.161:6789 /usr/local/cephfs_directory

7、

View disk mount information

/usr/local/cephfs_directory Directory mounted successfully .

7、 ... and 、

Client connection verification (Rados Java)

1、 Install well JDK、GIT and MAVEN

2、 download rados java Client source code

git clone https://github.com/ceph/rados-java.git

Download directory location ：

3、

perform MAVEN install ,  Ignore test cases ：

 

[[email protected] rados-java]# mvn install -Dmaven.test.skip=true

 

Generate jar package , rados-0.7.0.jar

4、

Create soft link ,  Join in CLASSPATH

ln -s /usr/local/sources/rados-java/target/rados-0.7.0.jar /opt/jdk1.8.0_301/jre/lib/ext/rados-0.7.0.jar

install jna

yum -y install jna

Create soft link

ln -s /usr/share/java/jna.jar /opt/jdk1.8.0_301/jre/lib/ext/jna.jar

 

see

[[email protected] target]# ll 
/opt/jdk1.8.0_301/jre/lib/ext/jna.jar lrwxrwxrwx 1 root root /opt/jdk1.8.0_301/jre/lib/ext/jna.jar -> /usr/share/java/jna.jar

[[email protected] target]# ll 
/opt/jdk1.8.0_301/jre/lib/ext/rados-0.7.0.jar lrwxrwxrwx 1 root root
/opt/jdk1.8.0_301/jre/lib/ext/rados- 0.7.0.jar -> /usr/local/sources/rados-java/target/rados-0.7.0.jar

5、

establish JAVA Test class  

CephClient class , Be careful ,  The latest version 0.6 The exception handling package location for has changed .

importcom.ceph.rados.Rados;
importcom.ceph.rados.exceptions.*;
importjava.io.File;
publicclassCephClient{
    publicstaticvoidmain(Stringargs[]){
        try{
            Radoscluster=newRados("admin");
            System.out.println("Createdclusterhandle.");
            Filef=newFile("/etc/ceph/ceph.conf");
            cluster.confReadFile(f);
            System.out.println("Readtheconfigurationfile.");
            cluster.connect();
            System.out.println("Connectedtothecluster.");
 }catch(RadosExceptione){
 System.out.println(e.getMessage()+":"+e.getReturnValue());
 }
 }
}

6、

Run validation

Need to be in linux Operation in environment , And in client node .

Compile and run ：

[[email protected] sources]# javac CephClient.java 
[[email protected] sources]# java CephClient 
Created cluster handle.
Read the configuration file.
Connected to the cluster.

Success and ceph Establishing a connection .

Be careful ：

If java and javac Different versions , have access to rpm -qa |grep java  Search for , rpm -e --nodeps jdk To delete ,source /etc/profile Take effect

jdk The number of digits is 64 position

​