ceph-deploy Deploy ceph colony

Introduction to the environment

Host name ip Address operating system role remarks
ceph-node110.153.204.13Centos7.6mon、osd、mds、mgr、rgw、ceph-deploychronyd Clock synchronization ( Lord )
ceph-node210.130.22.45Centos7.6mon、osd、mds、mgr、rgwchronyd Clock synchronization
ceph-node310.153.204.28Centos7.3mon、osdchronyd Clock synchronization

There are three machines in this environment , Before operation ntp Need to be synchronized ,node1 by ceph-deploy Deployment nodes , Three partitions per machine are used as osd disk .

ceph Component is introduced

name effect
osd Full name Object Storage Device, The main function is to store data 、 Copy the data 、 Balance data 、 Recover data, etc . Every OSD A heartbeat check will be performed between , And report some changes to Ceph Monitor.
mon Full name Monitor, Responsible for monitoring Ceph colony , maintain Ceph The health of the cluster , And maintain Ceph All kinds of Map chart , such as OSD Map、Monitor Map、PG Map and CRUSH Map, these Map Collectively referred to as Cluster Map, according to Map Map and object id Calculate the final location of data storage .
mgr Full name Manager, Responsible for tracking runtime metrics and Ceph The current state of the cluster , Including storage utilization , Current performance indicators and system load .
mds The full name is MetaData Server, The metadata of the file system services mainly saved , If you use cephfs Function will enable it , Object storage and block storage devices do not need to use this service .
rgw Full name radosgw, It's a set based on the current popular RESTFUL Protocol gateway ,ceph Entry to object storage , Do not enable object storage , No installation is required .

Each component needs to ensure high availability ：

1.osd The more services , In the case of the same replica, the higher availability is .

2.mon Three are generally deployed , Guaranteed high availability .

3.mgr Two are generally deployed , Guaranteed high availability .

4.mds Two sets are generally deployed to ensure high availability , Each set is master-slave .

5.rgw Two are generally deployed , Guaranteed high availability .

ceph Version introduction

first Ceph The version is 0.1 , Go back to 2008 year 1 month . these years , The version number scheme has not changed , until 2015 year 4 month 0.94.1 （ Hammer The first revision of ） After the release of , for fear of 0.99 （ as well as 0.100 or 1.00 ）, Developed a new strategy ：

  • x.0.z - Development Edition （ For early testers and Warriors ）
  • x.1.z - Candidate Edition （ For testing clusters 、 Master ）
  • x.2.z - Stable 、 Revised version （ For users ）

the ceph version 15.2.9,ceph-deploy 2.0.1

ceph Preparations before installation

1. Upgrade the system kernel to 4 Department or above

I upgraded here to 4.17, Upgrade steps are omitted here .

2.firewalld、iptables、SElinux close 
##  A firewall 

systemctl stop firewalld.service

systemctl disable firewalld.service

## selinux

setenforce

sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
3.chronyd Time synchronization

Here we use node1 It is the clock server , Other nodes are clock clients 

[master Lower operation ]

vim /etc/chrony.conf

...

##  The main points are as follows 

server 10.153.204.13 iburst # Specify server 

allow 10.0.0.0/8 # Think of yourself as a server 

...

[slave Lower operation ]

vim /etc/chrony.conf

...

server 10.153.204.13 iburst # Specify server 

...

##  Then restart the service , Check the status 

systemctl enable chronyd

systemctl restart chronyd

timedatectl

chronyc sources -v
4. stay ceph-deploy Node write temporary hosts file 
# cat /etc/hosts

10.153.204.13  ceph-node1

10.130.22.45 ceph-node2

10.153.204.28 ceph-node3
5. Create a normal user , give sudo jurisdiction , And will ceph-deploy Nodes do secret free operations on other nodes 
##  utilize ansible Create... For all machines  cephadmin  user 

ansible all -m shell -a 'groupadd -r -g 2022 cephadmin && useradd -r -m -s /bin/bash -u 2022 -g 2022 cephadmin && echo cephadmin:123456 | chpasswd'

##  give sudo jurisdiction , You don't need a password 

ansible node -m shell -a 'echo "cephadmin    ALL=(ALL)    NOPASSWD:ALL" >> /etc/sudoers'

##  No secret 

su - cephadmin

ssh-keygen

ssh-copy-id ceph-node2

ssh-copy-id ceph-node3
6. take osd The disk is ready , It's better to have one disk osd, This environment is short of resources , I have one partition here and one osd
[[email protected] ~]$ lsblk

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

nvme0n1     259:0    0 931.5G  0 disk

├─nvme0n1p5 259:7    0   100G  0 part

├─nvme0n1p3 259:5    0   100G  0 part

├─nvme0n1p6 259:8    0   100G  0 part

├─nvme0n1p4 259:6    0   100G  0 part

all osd The distribution of machine disks is the same , Just partition , Don't create lvm、 Format, etc .

7. If it's an intranet machine , You need to build your own local ceph yum Source

（1） Find an Internet machine , Execute this script , You can change the version information and url Address 

#!/usr/bin/env bash

URL_REPO=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-15.2.9/el7/x86_64/

URL_REPODATA=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-15.2.9/el7/x86_64/repodata/

function get_repo()

{

test -d ceph_repo || mkdir ceph_repo

cd ceph_repo

for i in `curl $URL_REPO | awk -F '"' '{print $4}' | grep rpm`;do

    curl -O $URL_REPO/$i

done

}

function get_repodata()

{

test -d ceph_repo/repodata || mkdir ceph_repo/repodata

cd ceph_repo/repodata

for i in `curl $URL_REPODATA | awk -F '"' '{print $4}' | grep xml`;do

    curl -O $URL_REPODATA/$i

done

}

if [ $1 == 'repo' ];then

    get_repo()

elif [ $1 == 'repodata' ];then

    get_repodata()

elif [ $1 == 'all' ];then

    get_repo()

    get_repodata()

else

    echo ' Please enter one of the parameters [ repo | repodata | all ]'

fi

（2） Upload to intranet server , install To configure nginx

yum -y install nginx

##  Mainly modify the following fields ,/home/ceph_repo  Replace with your real Directory .

vim /etc/nginx/nginx.conf

    server {

        listen       8080;

        listen       [::]:8080;

        server_name  _;

        root         /home/ceph;

        # Load configuration files for the default server block.

        include /etc/nginx/default.d/*.conf;

        location / {

           autoindex on;

        }

    }

systemctl start nginx

（3） To configure yum Source -- Each node should be configured 

cat > /etc/yum.repos.d/ceph-http.repo << EOF

[local-ceph]

name=local-ceph

baseurl=http://ceph-node1:8080/ceph_repo

gpgcheck=0

enable=1

[noarch-ceph]

name=local-ceph

baseurl=http://ceph-node1:8080/noarch_repo

gpgcheck=0

enable=1

EOF

then 

yum makecache

##  Check whether it is effective 

yum list | grep ceph

ceph-deploy Deploy

1. View and present ceph-deploy edition 

# yum list ceph-deploy --showduplicates

Loaded plugins: fastestmirror, langpacks, priorities

Loading mirror speeds from cached hostfile

Available Packages

ceph-deploy.noarch                                     1.5.25-1.el7                                     epel

ceph-deploy.noarch                                     1.5.29-0                                         noarch-ceph

ceph-deploy.noarch                                     1.5.30-0                                         noarch-ceph

ceph-deploy.noarch                                     1.5.31-0                                         noarch-ceph

ceph-deploy.noarch                                     1.5.32-0                                         noarch-ceph

ceph-deploy.noarch                                     1.5.33-0                                         noarch-ceph

ceph-deploy.noarch                                     1.5.34-0                                         noarch-ceph

ceph-deploy.noarch                                     1.5.35-0                                         noarch-ceph

ceph-deploy.noarch                                     1.5.36-0                                         noarch-ceph

ceph-deploy.noarch                                     1.5.37-0                                         noarch-ceph

ceph-deploy.noarch                                     1.5.38-0                                         noarch-ceph

ceph-deploy.noarch                                     1.5.39-0                                         noarch-ceph

ceph-deploy.noarch                                     2.0.0-0                                          noarch-ceph

ceph-deploy.noarch                                     2.0.1-0                                          noarch-ceph

It was first installed here 1.5.38 edition , But initialization osd When an error , The end use is 2.0.1. Find Alibaba cloud or the new version of Tsinghua cloud ：https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-15.2.9/el7/noarch/ or https://mirrors.aliyun.com/ceph

2. install ceph-deploy

## ceph Needed python Environment depends on , Put together 

yum -y install ceph-common python-pkg-resources python-setuptools python2-subprocess32

##  loading deploy

yum -y install ceph-deploy-2.0.1

##  After installation , You can view help commands 

ceph-deploy --help

If you make a mistake ImportError: No module named pkg_resources, Put on python-setuptools It's just a bag .

ceph Cluster initialization , Deploy

1. initialization mon The server ( First initialize one , Back again add Other nodes )

##  Before initialization , It's best to do it in advance at each  mon  All nodes will mon The package is installed , In subsequent installations, the program will automatically install , It is installed in advance to find problems in advance 

yum -y install ceph-mon

(1) Start initializing the configuration file , Specify the network segments of public and private networks , Generate ceph.conf The configuration file 

$ ceph-deploy new --cluster-network 10.0.0.0/8 --public-network 10.0.0.0/8 ceph-node1

[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (1.5.25): /bin/ceph-deploy new --cluster-network 10.0.0.0/8 --public-network 10.0.0.0/8 ceph-node1

[ceph_deploy.new][DEBUG ] Creating new cluster named ceph

[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds

[ceph_deploy][ERROR ] Traceback (most recent call last):

[ceph_deploy][ERROR ]   File "/usr/lib/python2.7/site-packages/ceph_deploy/util/decorators.py", line 69, in newfunc

[ceph_deploy][ERROR ]     return f(*a, **kw)

[ceph_deploy][ERROR ]   File "/usr/lib/python2.7/site-packages/ceph_deploy/cli.py", line 162, in _main

[ceph_deploy][ERROR ]     return args.func(args)

[ceph_deploy][ERROR ]   File "/usr/lib/python2.7/site-packages/ceph_deploy/new.py", line 141, in new

[ceph_deploy][ERROR ]     ssh_copy_keys(host, args.username)

[ceph_deploy][ERROR ]   File "/usr/lib/python2.7/site-packages/ceph_deploy/new.py", line 35, in ssh_copy_keys

[ceph_deploy][ERROR ]     if ssh.can_connect_passwordless(hostname):

[ceph_deploy][ERROR ]   File "/usr/lib/python2.7/site-packages/ceph_deploy/util/ssh.py", line 15, in can_connect_passwordless

[ceph_deploy][ERROR ]     if not remoto.connection.needs_ssh(hostname):

[ceph_deploy][ERROR ] AttributeError: 'module' object has no attribute 'needs_ssh'

[ceph_deploy][ERROR ]

This question is related to ceph-deploy The version is about , Instruction to add parameters “--no-ssh-copykey” that will do ：

$ ceph-deploy new --cluster-network 10.0.0.0/8 --public-network 10.0.0.0/8 ceph-node1 --no-ssh-copykey

[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (1.5.38): /bin/ceph-deploy new --cluster-network 10.0.0.0/8 --public-network 10.0.0.0/8 ceph-node1 --no-ssh-copykey

[ceph_deploy.new][DEBUG ] Creating new cluster named ceph

[ceph-node1][DEBUG ] connection detected need for sudo

[ceph-node1][DEBUG ] connected to host: ceph-node1

[ceph-node1][DEBUG ] detect platform information from remote host

[ceph-node1][DEBUG ] detect machine type

[ceph-node1][DEBUG ] find the location of an executable

[ceph-node1][INFO  ] Running command: sudo /usr/sbin/ip link show

[ceph-node1][INFO  ] Running command: sudo /usr/sbin/ip addr show

[ceph-node1][DEBUG ] IP addresses found: [u'192.168.42.1', u'10.153.204.13', u'10.233.64.0', u'10.233.64.1', u'169.254.25.10']

[ceph_deploy.new][DEBUG ] Resolving host ceph-node1

[ceph_deploy.new][DEBUG ] Monitor ceph-node1 at 10.153.204.13

[ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph-node1']

[ceph_deploy.new][DEBUG ] Monitor addrs are [u'10.153.204.13']

[ceph_deploy.new][DEBUG ] Creating a random mon key...

[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...

[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...

If ceph-deploy The version is 1.5.25 Left and right words , The best solution is to ceph-deploy Upgrade the program to 2.0.1; Execute again after upgrading .

(2) Start initializing mon node 

ceph-deploy mon create-initial

Report errors ：

[ceph-node1][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status

[ceph-node1][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory

This should be because I have deployed , The environment is not deleted clean when deleting , Delete it completely again , And then execute ：

##  Delete 

rm -rf /etc/ceph/* /var/lib/ceph/* /var/log/ceph/* /var/run/ceph/*

Re execution , success ：

[ceph-node1][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-node1/keyring auth get client.bootstrap-rgw

[ceph_deploy.gatherkeys][INFO  ] Storing ceph.client.admin.keyring

[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-mds.keyring

[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-mgr.keyring

[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.mon.keyring' already exists

[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-osd.keyring

[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-rgw.keyring

[ceph_deploy.gatherkeys][INFO  ] Destroy temp directory /tmp/tmps6CzLR

verification mon Startup successful 

# ps -ef | grep ceph-mon

ceph     23737     1  0 16:22 ?        00:00:00 /usr/bin/ceph-mon -f --cluster ceph --id ceph-node1 --setuser ceph --setgroup ceph

mon Initialization complete .

mon After initialization , You can look at ceph The state of the cluster , You can set up more management terminals

The cluster configuration file as well as admin User key Transfer to the target machine /etc/ceph/, Can operate ceph colony ：

ceph-deploy admin ceph-node1 ceph-node2 ceph-node2

$ ll -h /etc/ceph/

total 8.0K

-rw------- 1 root root 151 Feb 12 16:35 ceph.client.admin.keyring

-rw-r--r-- 1 root root 265 Feb 12 16:35 ceph.conf

-rw------- 1 root root   0 Feb 12 16:22 tmppE21x5

##  View the cluster status 

$ sudo ceph -s

  cluster:

    id:     537175bb-51de-4cc4-9ee3-b5ba8842bff2

    health: HEALTH_OK

  services:

    mon: 1 daemons, quorum ceph-node1 (age 14m)

    mgr: no daemons active

    osd: 0 osds: 0 up, 0 in

  data:

    pools:   0 pools, 0 pgs

    objects: 0 objects, 0 B

    usage:   0 B used, 0 B / 0 B avail

    pgs:

Now there is only one mon

2. add to mgr service

（1） install mgr package , Every mgr All nodes are installed 

yum -y install ceph-mgr

（2） add to mgr To cluster 

$ ceph-deploy mgr create ceph-node1

[ceph-node1][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.ceph-node1 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-ceph-node1/keyring

[ceph-node1][INFO  ] Running command: sudo systemctl enable [email protected]

[ceph-node1][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/[email protected] to /usr/lib/systemd/system/[email protected]

[ceph-node1][INFO  ] Running command: sudo systemctl start [email protected]

[ceph-node1][INFO  ] Running command: sudo systemctl enable ceph.target

##  Look again ceph State of the cluster 

# ceph -s

  cluster:

    id:     537175bb-51de-4cc4-9ee3-b5ba8842bff2

    health: HEALTH_WARN

            Module 'restful' has failed dependency: No module named 'pecan'

            OSD count 0 < osd_pool_default_size 3

  services:

    mon: 1 daemons, quorum ceph-node1 (age 46m)

    mgr: ceph-node1(active, since 100s)

    osd: 0 osds: 0 up, 0 in

  data:

    pools:   0 pools, 0 pgs

    objects: 0 objects, 0 B

    usage:   0 B used, 0 B / 0 B avail

    pgs:

There are three alarms ：

  • Module 'restful' has failed dependency: No module named 'pecan'

  • Module 'restful' has failed dependency: No module named 'werkzeug'

    This is mgr The machine lacks pecan and werkzeug modular , You can find an Internet machine to use pip3 Download the offline package and the offline package and then upload them to install ：https://pypi.tuna.tsinghua.edu.cn/simple/;https://pypi.org/simple.

  • OSD count 0 < osd_pool_default_size 3:

    osd The default number of copies per object in the is 3, This alarm prompts osd The number is less than three , This can be ignored for a while .

3. initialization osd

##  First check the available disks of the target host 

$ ceph-deploy disk list ceph-node1

 Report errors ：

[ceph_deploy][ERROR ] ExecutableNotFound: Could not locate executable 'ceph-disk' make sure it is installed and available on ceph-node1

Later, check the official website https://docs.ceph.com/en/pacific/ceph-volume/ Find out , stay Ceph version 13.0.0 when ,ceph-disk It has been deprecated , change to the use of sth. ceph-volume, Look at all the commands. There is really no ceph-disk Only ceph-volume

# locate ceph- |grep bin

/usr/bin/ceph-authtool

/usr/bin/ceph-bluestore-tool

/usr/bin/ceph-clsinfo

/usr/bin/ceph-conf

/usr/bin/ceph-crash

/usr/bin/ceph-dencoder

/usr/bin/ceph-deploy

/usr/bin/ceph-kvstore-tool

/usr/bin/ceph-mds

/usr/bin/ceph-mgr

/usr/bin/ceph-mon

/usr/bin/ceph-monstore-tool

/usr/bin/ceph-objectstore-tool

/usr/bin/ceph-osd

/usr/bin/ceph-osdomap-tool

/usr/bin/ceph-post-file

/usr/bin/ceph-rbdnamer

/usr/bin/ceph-run

/usr/bin/ceph-syn

/usr/sbin/ceph-create-keys

/usr/sbin/ceph-volume

/usr/sbin/ceph-volume-systemd

So it looks like , It should be me ceph-deploy Version and to be deployed ceph Version mismatch , Replace ceph-deploy Version is 2.0.1.

$ ceph-deploy --version

2.0.1

$ ceph-deploy disk list ceph-node1

[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf

[ceph-node1][DEBUG ] connection detected need for sudo

[ceph-node1][DEBUG ] connected to host: ceph-node1

[ceph-node1][DEBUG ] detect platform information from remote host

[ceph-node1][DEBUG ] detect machine type

[ceph-node1][DEBUG ] find the location of an executable

[ceph-node1][INFO  ] Running command: sudo fdisk -l

[ceph-node1][INFO  ] Disk /dev/nvme1n1: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors

[ceph-node1][INFO  ] Disk /dev/nvme0n1: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors

[ceph-node1][INFO  ] Disk /dev/mapper/data-ceph--data1: 107.4 GB, 107374182400 bytes, 209715200 sectors

[ceph-node1][INFO  ] Disk /dev/mapper/data-ceph--data2: 107.4 GB, 107374182400 bytes, 209715200 sectors

[ceph-node1][INFO  ] Disk /dev/mapper/data-ceph--data3: 107.4 GB, 107374182400 bytes, 209715200 sectors

[ceph-node1][INFO  ] Disk /dev/mapper/data-ceph--data4: 107.4 GB, 107374182400 bytes, 209715200 sectors

initialization node node ：

$  initialization node node , It's actually installation ceph、ceph-radosgw And some related basic components 

$ ceph-deploy install --no-adjust-repos --nogpgcheck ceph-node1 ceph-node2 ceph-node3

    - --no-adjust-repos Indicates that the local repo Transfer the file to the target machine , Because the front has been manually configured 

    - --nogpgcheck Do not check yum Of key

install osd service ：

##  In need of installation osd Execute... In a machine 

yum -y install ceph-osd ceph-common

Erase all node The node needs to be initialized to osd The data of the disk ：

##  give an example , Do the same for other disks 

$ ceph-deploy disk zap ceph-node1 /dev/nvme0n1p3 /dev/nvme0n1p4 /dev/nvme0n1p5 /dev/nvme0n1p6

[ceph-node1][WARNIN] --> Zapping: /dev/nvme0n1p3

[ceph-node1][WARNIN] Running command: /bin/dd if=/dev/zero of=/dev/nvme0n1p3 bs=1M count=10 conv=fsync

[ceph-node1][WARNIN]  stderr: 10+0 records in

[ceph-node1][WARNIN] 10+0 records out

[ceph-node1][WARNIN] 10485760 bytes (10 MB) copied

[ceph-node1][WARNIN]  stderr: , 0.0221962 s, 472 MB/s

[ceph-node1][WARNIN] --> Zapping successful for: <Partition: /dev/nvme0n1p3>

Start to create osd：

##  give an example , Do the same for other disks 

$ ceph-deploy osd create ceph-node1 --data /dev/nvme0n1p3

$ ceph-deploy osd create ceph-node1 --data /dev/nvme0n1p4

...

osd It will be numbered and named according to the creation order , The first is 0, And so on ...

Viewed osd process ：

## ceph-node1 process 

# ps -ef | grep ceph-osd

ceph       61629       1  0 17:16 ?        00:00:01 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph

ceph       62896       1  0 17:17 ?        00:00:01 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph

ceph       63569       1  0 17:18 ?        00:00:01 /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph

ceph       64519       1  0 17:18 ?        00:00:01 /usr/bin/ceph-osd -f --cluster ceph --id 3 --setuser ceph --setgroup ceph

## ceph-node2 process 

# ps -ef | grep osd

ceph       64649       1  0 17:27 ?        00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph

ceph       65423       1  0 17:27 ?        00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph

ceph       66082       1  0 17:28 ?        00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph

ceph       66701       1  0 17:28 ?        00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 7 --setuser ceph --setgroup ceph

## ceph-node3 process 

# ps -ef | grep osd

ceph       30549       1  0 11:30 ?        00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 8 --setuser ceph --setgroup ceph

ceph       31270       1  0 11:30 ?        00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 9 --setuser ceph --setgroup ceph

ceph       32220       1  1 11:31 ?        00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 10 --setuser ceph --setgroup ceph

ceph       32931       1  1 11:31 ?        00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 11 --setuser ceph --setgroup ceph

osd The number is 0-11, common 12 Block plate .

osd Service startup completed , Look again ceph State of the cluster ：

$ ceph -s

  cluster:

    id:     537175bb-51de-4cc4-9ee3-b5ba8842bff2

    health: HEALTH_OK

  services:

    mon: 1 daemons, quorum ceph-node1 (age 19h)

    mgr: ceph-node1(active, since 19h)

    osd: 12 osds: 12 up (since 99s), 12 in (since 99s)

  data:

    pools:   1 pools, 1 pgs

    objects: 2 objects, 0 B

    usage:   12 GiB used, 1.2 TiB / 1.2 TiB avail

    pgs:     1 active+clean

##  There is a default pool, this pool Is to add osd Created automatically by the system 

$ ceph osd lspools

1 device_health_metrics

$ ceph df

--- RAW STORAGE ---

CLASS  SIZE     AVAIL    USED     RAW USED  %RAW USED

ssd    1.2 TiB  1.2 TiB  9.7 MiB    12 GiB       1.00

TOTAL  1.2 TiB  1.2 TiB  9.7 MiB    12 GiB       1.00

--- POOLS ---

POOL                   ID  PGS  STORED  OBJECTS  USED  %USED  MAX AVAIL

device_health_metrics   1    1     0 B        2   0 B      0    376 GiB
thus , Basic ceph The cluster has been built successfully ,rbd The function is ready to use .
In addition, if you want to turn on the function of object storage and file system , We need to deploy rgw、mds and cephfs. At this time mon、mgr And other components have not achieved high availability , Start with the horizontal expansion of these important components .

4. Expand ceph-mon node

（1） Target machine installation ceph-mon Components 

# yum -y install ceph-mon ceph-common

（2） add to mon machine 

$ ceph-deploy mon add ceph-node2

$ ceph-deploy mon add ceph-node3

（3） Check cluster status 

$ ceph -s

  cluster:

    id:     537175bb-51de-4cc4-9ee3-b5ba8842bff2

    health: HEALTH_OK

  services:

    mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3 (age 10m)

    mgr: ceph-node1(active, since 25h)

    osd: 12 osds: 12 up (since 6h), 12 in (since 6h)

  data:

    pools:   1 pools, 1 pgs

    objects: 2 objects, 0 B

    usage:   12 GiB used, 1.2 TiB / 1.2 TiB avail

    pgs:     1 active+clean

##  You can use this command to view mon Details and status of 

$ ceph quorum_status --format json-pretty

Now? mon It has become 3 Nodes

5. Expand ceph-mgr node

（1） Target machine installation ceph-mgr Components 

# yum -y install ceph-mgr ceph-common

（2） add to mgr machine 

$ ceph-deploy mgr create ceph-node2

（3） Verify cluster status 

$ ceph -s

  cluster:

    id:     537175bb-51de-4cc4-9ee3-b5ba8842bff2

    health: HEALTH_OK

  services:

    mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3 (age 24m)

    mgr: ceph-node1(active, since 25h), standbys: ceph-node2

    osd: 12 osds: 12 up (since 6h), 12 in (since 6h)

  data:

    pools:   1 pools, 1 pgs

    objects: 2 objects, 0 B

    usage:   12 GiB used, 1.2 TiB / 1.2 TiB avail

    pgs:     1 active+clean

mgr The high availability of is in the form of active and standby , and mon It is the main form of cluster selection .

6. increase mds( Metadata services )、cephfs Provide file system functions

mds The service is a separate storage service , To function properly, you must specify two storage pools separately , One for storing cephfs Metadata , The other is used to store data data , The metadata pool mainly stores metadata such as the size and name of the file directory ,data Pools are used to hold actual files, etc .

（1） install mds Installation package 

$ ceph-deploy mds create ceph-node1

$ ceph-deploy mds create ceph-node2

Check ceph state 

$ ceph -s

  cluster:

    id:     537175bb-51de-4cc4-9ee3-b5ba8842bff2

    health: HEALTH_OK

  services:

    mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3 (age 18h)

    mgr: ceph-node1(active, since 43h), standbys: ceph-node2

    mds:  2 up:standby

    osd: 12 osds: 12 up (since 24h), 12 in (since 24h)

  task status:

  data:

    pools:   1 pools, 1 pgs

    objects: 3 objects, 0 B

    usage:   12 GiB used, 1.2 TiB / 1.2 TiB avail

    pgs:     1 active+clean

at present mds Has joined the cluster , But they are all in standby( spare ) state , because mds Storage pools for metadata and data must be specified separately ：

##  First create the metadata pool and data pool , In the back is the data pg and pgp The number of 

$ ceph osd pool create cephfs-metedata 32 32

pool 'cephfs-metedata' created

$ ceph osd pool create cephfs-data 64 64

pool 'cephfs-data' created

$ ceph osd lspools

1 device_health_metrics

2 cephfs-metedata

3 cephfs-data

（2） establish cephfs

$ ceph fs new mycephfs cephfs-metedata cephfs-data

new fs with metadata pool 2 and data pool 3

##  Create Syntax 

ceph fs new <fs_name> <metadata> <data> [--force] [--allow-dangerous-metadata-overlay]

Check again ceph State of the cluster 

$ ceph -s

  cluster:

    id:     537175bb-51de-4cc4-9ee3-b5ba8842bff2

    health: HEALTH_OK

  services:

    mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3 (age 18h)

    mgr: ceph-node1(active, since 43h), standbys: ceph-node2

    mds: mycephfs:1 {0=ceph-node2=up:active} 1 up:standby

    osd: 12 osds: 12 up (since 24h), 12 in (since 24h)

  task status:

  data:

    pools:   3 pools, 97 pgs

    objects: 25 objects, 2.2 KiB

    usage:   12 GiB used, 1.2 TiB / 1.2 TiB avail

    pgs:     97 active+clean

$ ceph mds stat

mycephfs:1 {0=ceph-node2=up:active} 1 up:standby

cephfs The function is set up

6. increase rgw Components , Provide object storage function

rgw What to offer REST Interface , Client pass http Interact with it , Complete management operations such as data addition, deletion, modification and query . Usually there will be more than one rgw Guaranteed high availability ,rgw A load balancer is hung at the front for distribution .

（1） install rgw Components 

# yum -y install ceph-radosgw

（2） Deploy rgw

$ ceph-deploy --overwrite-conf rgw create ceph-node1

$ ceph-deploy --overwrite-conf rgw create ceph-node2

[ceph-node1][INFO  ] Running command: sudo systemctl start [email protected]

[ceph-node1][INFO  ] Running command: sudo systemctl enable ceph.target

[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-node1 and default port 7480

Check ceph State of the cluster 

$ ceph -s

  cluster:

    id:     537175bb-51de-4cc4-9ee3-b5ba8842bff2

    health: HEALTH_OK

  services:

    mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3 (age 24h)

    mgr: ceph-node1(active, since 2d), standbys: ceph-node2

    mds: mycephfs:1 {0=ceph-node2=up:active} 1 up:standby

    osd: 12 osds: 12 up (since 30h), 12 in (since 30h)

    rgw: 2 daemons active (ceph-node1, ceph-node2)

  task status:

  data:

    pools:   7 pools, 201 pgs

    objects: 212 objects, 6.9 KiB

    usage:   12 GiB used, 1.2 TiB / 1.2 TiB avail

    pgs:     201 active+clean

  io:

    client:   35 KiB/s rd, 0 B/s wr, 34 op/s rd, 23 op/s wr

follow-up

  1. Because of initialization mon Node time , Only one... Is initialized , So at present ceph.conf There is still only one mon_host, As a result, high availability is not achieved , You need to retrieve the cluster information , rewrite ceph.conf file ：
$ ceph-deploy --overwrite-conf config push ceph-node1 ceph-node2 ceph-node3

$ cat /etc/ceph/ceph.conf

[global]

fsid = 537175bb-51de-4cc4-9ee3-b5ba8842bff2

public_network = 10.0.0.0/8

cluster_network = 10.0.0.0/8

mon_initial_members = ceph-node1

mon_host = 10.153.204.13,10.130.22.45,10.153.204.28

auth_cluster_required = cephx

auth_service_required = cephx

auth_client_required = cephx
  1. ceph Cluster startup and shutdown sequence

Before restarting , Set it up in advance Ceph Don't cluster OSD Marked as out, avoid node The node is kicked out after shutting down the service Ceph Outside the cluster , Once kicked out ,ceph It will automatically balance the data ：

##  Set up noout

$ ceph osd set noout

noout is set

##  Cancel noout

$ ceph osd unset noout

noout is unset

Stop sequence ：

  1. Set before shutting down the service noout;
  2. Close the storage client and stop reading and writing data ;
  3. If used RGW, Then close RGW service ;
  4. close CephFS Metadata services ;
  5. close Ceph OSD service ;
  6. close Ceph Manager service ;
  7. close Ceph Monitor service ;

Startup sequence ：

  1. start-up Ceph Monitor service ;
  2. start-up Ceph Manager service ;
  3. start-up Ceph OSD service ;
  4. start-up CephFS Metadata services ;
  5. start-up RGW service ;
  6. Start storage client ;
  7. Final cancellation noout Set up ;

summary

Only this and nothing more , A complete high availability ceph The cluster has been set up , Now it's just building , The next article details how to use rbd、cephfs、 And object storage .

ceph Common operation and maintenance problems FAQ

1. osd Offline process

（1） If osd The machine is still in normal operation , Not deleted abnormally osd, First of all, we should osd The weight of is set to 0, Wait for this osd Migrate all data out of and no longer accept new data .

$ ceph osd crush reweight osd.8 0

reweighted item id 8 name 'osd.8' to 0 in crush map

If the amount of data is too large , It's best to adjust the weight value slowly ,0.7->0.4>0.1>0, In order to make sure ceph The maximum stability of the cluster .

（2） stop it osd process 

# systemctl stop [email protected]

stop it osd The process of , This is to inform the cluster of this osd The process is gone , No more services , Because it has no weight , It won't affect the overall distribution , No data migration .

（3） Mark the node status as out

$ ceph osd out osd.8

This step is to tell mon, This node is out of service , Need to be in other osd Data recovery on , But I've done it before reweight, So there will be no data migration .

（4） from crush Remove node from table 

$ ceph osd crush remove osd.8

removed item id 8 name 'osd.8' from crush map

from crush Deleting in tells the cluster that this node should be completely eliminated , Let the cluster crush Do a recalculation , Because I've done reweight, therefore crush weight It has also become 0.

（5） Delete osd node 

$ ceph osd rm osd.8

removed osd.8

Delete the record of this node from the cluster

（6） Remove node authentication 

$ ceph auth del osd.8

updated

If this certification is not deleted ,osd The number of will not be released .

（7） Finally, check the cluster status 

$ ceph -s

  cluster:

    id:     537175bb-51de-4cc4-9ee3-b5ba8842bff2

    health: HEALTH_WARN

            Degraded data redundancy: 152/813 objects degraded (18.696%), 43 pgs degraded, 141 pgs undersized

  services:

    mon: 2 daemons, quorum ceph-node1,ceph-node2 (age 111s)

    mgr: ceph-node1(active, since 11d), standbys: ceph-node2

    mds: mycephfs:1 {0=ceph-node2=up:active} 1 up:standby

    osd: 8 osds: 8 up (since 3d), 8 in (since 3d); 124 remapped pgs

    rgw: 2 daemons active (ceph-node1, ceph-node2)

  task status:

  data:

    pools:   8 pools, 265 pgs

    objects: 271 objects, 14 MiB

    usage:   8.1 GiB used, 792 GiB / 800 GiB avail

    pgs:     152/813 objects degraded (18.696%)

             114/813 objects misplaced (14.022%)

             111 active+clean+remapped

             98  active+undersized

             43  active+undersized+degraded

             13  active+clean

because ceph-node3 nodes osd All services have been eliminated , So now osd Node left 8 individual , Because of this 8 individual osd All concentrated in two hosts , Many of them are not active+clean Of pg, Once there is a new machine osd Node online ,pgp Will automatically distribute .

2. mon Offline process

（1） see mon state 

$ ceph mon stat

e3: 3 mons at {ceph-node2=[v2:10.130.22.45:3300/0,v1:10.130.22.45:6789/0],ceph-node1=[v2:10.153.204.13:3300/0,v1:10.153.204.13:6789/0],ceph-node3=[v2:10.153.204.28:3300/0,v1:10.153.204.28:6789/0]}, election epoch 48, leader 0 ceph-node1, quorum 0,1 ceph-node1,ceph-node2

（2） stop it mon

systemctl stop [email protected]

（3） Removed from the mon

$ ceph mon remove ceph-node3

removing mon.ceph-node3 at [v2:10.153.204.28:3300/0,v1:10.153.204.28:6789/0], there will be 2 monitors

（4） stay ceph.conf Delete... From the configuration file mon.host Field 

$ cat ceph.conf

[global]

fsid = 537175bb-51de-4cc4-9ee3-b5ba8842bff2

public_network = 10.0.0.0/8

cluster_network = 10.0.0.0/8

mon_initial_members = ceph-node1

mon_host = 10.153.204.13,10.130.22.45

auth_cluster_required = cephx

auth_service_required = cephx

auth_client_required = cephx

（5） Check the cluster status again 

$ ceph -s

  cluster:

    id:     537175bb-51de-4cc4-9ee3-b5ba8842bff2

    health: HEALTH_WARN

            Degraded data redundancy: 152/813 objects degraded (18.696%), 43 pgs degraded, 141 pgs undersized

  services:

    mon: 2 daemons, quorum ceph-node1,ceph-node2 (age 111s)

    mgr: ceph-node1(active, since 11d), standbys: ceph-node2

    mds: mycephfs:1 {0=ceph-node2=up:active} 1 up:standby

    osd: 8 osds: 8 up (since 3d), 8 in (since 3d); 124 remapped pgs

    rgw: 2 daemons active (ceph-node1, ceph-node2)

  task status:

  data:

    pools:   8 pools, 265 pgs

    objects: 271 objects, 14 MiB

    usage:   8.1 GiB used, 792 GiB / 800 GiB avail

    pgs:     152/813 objects degraded (18.696%)

             114/813 objects misplaced (14.022%)

             111 active+clean+remapped

             98  active+undersized

             43  active+undersized+degraded

             13  active+clean

ceph-deploy Offline deployment ceph Cluster and error reporting FAQ More articles about

  1. 【docker】【redis】2.docker Set up redis colony ---Redis Cluster Deploy 【 The cluster service 】【 The solution is docker in redis After starting , Status as Restarting, Log error ：Configured to not listen anywhere, exiting. problem 】【Waiting for the cluster to join... problem 】

    Reference address :https://www.cnblogs.com/zhoujinyi/p/6477133.html https://www.cnblogs.com/cxbhakim/p/9151720.htm ...

  2. perform redis command redis-trib.rb Check the cluster information and report an error cannot load such file -- redis (LoadError)

    Problem description : In execution redis-trib.rb Command to view the cluster status , Report errors : [[email protected] ~]$ redis-trib.rb check Traceback (most recent ...

  3. Redis Cluster creation reports an error

    Redis Cluster environment :och163/och164/och165 An error occurred while executing the following script : ./src/redis-trib.rb create 10.1.253.163: 10.1.253.164: 10 ...

  4. 【 Problem set 】redis colony set Report errors (error) MOVED 11469 192.168.181.201:7002

    Cluster mode is not started ( That is, the lack of that "-c"): redis-cli -c -h yourhost -p yourpost

  5. elasticsearch Cluster setup error ： not enough master nodes discovered during pinging

    Use one by yourself Alibaba cloud The server build ES In clusters , Always report the above questions . And two ES All the services reported the same problem . Your configuration file is as follows : es service 1 The configuration file cluster.name: elasticsearch ...

  6. IMPI Python Cluster operation error :

    Intel MPI Environmental utilization hostfile An error is reported under multi host operation HYDU_process_mfile_token (../../utils/args/args.c:523): token slots no ...

  7. elasticsearch Configure clusters +elk Error report summary

    To configure ELK When , I usually encounter the following error reporting situations , It is arranged as follows ( Ongoing update ): elasticsearch Boot failure # systemctl start elasticsearch Job for elast ...

  8. HBase 2.1.3 colony web Report errors InvalidProtocolBufferException resolvent

    Set up well HBase After cluster , Various background processes are normal , Construction manual reference : Hbase 2.1.3 Cluster building manual https://www.cndba.cn/dave/article/3322 But by web visit , But report ...

  9. rabbimq Cluster setup error ：Error: unable TO perform an operation ON node &#39;[email protected]&#39;. Please see diagnostics information AND suggestions below.

    In the building rabbitmq In clusters , When adding a memory node , Throw an exception :Error: unable TO perform an operation ON node '[email protected]'. Please s ...

  10. Unable to connect to the Internet Kafka colony （ Report errors ：NoBrokersAvailable）

    Local Consumer and Producer Cannot use remote Kafka Server Solutions : Modify each server separately Kafka The configuration file server.properties, stay #listeners=PLAINTEXT:/ ...

Random recommendation

  1. SQL Multi table join query

    SQL Multi table join query Multi table join queries have two specifications ,SQL92 and SQL99 standard . SQL92 The specification supports the following multi table join queries : (1) Equivalent connection : (2) Non equivalent connection : (3) External connection : (4) The generalized Cartesian product : SQL9 ...

  2. ThinkPHP Use PHPmailer send out Email mail

    Let's introduce thinkphp How to use phpmailer Send E-mail , Use this mail sending class , After the parameters are configured , You can send an email in one sentence . Only suitable for thinkphp frame . First step , Download the class library take Mail.class.php Copied to the ...

  3. devexpress in gridcontrol Some style changes

    change footer For flattening effect Whole footer Background color CustomDrawFootere.Appearance.BackColor = Color.Transparent; e.Appearance.D ...

  4. Java in setCharAt() Methods to introduce

    -- Reprinted from the Internet , Memo This is a StringBuffer Class : Mainly for replacement , Method has two parameters setCharAt(int index,Char ch), The first parameter is the position of the substitution Index from 0 Start ...

  5. OpenID Connect：OAuth 2.0 The simple identity layer above the protocol

    OpenID Connect What is it? ?OpenID Connect( The current version is 1.0) yes OAuth 2.0 agreement ( Please refer to my article :OAuth 2.0 / RCF6749 Interpretation of the agreement ) The simple identity layer above , use A ...

  6. require.js Basic usage

    1.require.js Loading Use require.js The first step , First go to the official website to download the latest version . After downloading , Suppose you put it in js Under subdirectories , You can load . 1 <script src="js/ ...

  7. P4027 [NOI2007] Currency conversion （ Slope optimization dp+cdq Divide and conquer ）

    P4027 [NOI2007] Currency conversion obviously , If you want to buy a coupon one day , Must have spent all the money . Otherwise it is not optimal ( What are you saving for ) We set up $f[j]$ For the first time $j$ How much money does Tianshi users have at most set up $w$ I bought it for my money $B$ Number of bonds $ ...

  8. The export function __declspec(dllexport)

    generally speaking , There are two functions defined in the dynamic link library : The export function (export function) And internal functions (internal function).  The export function can be called by other modules , Internal functions are defining their DLL Program internal use . ...

  9. error C4430: error 2141

    c:\evan\workspace\1\1\netwowkippack.h(50) : error C2146: Grammar mistakes : The lack of “;”( In the identifier “nSourPort” In front of ) c:\evan\wo ...

  10. 《mysql Technology insider InnoDB Storage engine （ The second edition ）》 Reading notes

    One .mysql framework mysql Is a single process multithreading architecture of the database . Two . Storage engine InnoDB: Support transactions Row lock No lock for reading operation 4 Kind of isolation level , The default is repeatable The adaptive hash Indexes The storage of each table is ...