Introduction, deployment and application of moosefs

mooseFS brief introduction

MooseFS The official launch is in 2008-05-30, To 2009-10-12 until , The latest version is 1.5.12.

mooseFS Is a network Distributed file system . It spreads data across multiple servers The server On , But for users , What you see is just a source .MFS Also like other classes unix The file system is the same , Contains a hierarchy （ Directory tree ）, Storing file attributes （ jurisdiction , Last access and modification time ）, You can create special files （ Block device , Character device , The Conduit , Socket ）, A symbolic link , Hard links .

mooseFS The composition of the system

1.master（ Metadata The server ）

master Responsible for managing data throughout the system . Is the maintainer of the whole system . But it has a weakness master only one ！ That is to say if master Broken , The whole system will stop working ！ But this is not a fatal weakness , Because after we backup the data , recovery master It's a simple thing .

2.chunkserver（ Data storage server ）

chunkserver yes mfs The data storer in the system . The real user data is divided into chunk, And distribute it to each chunkserver On . This ensures the security of the data .

3.client

Those who use mfs File system machines can be called client.client yes mfs Users of the system . When client hold mfs After the file system is mounted to the local computer , It can be used like a normal disk partition , To use mfs.

Test environment

This article only introduces moosefs Deployment and application of , For experimental purposes , Performance factors are not taken into account , Therefore, all servers use virtual machines to implement . In the future, I have the opportunity to be a separate physical server , Then test its performance .

Topology

remarks ：

（1） chunkserver1-4 A virtual disk has been added sda（ The disk space must be greater than 1G, I set up here 4G）, Attached to the system /data Next . All servers are upgraded to the kernel 2.6.18-164 This edition , Why upgrade the kernel , Here's how .

（2） All servers have a full version of mfs Components

Installation and configuration

install mfs The premise of

because mfs The client program of is loading mfs The disk system command is to use fuse Compiling , So as long as you want to mount mfs Server for , The necessary prerequisite is to install fuse, Compile like this mfs Can pass smoothly . Another point needs attention ：linux 2.6.18-164.e15 Version of linux The kernel already has fuse modular . But before this version linux The kernel does not contain this module . another fuse from 2.8.0-pre1 Version of the source code package removed fuse Source code of system module , The reason is mentioned in the previous point . In this way, we are compiling and installing the latest version of fuse(2.8 Above version ) And in use linux Kernel version is lower than 2.6.18-164.e15 edition , Then... Is not included in the system fuse Modular .

solution ：

1、 Upgrade the system kernel to 2.6.18-164 edition

Use directly for convenience yum Upgrade the system kernel .yum install kernel

After successful installation, restart the system

2、 Use fuse 2.7x Version compilation installation

The version fuse It contains linux The kernel needs fuse modular , Specify... When configuring compilation options –enable-kernel-module Options ,make Will compile the corresponding fuse modular ,make intall Will fuse.ko Copied to the /lib/modules/`uname -r`/kernel/fs/fuse/ Under the table of contents Use after installation

1. # modprobe -l | grep fuse
2. /lib/modules/2.6.18-128.el5xen/kernel/fs/fuse/fuse.ko

Check whether the installation is normal

3、 utilize yum Install the current kernel version fuse modular

1. yum install -y dkms-fuse dkms

install fuse

If you only compile the metadata server or data storage server, there is no need to install fuse Of . Only mfsmount need fuse Support （ Compile time requires fuse Development package for , Use mfsmount To mount, you need fuse.ko System module ）. You can use the source code or yum Two ways to install fuse

1、 Source code installation

1. # wget http://ncu.dl.sourceforge.net/project/fuse/fuse-2.X/2.8.1/fuse-2.8.1.tar.gz
2. # tar -xvzf fuse-2.8.1.tar.gz
3. # cd fuse-2.8.1
4. # ./configure –prefix=/usr/ –libdir=/usr/lib64
5. # make && make install

Because my system is 64 position , So compiling fuse When will lib Directory set as /usr/lib64. In this way, compile and install below mfs When you can't find it fuse Of lib The document is wrong .

2、yum install

1. # yum install -y fuse fuse-devel

install mfs

1. # useradd mfs -s /sbin/nologin
2. # ./configure –prefix=/usr/local/mfs –with-default-user=mfs –with-default-group=mfs –enable-mfsmount
3. # make && make install

View the directory structure after installation

1. # ll /usr/local/mfs/
2. total 20
3. drwxr-xr-x 2 root root 4096 Oct 14 15:14 bin
4. drwxr-xr-x 2 root root 4096 Oct 14 12:13 etc
5. drwxr-xr-x 2 root root 4096 Oct 14 15:14 sbin
6. drwxr-xr-x 3 root root 4096 Oct 14 12:13 share
7. drwxr-xr-x 3 root root 4096 Oct 14 12:13 var

bin – Client tools

etc – metadata server , The configuration files of the data storage server are placed in this directory

sbin – Metadata server-side program mfsmaster、 Data storage server-side service program mfschunkserver

share – file

var – Metadata catalog （ You can customize to other directories in the configuration file ）

To configure master（ metadata server ）

IP:192.168.108.108

1. [[email protected]~]#vi/usr/local/mfs/etc/mfsmaster.cfg
2. #WORKING_USER=mfs
3. #WORKING_GROUP=mfs
4. #LOCK_FILE=/var/run/mfs/mfsmaster.pid
5. #DATA_PATH=/usr/local/mfs/var/mfs
6. #SYSLOG_IDENT=mfsmaster
7. #BACK_LOGS=50
8. #REPLICATIONS_DELAY_INIT=300
9. #REPLICATIONS_DELAY_DISCONNECT=3600
10. MATOCS_LISTEN_HOST=192.168.108.108
11. #MATOCS_LISTEN_PORT=9420
12. #MATOCU_LISTEN_HOST=*
13. #MATOCU_LISTEN_PORT=9421
14. #CHUNKS_LOOP_TIME=300
15. #CHUNKS_DEL_LIMIT=100
16. #CHUNKS_REP_LIMIT=15

notes ： All the settings noted in this configuration file are the default configuration . Here I only changed MATOCS_LISTEN_HOST Value , That is, change it to the native ip Address ：192.168.108.108 . It can be modified if necessary DATA_PATH The metadata directory is stored in other partitions or disks . Other parameters are simple and can be adjusted as needed .

master Will open 9420 Port waiting mfschunkserver Connect

start-up mfsmaster

1. [[email protected]~]#/usr/local/mfs/sbin/mfsmaster
2. [[email protected]~]#ps-ef|grepmfsmaster|grep-vgrep
3. mfs101321017:37?00:00:00/usr/local/mfs/sbin/mfsmaster
4. [[email protected]~]#netstat-tulnp|grepmfsmaster
5. tcp00192.168.108.108:94200.0.0.0:*LISTEN10132/mfsmaster
6. tcp000.0.0.0:94210.0.0.0:*LISTEN10132/mfsmaster

Check the system log

1. [[email protected]~]#tail-f/var/log/messages
2. Oct1417:37:35mastermfsmaster:config:usingdefaultvalueforoption‘SYSLOG_IDENT’–‘mfsmaster’
3. Oct1417:37:35mastermfsmaster[10130]:config:usingdefaultvalueforoption‘WORKING_USER’–‘mfs’
4. Oct1417:37:35mastermfsmaster[10130]:config:usingdefaultvalueforoption‘WORKING_GROUP’–‘mfs’
5. …
6. Oct1417:37:35mastermfsmaster[10132]:config:usingdefaultvalueforoption‘CHUNKS_LOOP_TIME’–’300′

# Many of the above logs are omitted , It is mainly the process of program reading configuration .

# The following sections are about checking the metadata and checking the data storage server （ Every time 1 Check every minute ）.

# Because I haven't started here yet chunkservers So in chunkservers status: The display result of is empty .total: usedspace: The result of is also empty .

1. Oct1417:38:00mastermfsmaster[10132]:inodes:45
2. Oct1417:38:00mastermfsmaster[10132]:dirnodes:3
3. Oct1417:38:00mastermfsmaster[10132]:filenodes:42
4. Oct1417:38:00mastermfsmaster[10132]:chunks:14
5. Oct1417:38:00mastermfsmaster[10132]:chunkstodelete:0
6. Oct1417:38:00mastermfsmaster[10132]:chunkserversstatus:
7. Oct1417:38:00mastermfsmaster[10132]:total:usedspace:0(0GB),totalspace:0(0GB),usage:0.00%

Set the service to start with the system

1. [[email protected]~]#echo“/usr/local/mfs/sbin/mfsmaster”>>/etc/rc.local

To configure chunkserver（ Data storage server ）

IP:192.168.108.161~164

1. [[email protected]~]#vi/usr/local/mfs/etc/mfschunkserver.cfg
2. #WORKING_USER=mfs
3. #WORKING_GROUP=mfs
4. #DATA_PATH=/usr/local/mfs/var/mfs
5. #LOCK_FILE=/var/run/mfs/mfschunkserver.pid
6. #SYSLOG_IDENT=mfschunkserver
7. #BACK_LOGS=50
8. #MASTER_RECONNECTION_DELAY=30
9. MASTER_HOST=192.168.108.108
10. #MASTER_PORT=9420
11. #MASTER_TIMEOUT=60
12. #CSSERV_LISTEN_HOST=*
13. #CSSERV_LISTEN_PORT=9422
14. #CSSERV_TIMEOUT=60
15. #CSTOCS_TIMEOUT=60
16. #HDD_CONF_FILENAME=/usr/local/mfs/etc/mfshdd.cfg

Configure storage partitions

1. [[email protected]~]#vi/usr/local/mfs/etc/mfshdd.cfg
2. Delete
3. /mnt/hd1
4. /mnt/hd2
5. /mnt/hd3
6. /mnt/hd4

Add separate partitions

/data

Change the partition owner to mfs

1. [[email protected]~]#chownmfs.mfs/data

notes ：mfschunkserver The main configuration of the server is simple , There are no special requirements, just need to modify MASTER_HOST The address of . Select an independent disk partition for storage partition configuration （ Partition must be greater than 1G）.

start-up mfschunkserver

1. [[email protected]~]#/usr/local/mfs/sbin/mfschunkserver
2. [[email protected]~]#netstat-an|grep9420
3. tcp00192.168.108.161:15099192.168.108.108:9420ESTABLISHED

View the system log at the same time

1. Oct1417:53:45vm_web_1mfschunkserver[1992]:connecting…
2. Oct1417:53:45vm_web_1mfschunkserver[1992]:connectedtoMaster

The explanation has been made with master Server successfully connected

Check it again master Log view of

1. Oct1417:59:00experimentmfsmaster[10132]:server1(192.168.108.161):usedspace:560484352(0GB),totalspace:4226125824(3GB),usage:13.26%
2. Oct1417:59:00experimentmfsmaster[10132]:total:usedspace:560484352(0GB),totalspace:4226125824(3GB),usage:13.26%

It also means that there is one chunkserver Connect with yourself , And gives the use space and the remaining disk space

remainder 3 platform mfschunkserver Use the same method to configure and start mfschunkserver Program , Then check master system log ：

1. Oct1615:27:00experimentmfsmaster[10132]:inodes:9
2. Oct1615:27:00experimentmfsmaster[10132]:dirnodes:1
3. Oct1615:27:00experimentmfsmaster[10132]:filenodes:8
4. Oct1615:27:00experimentmfsmaster[10132]:chunks:18
5. Oct1615:27:00experimentmfsmaster[10132]:chunkstodelete:0
6. Oct1615:27:00experimentmfsmaster[10132]:chunkserversstatus:
7. Oct1615:27:00experimentmfsmaster[10132]:server1(192.168.108.162):usedspace:924282880(0GB),totalspace:4226125824(3GB),usage:21.87%
8. Oct1615:27:00experimentmfsmaster[10132]:server2(192.168.108.164):usedspace:924282880(0GB),totalspace:4226125824(3GB),usage:21.87%
9. Oct1615:27:00experimentmfsmaster[10132]:server3(192.168.108.163):usedspace:924282880(0GB),totalspace:4226125824(3GB),usage:21.87%
10. Oct1615:27:00experimentmfsmaster[10132]:server4(192.168.108.161):usedspace:924028928(0GB),totalspace:4226125824(3GB),usage:21.86%
11. Oct1615:27:00experimentmfsmaster[10132]:total:usedspace:3696877568(3GB),totalspace:16904503296(15GB),usage:21.87%

You can see here that there is 4 platform chunkserver Already connected to master

Client mount and tool use

192.168.108.109

mount MFS

mfsmount

1. [[email protected]~]#mkdir/mnt/mfs
2. [[email protected]~]#mfsmount-h192.168.108.108
3. *mfsmount By default, partitions are loaded into /mnt/mfs Under the table of contents . If you want to load into another directory, please use -w Parameters .
4. mfsmount–help
5. usage:/usr/local/mfs/bin/mfsmount[-r][-m][-c][-v0..2][-hmasterhost][-pmasterport][-lpath][-wmountpoint]
6. r:readonlymode
7. m:mountmetadata
8. c:allowusingcache
9. v:verboselevel
10. defaults:
11. h:mfsmaster
12. p:9421
13. l:/
14. w:/mnt/mfs

modify MFS File deletion delay time

1. mfsrsettrashtime
2. [[email protected]~]#mfsrsettrashtime100/mnt/mfs/
3. /mnt/mfs/:
4. inodeswithtrashtimechanged:1(1)
5. inodeswithtrashtimenotchanged:0(0)
6. inodeswithpermissiondenied:0(0)

see MFS File deletion delay time

1. mfsrgettrashtime
2. #mfsrgettrashtime/mnt/mfs/
3. /mnt/mfs/:
4. directorieswithtrashtime100:1(1)
5. mfssettrashtime
6. [[email protected]~]#mfssettrashtime60/mnt/mfs/
7. /mnt/mfs/:60
8. mfsgettrashtime
9. [[email protected]~]#mfsgettrashtime/mnt/mfs/
10. /mnt/mfs/:60
11. Set the number of copies saved
12. mfssetgoal
13. [[email protected]~]#mfssetgoal4/mnt/mfs/
14. /mnt/mfs/:4
15. View the number of saved files
16. mfsgetgoal
17. [[email protected]~]#mfsgetgoal/mnt/mfs/
18. /mnt/mfs/:4
19. mfsrsetgoal
20. [[email protected]~]#mfsrsetgoal3/mnt/mfs/
21. /mnt/mfs/:
22. inodeswithgoalchanged:1(1)
23. inodeswithgoalnotchanged:0(0)
24. inodeswithpermissiondenied:0(0)
25. mfsrgetgoal
26. [[email protected]~]#mfsrgetgoal/mnt/mfs/
27. /mnt/mfs/:
28. directorieswithgoal3:1(1)

file （ Folder ） View of （ Check ） command

1. Mr Into 1 individual 10M The file of
2. [[email protected]~]#ddif=/dev/zeroof=filebs=1Mcount=10
3. [[email protected]~]#cpfile/mnt/mfs
4. File information view
5. mfsfileinfo
6. [[email protected]~]#mfsfileinfo/mnt/mfs/file
7. /mnt/mfs/file:
8. chunk0:0000000000000001_00000001/(id:1ver:1)
9. copy1:192.168.108.161:9422
10. copy2:192.168.108.162:9422
11. copy3:192.168.108.163:9422
12. Document inspection
13. mfscheckfile
14. [[email protected]~]#mfscheckfile/mnt/mfs/file
15. /mnt/mfs/file:
16. 3copies:1chunks

View directory information

mfsdirinfo

1. [[email protected]~]#rm/mnt/mfs/file
2. [[email protected]~]#mkdir/mnt/mfs/newdir
3. [[email protected]~]#cpfile/mnt/mfs/newdir
4. [[email protected]~]#mfsdirinfo/mnt/mfs/newdir
5. /mnt/mfs/newdir/:
6. inodes:2(2)//inode Number ,1 A catalog ,1 File
7. directories:1(1)//1 A catalog
8. files:1(1)//1 File
9. goodfiles:1(1)// Number of normal files
10. undergoalfiles:0(0)
11. missingfiles:0(0)
12. chunks:1(1)
13. goodchunks:1(1)
14. undergoalchunks:0(0)
15. missingchunks:0(0)
16. length:10M(10485760)
17. size:10M(10490880)// Here is the file size
18. hddusage:30M(31472640)// Because I set the number of copies of documents to 3, So here is 3*100M
19. [[email protected]~]#cp/data/filenewdir/file_2
20. /mnt/mfs/newdir/:
21. inodes:3(3)
22. directories:1(1)
23. files:2(2)
24. goodfiles:2(2)
25. undergoalfiles:0(0)
26. missingfiles:0(0)
27. chunks:2(2)
28. goodchunks:2(2)
29. undergoalchunks:0(0)
30. missingchunks:0(0)
31. length:20M(20971520)
32. size:20M(20981760)
33. hddusage:60M(62945280)

Create a file snapshot

mfssnapshot

1. [[email protected]/mnt/mfs/newdir]#mfssnapshotmysnapshotfile
2. [[email protected]/mnt/mfs/newdir]#ll
3. total30720
4. -rw-r–r–1rootroot10485760Oct1616:43file
5. -rw-r–r–1rootroot10485760Oct1616:44file_2
6. -rw-r–r–1rootroot10485760Oct1616:58mysnapshot
7. [[email protected]/mnt/mfs/newdir]#mfsfileinfomysnapshot
8. mysnapshot:
9. chunk0:0000000000000003_00000001/(id:3ver:1)
10. copy1:192.168.108.161:9422
11. copy2:192.168.108.162:9422
12. copy3:192.168.108.163:9422
13. [[email protected]/mnt/mfs/newdir]#mfsfileinfofile
14. file:
15. chunk0:0000000000000003_00000001/(id:3ver:1)
16. copy1:192.168.108.161:9422
17. copy2:192.168.108.162:9422
18. copy3:192.168.108.163:9422

adopt mfsfileinfo Command to view the created file snapshot , It takes only one inode, It doesn't take up disk space , It's like ln Command to create a hard link similar to . but mfsdirinfo There seems to be something wrong with the display of ：

1. [[email protected]/mnt/mfs/newdir]#mfsdirinfo/mnt/mfs/newdir/
2. /mnt/mfs/newdir/:
3. inodes:4(4)
4. directories:1(1)
5. files:3(3)
6. goodfiles:3(3)
7. undergoalfiles:0(0)
8. missingfiles:0(0)
9. chunks:3(3)
10. goodchunks:3(3)
11. undergoalchunks:0(0)
12. missingchunks:0(0)
13. length:30M(31457280)
14. size:30M(31472640)
15. hddusage:90M(94417920)

utilize mfsdirinfo Looking at the directory information, it is found that the newly created snapshot file also occupies the same disk space as the original file . But is this really the case ？

see master The system log of

1. Oct1617:14:00experimentmfsmaster[24445]:inodes:5
2. Oct1617:14:00experimentmfsmaster[24445]:dirnodes:2
3. Oct1617:14:00experimentmfsmaster[24445]:filenodes:3
4. Oct1617:14:00experimentmfsmaster[24445]:chunks:2
5. Oct1617:14:00experimentmfsmaster[24445]:chunkstodelete:0

Through the log chunks: 2 Do you know 2 individual chunks just mfsfineinfo Check the file to know file File occupancy 1 individual chunks that 2 individual chunks Description yes 2 File . Look up again filenodes: 3 Documentation inode The number of 3 Description yes 3 File , That's exactly what it is. 2 File +1 A snapshot file .

Look at each mfschunkserver Of mfs file

By looking at master Log and view the data on each data storage server mfs file , It verifies that we just used mfsfileinfo The results ！ File snapshot , It takes only one inode, It doesn't take up disk space , It's like ln Command to create a hard link similar to .