Mongodb - replica set and sharding

MongoDB Copy （ Replica set ）

Introduce

  MongoDB Replication is the process of synchronizing data on multiple servers .

   Replication provides redundant backup of data , And store copies of data on multiple servers , Improved data availability , And can guarantee the security of data .

   Replication also allows you to recover data from hardware failures and service outages .

What is replication

  1 Secure data
  2 High availability of data (24*7)
  3 disaster recovery
  4 No need to shut down for maintenance （ Backup , Rebuild index , Compress ）
  5 Distributed read data
  6MongoDB The principle of replication
  7mongodb Replication of requires at least two nodes . One of them is the master node , Responsible for handling client requests , The rest are from nodes , Responsible for copying the data on the primary node .

mongodb Common collocation methods of each node

   mongodb The common collocation mode of each node is ： A master from 、 One master, many followers

   All operations recorded on the master node oplog, Periodically polling the master from the node for these operations , Then perform these operations on your own copy of the data , So as to ensure that the data of the slave node is consistent with that of the master node .

  MongoDB Copy the structure diagram as follows ：

Title replica set features

  1、N Cluster of nodes
  2、 Any node can be the master node
  3、 All writes are on the primary node
  4、 Automatic failover
  5、 Automatic recovery
  6、MongoDB Replica set settings

Main operation steps

1、 Close running MongoDB The server .

   Now let's specify --replSet Option to start mongoDB.–replSet The basic syntax is as follows ：

mongod --port "PORT" --dbpath "YOUR_DB_DATA_PATH" --replSet "REPLICA_SET_INSTANCE_NAME"

   example

mongod --port 27017 --dbpath "D:\set up\mongodb\data" --replSet rs0

   The above example will start a named rs0 Of MongoDB example , Its port number is 27017.

   After startup, open the command prompt box and connect mongoDB service .

   stay Mongo The client uses the command rs.initiate() To start a new replica set .

   We can use rs.conf() To view the configuration of the replica set

   View replica set status using rs.status() command

2、 Replica set add members
   Add members of the replica set , We need to use multiple servers to start mongo service . Get into Mongo client , And use rs.add() Method to add members of the replica set .

   grammar
  rs.add() The basic syntax of the command is as follows ：

>rs.add(HOST_NAME:PORT)
 example 
 Suppose you have started a program called mongod1.net, The port number is 27017 Of Mongo service .  Use... In the client command window rs.add()  Command to add it to the replica set , The command is as follows ：

>rs.add("mongod1.net:27017")
>

  MongoDB In, you can only use the main node to Mongo Add service to replica set , Determine the current running Mongo Whether the service is the master node can use the command db.isMaster() .

  MongoDB The replica set of is different from our common master-slave , After the master-slave host goes down, all services will stop , The replica set after the host goes down , The replica takes over the master node and becomes the master node , No downtime .

MongoDB Fragmentation

Introduce

   stay Mongodb There's another kind of cluster , It's sharding technology , You can meet MongoDB The demand for a huge increase in data volume .
   When MongoDB When storing massive amounts of data , A machine may not be enough to store data , It may not be enough to provide acceptable read and write throughput . At this time , We can split data on multiple machines , Make database system can store and process more data .

Why use shards

  1、 Copy all writes to the primary node
  2、 Delayed sensitive data will be queried at the primary node
  3、 A single replica set is limited to 12 Nodes
  4、 When the request volume is large, there will be insufficient memory .
  5、 Not enough local disks
  6、 Vertical expansion is expensive

MongoDB The distribution of the cluster structure

The figure below shows MongoDB Use the partition cluster structure to distribute ：

In the figure above, there are three main components as follows ：

Shard:
   Used to store actual data blocks , In a real production environment shard server A character can be made up of several machines replica set To undertake , Prevent single point failure of the main engine

Config Server:
  mongod example , Store the whole ClusterMetadata, These include chunk Information .

Query Routers:
   Front-end routing , The client is connected by this , And make the whole cluster look like a single database , Front end applications can be used transparently .

Slice instance

The port distribution of slice structure is as follows ：

Shard Server 1：27020
Shard Server 2：27021
Shard Server 3：27022
Shard Server 4：27023
Config Server ：27100
Route Process：40000

Step one ： start-up Shard Server

[root@100 /]# mkdir -p /www/mongoDB/shard/s0
[root@100 /]# mkdir -p /www/mongoDB/shard/s1
[root@100 /]# mkdir -p /www/mongoDB/shard/s2
[root@100 /]# mkdir -p /www/mongoDB/shard/s3
[root@100 /]# mkdir -p /www/mongoDB/shard/log
[root@100 /]# /usr/local/mongoDB/bin/mongod --port 27020 --dbpath=/www/mongoDB/shard/s0 --logpath=/www/mongoDB/shard/log/s0.log --logappend --fork
....
[root@100 /]# /usr/local/mongoDB/bin/mongod --port 27023 --dbpath=/www/mongoDB/shard/s3 --logpath=/www/mongoDB/shard/log/s3.log --logappend --fork

Step two ： start-up Config Server

[root@100 /]# mkdir -p /www/mongoDB/shard/config
[root@100 /]# /usr/local/mongoDB/bin/mongod --port 27100 --dbpath=/www/mongoDB/shard/config --logpath=/www/mongoDB/shard/log/config.log --logappend --fork

   Be careful ： Here we can start up like ordinary mongodb Start like a service , No need to add —shardsvr and configsvr Parameters . Because the function of these two parameters is to change the startup port , So we can specify the port by ourselves .

Step three ： start-up Route Process

/usr/local/mongoDB/bin/mongos --port 40000 --configdb localhost:27100 --fork --logpath=/www/mongoDB/shard/log/route.log --chunkSize 500

  mongos In the startup parameter ,chunkSize This item is used to specify chunk The size of the , The unit is MB, The default size is 200MB.

Step four ： To configure Sharding

   Next , We use MongoDB Shell Log in to mongos, add to Shard node

[root@100 shard]# /usr/local/mongoDB/bin/mongo admin --port 40000
MongoDB shell version: 2.0.7
connecting to: 127.0.0.1:40000/admin
mongos> db.runCommand({
     addshard:"localhost:27020" })
{
     "shardAdded" : "shard0000", "ok" : 1 }
......
mongos> db.runCommand({
     addshard:"localhost:27029" })
{
     "shardAdded" : "shard0009", "ok" : 1 }
mongos> db.runCommand({
     enablesharding:"test" }) # Set the database stored in slices 
{
     "ok" : 1 }
mongos> db.runCommand({
     shardcollection: "test.log", key: {
     id:1,time:1}})
{
     "collectionsharded" : "test.log", "ok" : 1 }

   Step five ： No major changes are needed in the program code , Connect ordinary... Directly according to mongo Like the database , Connect the database to the interface 40000