Mongodb replica set and partitioned cluster

mongoDb Replicate set

It mainly provides two functions

1. The data is written to the master node （Primary） Copy data to another replica section （Secondary） light

2. When the primary node fails, a new replacement node is automatically selected

Typical replica set structure

A typical replica set consists of three or more nodes with voting rights , One of the master nodes （Primary）： Receive write operation , Read operation and vote at election , Two or more slave nodes (Secondary)： Copy the new data on the master node and vote on the election

How data is copied

When a modification operation , Whether it's insertion , Update or delete , When you reach the master node , Its operations on the data will be recorded （ After some necessary conversion ）. These records are called oplog

The slave node continuously obtains the information of the new master node from the master node oplog, And play back on your own data , So as to keep consistent with the data of the master node .

Is very similar redis In persistence AOF The way .

Replica set building

 1. Create a data catalog file

mkdir -p /data/db{1,2,3}

 

2. Prepare the configuration file for each database （ Copy each of the sets mongod The process should be on a different server . We now run three instances on one server , So configure them separately ）

1. Different ports ： 28017, 28018,28019.
2. Different data directories 
data/db1,data/db2,data/db3
3.  Different log file paths . Use in instance 
/data/db1/mongod.log
/data/db2/mongod.log
/data/db3/mongod.log   touch mongod.log

Then edit mongod.conf file  

data/db1/mongod.conf

systemLog:
  destination: file
  path: /data/db1/mongod.log
  logAppend: true
storage:
  dbPath: /data/db1
net:
  bindIp: 0.0.0.0
  port: 28017
replication:
  replSetName: rs0
processManagement:
  fork: true

/data/db2/mongod.conf

systemLog:
  destination: file
  path: /data/db2/mongod.log
  logAppend: true
storage:
  dbPath: /data/db2
net:
  bindIp: 0.0.0.0
  port: 28018
replication:
  replSetName: rs0
processManagement:
  fork: true

/data/db3/mongod.conf

systemLog:
  destination: file
  path: /data/db3/mongod.log
  logAppend: true
storage:
  dbPath: /data/db3
net:
  bindIp: 0.0.0.0
  port: 28019
replication:
  replSetName: rs0
processManagement:
  fork: true

  To start, respectively,

mongod -f /data/db1/mongod.conf

mongod -f /data/db2/mongod.conf

mongod -f /data/db3/mongod.conf

Link the first one mongo --port 28017

It is found that the three machines are still independent of each other , At this time, configure it to form a cluster

rs.initiate({

_id:"rs0",
members:[{
_id:0,
host:"localhost:28017"
},{
_id:1,
host:"localhost:28018"
},{
_id:2,
host:"localhost:28019"
}]
})

adopt rs.status() View replica set status

 

  It is found that the port formed at this time is 28017 Master node , The other two are slave node architectures .

By default, non primary nodes are not allowed to read data .

Can be executed by rs.secondaryOk() Open read permission

 

MongoDB Fragmentation cluster

The composition of the partitioned cluster

  Building a 2 It's a fragmented cluster

1. Create a data directory ： Prepare to use... For two replica sets , Each replica set has three instances , common 6 Data nodes . 

mkdir -p /data/shard1    /data/shard1second1     /data/shard1second2     /data/shard2    /data/shard2second1     /data/shard2second2   

2. Create a log file , common 6 File

touch  /data/shard1/mongod.log     /data/shard1second1/mongod.log    /data/shard1second2/mongod.log    /data/shard2/mongod.log     /data/shard2second1/mongod.log    /data/shard2second2/mongod.log 

 3. Start the first mongod Slice instance ( There are three examples )

mongod --bind_ip 0.0.0.0 --replSet shard1 --dbpath /data/shard1 --logpath /data/shard1/mongod.log --port 27010   --fork   --shardsvr

mongod --bind_ip 0.0.0.0 --replSet shard1 --dbpath /data/shard1second1  --logpath /data/shard1second1/mongod.log --port 27011   --fork   --shardsvr

mongod --bind_ip 0.0.0.0 --replSet shard1 --dbpath /data/shard1second2 --logpath /data/shard1second2/mongod.log --port 27012   --fork   --shardsvr

4. The first piece mongod After the instances are started , Add it to the replication set

rs.initiate(
{_id:"shard1",
    "members":[ 
                {"_id":0,"host":"192.168.120.101:27010"},
                {"_id":1,"host":"192.168.120.101:27011"},
                {"_id":2,"host":"192.168.120.101:27012"}
                ]
});

To configure Config Replicate set ： There are three examples

1. Create a data directory ：

mkdir -p /data/config    /data/configsecond1     /data/configsecond2

 2. Create a log file

touch  /data/config/mongod.log     /data/configsecond1/mongod.log    /data/configsecond2/mongod.log 

3. Start configuration replication set

mongod --bind_ip 0.0.0.0 --replSet config --dbpath /data/config  --logpath /data/config/mongod.log --port 37010   --fork   --configsvr 

mongod --bind_ip 0.0.0.0 --replSet config --dbpath /data/configsecond1 --logpath /data/configsecond1/mongod.log --port 37011   --fork   --configsvr 

mongod --bind_ip 0.0.0.0 --replSet config --dbpath /data/configsecond2  --logpath /data/configsecond2/mongod.log --port 37012   --fork   --configsvr 

4. Configure the replication set for initialization

  rs.initiate(
    {_id:"config",
        "members":[ 
                    {"_id":0,"host":"192.168.120.101:37010"},
                    {"_id":1,"host":"192.168.120.101:37011"},
                    {"_id":2,"host":"192.168.120.101:37012"}
                    ]
    });

To configure mongs Routing node

1. start-up mongos example , You need to specify the address list of the configuration server

mongos --bind_ip 0.0.0.0 --logpath /data/mongos/mongos.log --port 4000 --fork --configdb config/192.168.120.101:37010,192.168.120.101:37011,192.168.120.101:37012

  among configdb Configure the address list of the server for

2. Connect to mongos in , And add slices

mongo --port 4000 Local direct link

Execute the script

sh.addShard("shard1/192.168.120.101:27010,192.168.120.101:27011,192.168.120.101:27012");

Check the slice status

sh.status();

Create a shard table ：

 1. MongoDB Is based on set , Even if there is a fragmented cluster, it does not mean that the data will be automatically fragmented , Need to display the slice table

The first thing you need to Enable database sharding

sh.enableSharding(" Library name ");

Such as ：

sh.enableSharding("order");

sh.shardCollection(" Library name . Collection name ",{_id: "hashed"});

sh.shardCollection("order.account",{_id: "hashed"});

Dynamic capacity

  Create a second replica set to implement sharding

mongod --bind_ip 0.0.0.0 --replSet shard2 --dbpath /data/shard2 --logpath /data/shard2/mongod.log --port 27013   --fork   --shardsvr

mongod --bind_ip 0.0.0.0 --replSet shard2 --dbpath /data/shard2second1  --logpath /data/shard2second1/mongod.log --port 27014   --fork   --shardsvr

mongod --bind_ip 0.0.0.0 --replSet shard2 --dbpath /data/shard2second2 --logpath /data/shard2second2/mongod.log --port 27015   --fork   --shardsvr

Initialize the replica set

rs.initiate(
{_id:"shard2",
    "members":[ 
                {"_id":0,"host":"192.168.120.101:27013"},
                {"_id":1,"host":"192.168.120.101:27014"},
                {"_id":2,"host":"192.168.120.101:27015"}
                ]
});

Join the cluster partition

mongo --port 4000
sh.addShard("shard2/192.168.120.101:27013,192.168.120.101:27014,192.168.120.101:27015");

MongoDB Practical application  

1. java Native Client

introduce maven

<dependencies>
    <dependency>
        <groupId>org.mongodb</groupId>
        <artifactId>mongodb-driver-sync</artifactId>
        <version>4.1.1</version>
    </dependency>
</dependencies>

public class QuickStart {

    public static void main(String[] args) {
        //  Connect to the local default port Mongod
        // MongoClient mongoClient = MongoClients.create()
        //  Connect to the specified port of the remote service Mongod
        // MongoClient mongoClient = MongoClients.create("mongodb://host1:27017");
        //  Connect the specified port replication set 
        // MongoClient mongoClient = MongoClients.create("mongodb://host1:27017,host2:27017,host3:27017/?replicaSet=myReplicaSet");
        //  Connect  mongos route :  Connect a 
        // MongoClient mongoClient = MongoClients.create( "mongodb://localhost:27017" );
        //  Connect multiple mongos route 
        MongoClient mongoClient = MongoClients.create("mongodb://111.229.189.98:4000");


        // Get the database 
        MongoDatabase database = mongoClient.getDatabase("productdb");

        //  Get collection 
        MongoCollection<Document> productdesc=database.getCollection( "productdesc" );

        Document doc = new Document("name", "MongoDB")
                .append("type", "database")
                .append("count", 1)
                .append("versions", Arrays.asList("v3.2", "v3.0", "v2.6"))
                .append("info", new Document("x", 203).append("y", 102));

        productdesc.insertOne(doc);


        Bson eq = eq("name", "MongoDB");
        FindIterable<Document> find = productdesc.find(eq);
        Document first=find.first();
        System.out.println(first);

    }
}

 

 

Spring Boot Integrate ：

1. introduce maven

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>
   <parent>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-parent</artifactId>
      <version>2.4.0</version>
      <relativePath/> <!-- lookup parent from repository -->
   </parent>
   <groupId>com.example</groupId>
   <artifactId>demo</artifactId>
   <version>0.0.1-SNAPSHOT</version>
   <name>demo</name>
   <description>Demo project for Spring Boot</description>

   <properties>
      <java.version>1.8</java.version>
   </properties>

   <dependencies>
      <dependency>
         <groupId>org.springframework.boot</groupId>
         <artifactId>spring-boot-starter-data-mongodb</artifactId>
      </dependency>
      <dependency>
         <groupId>org.springframework.boot</groupId>
         <artifactId>spring-boot-starter-web</artifactId>
      </dependency>

      <dependency>
         <groupId>org.springframework.boot</groupId>
         <artifactId>spring-boot-starter-test</artifactId>
         <scope>test</scope>
      </dependency>
      <dependency>
         <groupId>org.projectlombok</groupId>
         <artifactId>lombok</artifactId>
         <version>1.18.12</version>
      </dependency>


   </dependencies>

   <build>
      <plugins>
         <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
         </plugin>
      </plugins>
   </build>

</project>

 

 2. Add configuration class

For example, connect the above mongos, Use MongoTemplate Perform database operations .

package com.example.demo.config;

import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.mongodb.core.MongoTemplate;

@Configuration
public class AppConfig {

    public @Bean
    MongoClient mongoClient() {
        return MongoClients.create("mongodb://111.229.189.98:4000");
    }

    public @Bean
    MongoTemplate mongoTemplate() {
        return new MongoTemplate(mongoClient(), "productdb");
    }
}

 

3. Test class

package com.example.demo;

public class Person {

  private String id;
  private String name;
  private int age;

  public Person(String name, int age) {
    this.name = name;
    this.age = age;
  }

  public String getId() {
    return id;
  }
  public String getName() {
    return name;
  }
  public int getAge() {
    return age;
  }

  @Override
  public String toString() {
    return "Person [id=" + id + ", name=" + name + ", age=" + age + "]";
  }

}

package com.example.demo;

import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.stereotype.Component;

import java.util.List;

import static org.springframework.data.mongodb.core.query.Criteria.where;
import static org.springframework.data.mongodb.core.query.Query.query;
import static org.springframework.data.mongodb.core.query.Update.update;
 
@Component
@Slf4j
public class ApplicationRunnerTest implements ApplicationRunner{

    @Autowired
    private MongoTemplate mongoOps;

    @Override
    public void run(ApplicationArguments applicationArguments) throws Exception {

        Person p = new Person("Joe", 34);

          //  Inserted into the document 
        mongoOps.insert(p);
        log.info("Insert: " + p);

        //  Query the document 
        p = mongoOps.findById(p.getId(), Person.class);
        log.info("Found: " + p);

        //  Update the document 
        mongoOps.updateFirst(query(where("name").is("Joe")), update("age", 35), Person.class);
        p = mongoOps.findOne(query(where("name").is("Joe")), Person.class);
        log.info("Updated: " + p);

        //  Delete the document 
        mongoOps.remove(p);

        // Check that deletion worked
        List<Person> people =  mongoOps.findAll(Person.class);
        log.info("Number of people = : " + people.size());
        mongoOps.dropCollection(Person.class);
    }
}