Implementation of cqrs architecture mode under Kratos microservice framework

Separation of responsibility for command inquiry Command Query Responsibility Segregation Usually simplified as Command query separation , That is, the separation of reading and writing .

In a specific scenario , It can provide better performance . however , In terms of strong consistency , It does not guarantee . and , It will also bring cognitive burden . therefore , In practice , Need to be careful .

What is? CQRS

This concept comes from Command and query are separated （CQS, Command Query Separation）, From 1987 year Bertrand Meyer Of 《 Object oriented software construction 》 A Book , Its original concept is that we can divide object operations into ： command （Command） and Inquire about （Query） Two forms .

• command （Command）： After execution , Will change the state of the object .
• Inquire about （Query）： Just view the data of the object , Without changing the object .

and Separation of responsibility for command inquiry Command Query Responsibility Segregation ( abbreviation CQRS) Pattern is an architecture pattern , It can separate the command of changing the state of the model from the query of the state of the model .

In the era of monomer application , It is read-write separation :

In the era of micro Services , It becomes the separation of responsibility of command query :

What does the separation of reading and writing solve ？

The separation of read and write of database is ： Divide the database into master and slave databases , A main library for writing data , Multiple slave libraries complete the operation of reading data , The master and slave libraries synchronize data through some mechanism .

Most Internet businesses , Often read more and write less . Now , The reading of the database will first become the bottleneck of the database . At this time , If we want to improve the read performance of the database linearly , Eliminate read-write lock conflicts to improve the write performance of the database , Then you can use the read-write separation architecture ： Master-slave , Master, master, etc .

MySQL The most used is master-slave , The main database passes BinLog Synchronize to the slave database . This raises a question , Data inconsistency . If the pressure of writing data is great ,binlog Congestion will occur , The slave database data is not updated in time , Will read the old dirty data . So this scheme limits its scope of application ： Only the scenes with low requirements for consistency can be used . such as , Log query , Reports, etc. .

Realization CQRS

Here we discuss the access scenario of sequential data of the Internet of things .

We are divided into two micro Services ：

Log query service (kratos.logger.service)

Mainly open API Used to query the database , Get log data .

Log writing service (kratos.logger.job)

subscribe Kafka Log data write Topic, Write to the timing database .

Docker Deploy the development server

TimeScaleDB

docker pull timescale/timescaledb:latest-pg14
docker pull timescale/timescaledb-postgis:latest-pg13
docker pull timescale/pg_prometheus:latest-pg11

docker run -itd \
    --name timescale-test \
    -p 5432:5432 \
    -e POSTGRES_PASSWORD=123456 \
    timescale/timescaledb-postgis:latest-pg13

Kafka

docker pull bitnami/kafka:latest
docker pull bitnami/zookeeper:latest
docker pull hlebalbau/kafka-manager:latest

docker run -itd \
    --name zookeeper-test \
    -p 2181:2181 \
    -e ALLOW_ANONYMOUS_LOGIN=yes \
    bitnami/zookeeper:latest

docker run -itd \
    --name kafka-standalone \
    --link zookeeper-test \
    -p 9092:9092 \
    -v /home/data/kafka:/bitnami/kafka \
    -e KAFKA_BROKER_ID=1 \
    -e KAFKA_LISTENERS=PLAINTEXT://:9092 \
    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092 \
    -e KAFKA_ZOOKEEPER_CONNECT=zookeeper-test:2181 \
    -e ALLOW_PLAINTEXT_LISTENER=yes \
    --user root \
    bitnami/kafka:latest

docker run -itd \
     -p 9000:9000  \
     -e ZK_HOSTS="localhost:2181" \
     hlebalbau/kafka-manager:latest

Consul

docker pull bitnami/consul:latest

docker run -itd \
    --name consul-server-standalone \
    -p 8300:8300 \
    -p 8500:8500 \
    -p 8600:8600/udp \
    -e CONSUL_BIND_INTERFACE='eth0' \
    -e CONSUL_AGENT_MODE=server \
    -e CONSUL_ENABLE_UI=true \
    -e CONSUL_BOOTSTRAP_EXPECT=1 \
    -e CONSUL_CLIENT_LAN_ADDRESS=0.0.0.0 \
    bitnami/consul:latest

Jaeger

docker pull jaegertracing/all-in-one:latest

docker run -d \
    --name jaeger \
    -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
    -p 5775:5775/udp \
    -p 6831:6831/udp \
    -p 6832:6832/udp \
    -p 5778:5778 \
    -p 16686:16686 \
    -p 14268:14268 \
    -p 14250:14250 \
    -p 9411:9411 \
    jaegertracing/all-in-one:latest

test

Download tool

• Postman^[1]
• Offset Explorer^[2]

To test

Write the test

Use Offset Explorer Analog equipment , towards Topic logger.sensor.ts send out JSON data :

[{"ts": 1646409307, "sensor_id": 1, "temperature":30, "cpu":20}]

Test reading

Use Postman Initiate to log service gRPC Ask for a query .

Technology stack

• Kratos^[3]
• TimeScaleDB^[4]
• Kafka^[5]
• Consul^[6]
• Jaeger^[7]
• Entgo^[8]

The sample code

• Kratos Examples^[9]

Reference material

[1]

Postman: https://www.postman.com/downloads/

[2]

Offset Explorer: https://www.kafkatool.com/download.html

[3]

Kratos: https://go-kratos.dev/

[4]

TimeScaleDB: https://www.timescale.com/

[5]

Kafka: https://kafka.apache.org/

[6]

Consul: https://www.consul.io/

[7]

Jaeger: https://www.jaegertracing.io/

[8]

Entgo: https://entgo.io/

[9]

Kratos Examples: https://github.com/go-kratos/examples/tree/main/cqrs

[10]

Talking about CQRS The implementation of : https://medium.brobridge.com/%E6%B7%BA%E8%AB%87-cqrs-%E5%AF%A6%E7%8F%BE%E6%96%B9%E6%B3%95-3b4fcb8d5c86

[11]

Talking about the principle of micro service splitting : https://medium.brobridge.com/%E6%B7%BA%E8%AB%87%E5%BE%AE%E6%9C%8D%E5%8B%99%E6%8B%86%E5%88%86%E5%8E%9F%E7%90%86-d43fbb33e722

[12]

Detailed explanation CQRS Architecture mode : https://www.infoq.cn/article/wdlpjosudoga34jutys9