New generation cloud native message queue (I)

Pulsar brief introduction

 Apache Pulsar It is a new generation of cloud native distributed message flow platform , Set message 、 Storage 、 Lightweight functional calculation as a whole , Adopt the architecture design of separation of computing and storage , Multi tenant support 、 Persistent storage 、 Multi machine room cross regional data replication , With strong consistency 、 High throughput 、 Low latency and high scalability streaming data storage features .

 Pulsar from Yahoo Development , On 2016 Open source at the end of the year , And in 2017 year 6 Submitted to Apache The incubator . until 2018 year 9 month ,Apache The foundation announced Apache Pulsar Graduate and become a top program .

Pulsar The advantages of ：

Pulsar framework

Pulsar The whole is divided into client and server , The client is the same as other message queues , Both producers and consumers . The specific content will be explained in detail later .

The structure of the server can be divided into three layers , They are the agent layer 、Broker Layer and the Bookie layer .

• Broker layer

  This layer is not required . If there is no agent layer , that Pulsar The client needs to communicate directly with each Broker Establishing a connection . But more often , As a whole Server The cluster exposes the agent layer IP Address , The agent layer forwards the corresponding request .

• Broker layer

 Broker And in general MQ Different , Data is not stored directly in Broker in , It's stored in BookKeeper In the cluster . because Broker The statelessness of , It can realize the rapid expansion and contraction of capacity in the container environment very conveniently .Broker Mainly responsible for the whole Pulsar The business logic of the cluster . Handle common data flow requests . The interfaces for managing flows are based on RESTFul Of HTTP, The interface of data flow is based on Pulsar Custom binary protocol , Use ProtoBuf As a serialization tool .

• BookKeeper layer

 BookKeeper Is a scalable 、 Fault tolerance 、 Low latency 、 A storage service that can only append data .Pulsar Use it to store data , Don't do any business logic . It has dynamic scaling 、 Automatic fault tolerant recovery 、 The ability to separate reading and writing .

 Broker and BookKeeper Will use ZooKeeper As a metadata storage service . But due to the ZooKeeper It's too heavy , from 2.10 after Pulsar Support optional metadata storage services .

Pulsar Source structure

•  distribution

  The final compiled products will be in this directory , It is divided into io、offloaders、servers Three sub modules .servers by Pulsar The main service of the application ;io Modules provide the same / Data conversion between different systems 、offloaders Cold data can be migrated to some third-party media

• buildtools

  Include the tools used during construction . For example GIthub Submit a PR, Will automatically trigger the test , Retry of test 、 The style check is completed by this module

• conf

  All configuration data directories

• docker

 pulsar Of Docker Mirror module , There's a corresponding dockerfile

• pulsar-metadata

  General metadata module , Will be managed-ledger、pulsar-sql Etc .

• pulsar-testclient

  Modules for unit testing

• pulsar-config-validation

  A general package used to verify the correctness of the configuration , stay Function and io Will be used in

• pulsar-common

 Broker and Client Public modules that will be referenced at the same time , It defines some objects that will be used on both sides

• bouncy-castle

 Java Class library , Default to JCE A supplement to . Than SUN Default provided JCE Many more cipher suites and algorithms

• tests

  Functional modules for testing

• testmocks

  During unit testing , It takes a lot of Mock Tool class of , Store here

• pulsar-broker

 Pulsar The core function module of the server

• pulsar-broker-auth-athenz

 Broker Of Athenz Authentication plug-ins

• pulsar-broker-auth-sasl

 Broker Of SASL Authentication plug-ins

• pulsar-broker-common

 Broker General function module of the end , Like authentication 、 Configure general capabilities such as caching

• pulsar-broker-shaded

Used to build Broker Of shaded package

• pulsar-client

Data stream client

• pulsar-client-1x-base

Pulsar Yes 1.X Version of the protocol is compatible with clients

• pulsar-client-admin

  Manage streaming clients , Used to call management related interfaces , Such as creating 、 Delete Topic

• pulsar-client-admin-shaded

Simply for building clients shaded package

• pulsar-client-all

  Build a complete client , Both management flow and data flow are in one package

• pulsar-client-api

 Client Common to API

• pulsar-client-auth-athenz

Client Of Athenz Authentication plug-in

• pulsar-client-auth-sasl

 Client Of SASL Authentication plug-ins

• pulsar-client-messagecrypto-bc

  Message encryption tool library

• pulsar-client-tools

some CLI Tools

• pulsar-functions

Pulsar Function The main implementation of

• pulsar-io

Some concrete IO Tools , It can make Pulsar Data and other software ( Such as Kafka、Redis etc. ) Get through

• pulsar-proxy

Pulsar The agent layer of

• pulsar-sql

Pulsar SQL The main implementation modules , It can be used SQL Inquire about Pulsar The messages in the

• pulsar-websocket

Pulsar Of WebSocket Communication implementation module

• tiered-storage

offload Tool implementation module , Cold data can be backed up to other media