In depth analysis of Apache bookkeeper series: Part 4 - back pressure

This paper is written by StreamNative Organization translation from 《Apache BookKeeper Internals — Part 4 — Back Pressure》, author Jack Vanlightly,Apache BookKeeper Committer.

Translator information

Zhangwenfeng , Love to share , Embrace open source , Community nickname zeroGozhang, Now he is the big data architect of xinghuan information technology finance .

This series of blogs is based on Apache Pulsar Configured Apache BookKeeper 4.14 To write .

In this article , We will introduce BookKeeper All back pressure mechanisms used by the server nodes , To protect yourself from read and write overloads . Back pressure mechanism is necessary in any data system , It can not only prevent resource exhaustion , At the same time, it can make the system handle overload more gracefully and smoothly . When Bookie Overload , Ideally , We hope it will continue to provide high throughput for the content it can handle , And reject other requests .

Some components restrict the data structures in their memory , To prevent Bookie Run out of memory . for example , in the light of Journal The line will be right entry Limit the number of , Write cache has a limit on the number of bytes written . in fact , The write cache is pre allocated , So no matter Bookie What's the load of , Its memory usage is fixed .

By default , Other components are unlimited , For example, suspended Netty Mission . Besides , Even after considering all the different limitations of the internal buffer , When they are combined, they may exceed Bookie Memory limit of .

Let's take a look at various back pressure and memory limitation mechanisms .

chart 1： Location of various back pressure mechanisms

1 - Total number of writes in progress

We can do it through configuration  maxAddsInProgressLimit  Parameter controls the total number of writes in progress . When the total number of write requests in progress reaches this limit , The system will block those processing new write requests Netty Threads , Until the number of writes in progress falls below this limit again . When Netty Stop reading data from network buffer , Data will be overstocked in TCP Buffer zone , triggering TCP Built in back pressure mechanism , Notify the upstream sender and suspend sending , take TCP Back pressure is applied to the client .

This can effectively avoid the thread pool task queue 、Journal Queue, etc. caused by backlog of too many requests OutOfMemory error . There are no restrictions by default .

2 - Total number of reads in progress

We can also configure  maxReadsInProgressLimit  Parameters , Limit the total number of reads in progress . This triggers the same Netty Thread blocking behavior .

3 - Number of pending write requests per write thread pool thread

We can also limit the size of the write thread pool task queue （ To configure maxPendingAddRequestsPerThread, The default value is 10000）, To reduce the number of pending write requests . Once the task queue is full , The client will receive  TOO_MANY_REQUESTS  Response , Causes the client to choose another Bookie To write .

4 - The number of pending read requests per read thread pool thread

The size of the read thread pool task queue can be limited in the same way as the write thread pool （ To configure  maxPendingReadRequestsPerThread）. Once the task queue is full , Read requests begin to be rejected in the same way as write requests , The affected client will choose another Bookie To read .

5 - Journal queue

Journal Get task from blocking queue . When Journal When the queue size reaches its maximum due to the gradual failure to keep up with the filling speed of the queue , The writing thread will be blocked , No more work can be performed until there is space on the queue .

Next , The write thread task queue will be filled , Write requests will begin to be rejected .

6 - DbLedgerStorage Write denied

If the active write cache is full , The swapped write cache is not yet available （ Because it's still brushing the plate ）, be Bookie Unable to process more writes . Only when an empty write cache becomes available , It can accept more writes . therefore , When no write cache is available after waiting for a period of time （ By default 10 second , from  dbStorage_maxThrottleTimeMs  Definition ）, Write requests will be rejected .

stay BookKeeper 4.14 In the version , This will result in a generic error code being returned to the client , instead of  TOO_MANY_REQUESTS  Code . This problem will be fixed in a future release .

7 - Netty Non writable channel

In order to prevent the passage of Netty The channel is out of memory when sending a response to the client , Can pass  waitTimeoutOnResponseBackpressureMs  Parameter configuration fallback mechanism . If the channel buffer is full , bring Netty The channel becomes non writable , The write response may be delayed to the corresponding waiting time . If the channel is still not writable , The system will not send a response , And it will send an indicator of channel error . If this fallback mechanism is not configured , Although its status is not writable , The response will still be sent to the channel ; If too many bytes are sent along the channel , May cause out of memory errors .

Besides ,Netty The number of queued tasks can also be determined by using Java Attribute to limit ：io.netty.eventLoop.maxPendingTasks. This parameter also prevents Netty Thread accepts too many tasks .

summary

The back pressure applied by the receiving end will propagate upward along the link , Until we reach the source . about BookKeeper Come on , Back pressure usually starts from Journal and DbLedgerStorage Component start , Then it goes up the link to the thread pool （ Their task queues ）、Netty, Finally arrive at the client .

But for the Apache Pulsar For clusters , More than that .Pulsar Broker It also has its own internal back pressure mechanism , It will eventually trigger a return to Pulsar producers and consumers .

This is the last article in this series . Next , We are going to launch a series of new blogs , The content is about BookKeeper Some indicators produced , And how to use the thinking patterns we tried to build in this series to explain them .

Recommended reading

1. 1.  In depth analysis of Apache BookKeeper series ： Chapter one — Framework principle

2. 2.  In depth analysis of BookKeeper Multi copy agreement （ One ）

3. 3.  In depth analysis of Apache BookKeeper series ： Second articles — Write operation principle

4. 4. Apache BookKeeper Insight （ One ） — External consensus and dynamic members

5. 5.  In depth analysis of Apache BookKeeper series ： Third articles —— Reading principle
   

---

▼ Focus on 「Apache Pulsar」, Get dry and dynamic ▼

reply BookKeeper, Join in  Pulsar Storage  team

Click on 「 Read the original 」, receive Pulsar vs Kafka The latest evaluation report PDF！