Flame diagram analysis Flink backpressure

The phenomenon

team A new version Flink Homework , There are modifications in the old homework , Make the final verification findings before publishing ,Fliink There is backpressure in operation , Some operators report red , see topic The consumer group has lag

analysis

The phenomenon at that time is similar to the following figure , The last dot shows busy max 100%

guess 1

Data skew
The degree of parallelism is 3, Look at each task The amount of data processed is relatively uniform

guess 2

It takes a long time to call the external interface
Use arthas trace Interface discovery takes time 20ms about

guess 3

taskmanager Nodes are frequent GC , Cause a pause
see task node GC The frequency and time consumption are normal

guess 4, Confirm code changes

Confirm that the code changes very little ？ It shouldn't be a code problem

reflection

A few hatchets come down , I still don't see the problem

Big recruit ( Flame chart )

Adjust configuration , Open the flame diagram
rest.flamegraph.enabled : true

On-cpu

see On-cpu The flame diagram is similar to the following

off-cpu chart

analysis

There is no lock related , So don't think about off-cpu

from on-cpu Look at , It shows that most of the homework cpu They're all carrying out socket Code
I didn't pay attention here at that time , Later, I found that I shouldn't have much time to execute this code , As for why it can be adjusted socket , From top to bottom, it is because the program adjusts an external http Interface .
The frequency of normal logic interface should not be very high , It indicates that there is a problem with the code change , Confirm again that there is a problem with the code change , After repairing the code and going online again , The task is normal , either lag, Always solve .

Flame chart

Flame diagram example code

public class HotTest {
    


    public static void main(String[] args) {
    

        run();

    }

    private static void run() {
    
        runA();
        runB();
    }

    private static void runB() {
    
        for (int i = 0; i < 300000; i++) {
    
            System.out.println("this is b");
        }
    }


    public static void runA() {
    
        runC();
        for (int i = 0; i < 500000; i++) {
    
            System.out.println("this is a");
        }
    }

    private static void runC() {
    
        for (int i = 0; i < 200000; i++) {
    
            System.out.println("this is c");

        }
    }

}

idea On the implementation

Generate flame chart

How to see

y Depth of axis code function call
x The axis width represents the number of function samples , It can be simple and time-consuming
From the flame diagram
run Function takes time =runA()+runB()
runA Function takes time =runC()+ Print
So although this picture looks very complicated , When analyzing problems , Just look at the width of all the top-level functions , Then find it from top to bottom according to the widest function , Which business is executing this , Analysis is enough

Back to the beginning

Most of cpu They're all carrying out socket Code ,socket The code is caused by frequent interface calls , Finally, I found that there was a problem here, which led to .

summary

Flame diagram is used in analyzing performance scenarios , Confirmation is very convenient , We should strengthen our study in this field in the future .