Log slimming operation: from 5g to 1g!

Source of the article ：https://c1n.cn/mtr19

In daily development , Usually for the convenience of debugging 、 It's convenient to check the problem , It will print a lot INFO Level of logging .

With more and more visits , carelessly , A log file for a day size Is greater than a certain threshold （ Such as 5G）, therefore , Received the alarm of optimizing the log size , Do not optimize feedback to your supervisor within a certain period of time , Embarrassed ...

Too large logs can easily cause some o & M operations to consume machine performance , Such as log file retrieval 、 Data collection 、 Disk cleaning, etc .

that , What are the common ideas for log slimming ？ This article combines a specific case to talk about my views .

Log slimming methodology

picture

Only print the necessary logs

Sometimes for the convenience of testing , A lot of temporary printing INFO The level of log . For this kind of log , Before the project goes online , Unnecessary logs can be deleted or adjusted to DEBUG Level .

But in some scenarios, some logs can be printed as DEBUG It can also be printed as INFO, Print as INFO Level takes up space , Print as DEBUG When checking problems online, you need to use , What about swelling ？

picture

We can transform the logging tool class , Support context passing when a switch （ There is no switch for normal call , Through the company's Tracer perhaps RPC Context passing ）, You can temporarily DEBUG Log promoted to INFO Level .

The pseudocode is as follows ：

if(log.isDebugEnable()){
   log.debug(xxx);
}else if(TracerUtils.openDebug2Info()){
 log.info("【debug2info】"+xxx);
}

such , You can print some entanglements as INFO The log log Print as DEBUG Level , When checking problems, it will be automatically promoted to INFO journal .

To avoid misunderstanding , distinguish DEBUG promote INFO   Log and ordinary INFO journal , Add something similar 【debug2info】 Log prefix .

Of course , You can also do some other coquettish operations , Here's just an example , Please draw inferences from one instance .

Merge print

Some logs can be merged , Consider merging .

For example, it is printed before and after the same method INFO journal ：

INFO   [64  position traceId]   XXXService  Before execution   size =10 INFO   [64  position traceId]   XXXService  After execution   size =4

Can be merged into one ：

INFO [64  position traceId] XXXService  Before execution  size =10  After execution  size =4

simplify & abbreviation & Compress

A certain log is very necessary , But the printed object is a little big , If it can meet the needs of problem troubleshooting , We can ：

• Choose to print only its ID

• Create a log specific object that retains only key fields , Convert to log specific objects , Re print .

• You can use abbreviations , Such as write Simplified as w, read Simplified as r,execute Simplified as e etc. ; such as pipeline There is 20 Core bean , When printing logs, you can use different numbers instead bean Full name , Such as S1,S2 , Although not so intuitive , But you can check the problem , It also reduces the amount of logs .

Optimization case

Scene description

A business scenario involves many bean, In order to reuse some general logic , these bean All inherit from some abstract class .

In an abstract class , Defines execution bean Some general logic before and after , Such as printing the current before and after execution pipeline in item The number of . the last one bean After performing the result conversion, you need to print the result .

Optimization analysis

① Print only the necessary logs

• Due to the current bean Before execution Equivalent to the previous bean After execution , Therefore, only print the log after execution , Before execution INFO The log can be deleted or changed to DEBUG （ Print only the necessary logs ）

• Usually, problems only appear before and after implementation size In case of inconsistency , Therefore, you can add a judgment before printing the log after execution , If before and after execution size The same is not printed .

The pseudocode is as follows ：

if(sizeBefore != sizeAfter){
log.info("service:{},  front size:{}, after size:{}", getName(),sizeBefore, sizeAfter)
}

The effect is obvious , Because most bean Before and after the execution of size It's the same , Will not print this log . And suppose there was 20 individual , This log needs to be printed 20 Time , After improvement, you may only need to print 2-3 Time .

② Log merge

In order to check the problem conveniently, you also need to print the pre execution size , Then we will execute the previous size Record in memory , When printing the post execution log, print the pre execution log size.

The pseudocode is as follows ：

log.info("service:{},  Before execution size:{}", getName(),sizeBefore)

log.info("service:{},  After execution size:{}", getName(),sizeBefore, sizeAfter)

After the merger ：

log.info("service:{},  front size:{}, after size:{}", getName(),sizeBefore, sizeAfter)

③ Log reduction

For the end result , The result object （ Such as XXDTO） Translate to include only key information , Such as id, title Log object for （XXSimpleLogDTO）, Print after converting to log object .

log.info("resultId:{}",result.getId());

perhaps ：

log.info("result:{}",toSimpleLog(result));

Effect evaluation

This log is generated in one day 5G about , It's 100 percent here 80% Both sides are before and after printing size,10% Print the final result on the left and right , There are other logs .

After the above method optimization , The daily log volume is insufficient 1G.

picture

In order to meet the needs of troubleshooting , And realize the trade-off between log slimming .

summary

Log slimming needs to be weighed , Keep the necessary logs for troubleshooting as simple as possible . You can delete unnecessary logs , Merge logs , Log simplification and other ways to optimize .

We can also carry out some operations , Support online DEBUG Temporary lifting INFO（ You can also use it arthas） To help us check the problem .

What other good log slimming skills do you have ？ Welcome to comment and exchange at the end of the article .

We have created a high-quality technical exchange group , With good people , I will be excellent myself , hurriedly Click Add group , Enjoy growing up together . in addition , If you want to change jobs recently , Years ago, I spent 2 A wave of large factory classics were collected in a week , Those who are ready to change jobs after the festival can Click here to get ！

Recommended reading

• Thread.sleep(0) What's the point ？
  

• illustrated ：JVM Memory layout details
  

• MySQL Sub database sub table and its smooth expansion scheme
  

··································

Hello , I'm a procedural ape DD,10 Old driver developed in 、 Alibaba cloud MVP、 Tencent cloud TVP、 I have published books and started a business 、 State-owned enterprises 4 In the Internet 6 year . From ordinary developers to architects 、 Then to the partner . Come all the way , My deepest feeling is that I must keep learning and pay attention to the frontier . As long as you can hold on , Think more 、 Don't complain 、 Do it frequently , It's easy to overtake on a curve ！ therefore , Don't ask me what I'm doing now, whether it's in time . If you are optimistic about one thing , It must be persistence to see hope , Instead of sticking to it when you see hope . believe me , Just stick to it , You must be better than now ！ If you don't have any direction , You can pay attention to me first , Some cutting-edge information is often shared here , Help you accumulate the capital to overtake on the curve .

Click to collect 2022 newest 10000T Learning materials