Selection and practice of distributed tracking system

APM(Application Performance Management & Monitoring) That is, application performance management and monitoring , It mainly analyzes and monitors the performance indicators of the program , To help develop 、 test 、 Operation and maintenance personnel to master the performance and operation of the program .

---

APM Principle and introduction

APM By aggregating the real-time data of each processing link of the business system , Analyze the path and processing time of each transaction in the business system , Realize the whole link performance monitoring of the application .

APM The difference between tools and traditional performance monitoring tools is , It not only provides some scattered resource monitoring points and indicators , Instead, it focuses on the performance bottleneck analysis of internal execution and inter system calls .

At present on the market APM The system is basically a reference Google Tracking system of large-scale distributed system Dapper The paper of ( Chinese translation ) To achieve .

APM The core idea of

When the application service nodes call each other , Record and pass an application level tag , Used to associate the entire call link . How to pass these tags , It is related to the communication protocol used between application nodes . If you use http agreement , Tags will be added to http In the head .

What the distributed tracking system should do is to record the identifier and timestamp of each sending and receiving action , Connect all services involved in a request . Some concepts involved are as follows ：

trace
Represents the trace of the complete call chain of a request , It's usually a tree structure , By multiple span form , all span Use a global traceId. As shown in the figure below ：

The figure describes a complete trace (Trace), Recorded Trace On each span Of id Father id. If one span No father id, Is called root span. all span Share a trace id( Not shown in the picture ).

span
The basic unit of tracking , One span It means a rpc call , Multiple span Form a trace.

APM The composition of

One APM The basic modules involved include ：

• data collection , Merge data in breadth and efficiency ——Agent.
• The data processing , Sort out the data , Transmission and storage ——Collector.
• data storage , Save the calculated indicators and aggregated link information in real time ——Storage.
• Data presentation , good-looking , Multifunctional display ——UI.

On the market APM Tools

In recent years, with the APM The rapid development of technology and market , Related products have sprung up in large numbers .

Foreign commercial products Dynatrace、Appdynamics、New Relic etc. , The domestic RichAPM、OneAPM、 Alibaba's business real-time monitoring service ARMS And baidu MTC etc. .

Of course , Open source products have also developed rapidly . The main APM Open source tools include Pinpoint,SkyWalking,Zipkin and CAT.

Pinpoint

This is open source by a Korean team , adopt JavaAgent The mechanism of , Implant at bytecode level , Join in traceid And the purpose of grabbing performance data . Because it is implanted at the bytecode level , The intrusion to business code is zero , It is widely used .
Pinpoint The basic architecture of is as follows ：

• Pinpoint collect： Used to collect various performance data .
• Pinpoint Agent： probe , With the application server ( Such as tomcat) relation , Deploy to the same server , Bytecode enhancement is used to realize no invasion of application code .
• Pinpoint Web UI： Data presentation layer
• Pinpoint Storage： Data storage layer , Store the collected data in Hbase in .

stay Pinpoint in , The core data structure consists of Span,Trace and TraceId form .Span and Trace Described above ,TraceId By TransactionId,SpanId and ParentSpanId Composed of key Set .TransactionId Point out the news id, It must be globally unique in the whole server .

SkyWalking

SkyWalking It was opened source by a domestic person named Wu Sheng , Has contributed to at present Apache The foundation , Become Apache Top projects .
SkyWaling Automatically collect performance indicators through the probe , And conduct distributed tracking , Low intrusion into business code .

Zipkin

The system is twitter Open source . It's characterized by light weight , Easy to use and deploy . By means of Java Introduce the client into the program , Implicitly interceptable http,thrift And other forms of service invocation .

Cat

Cat It is open source by public comment , The main means to realize tracking is hard coding in the code , Do data collection by burying points , More invasive .

Open source APM Compare

dimension	Cat	ZipKin	PinPoint	SkyWalking
Realization way	Code buried point	Intercept request , Send data to ZipKin service	Java probe , Bytecode enhancement	Java probe , Bytecode enhancement
Access mode	Code hacking	Import configuration	Bytecode implantation , No intrusion	Bytecode implantation , No intrusion
data storage	Mysql,hdfs	ES,mysql,Cassandra	Hbase,mysql	ES,H2,Mysql,Tidb
Analysis granularity	Code level , Global call statistics , Call the police ,JVM monitor	Interface level	Method level , Global call statistics , Call the police	Method level , Global call statistics
report form	Enrich	Less	in	in
The way of burying	invasion	invasion	No intrusion	No intrusion
Client support	Java、C/C++、Node.js、Python	Java、go、JavaScript、Ruby、scala、php（ In addition to official support , The community also supports various other language clients ）	Java、Php	Java、C#、Php、Node.js、Go

In addition to the above apm Besides the tools , also elasticsearch APM.

Elastic stack stay 6.3 The release began to come out APM function .ElasticAPM Is based on Elastic Stack Built application performance monitoring system , It can be embedded inside the application process , Collect various operation indicators in real time and transmit them to elasticsearch The platform performs aggregation statistics , adopt kibana The interface side shows the performance and processing capacity of the application system .

The following will introduce Elastic APM Installation and use .

Reference material

[1]. https://www.jianshu.com/p/d2d912cc982a
[2]. https://www.jianshu.com/p/07a00d66e703
[3]. https://cloud.tencent.com/developer/article/1501891