Full link service tracking implementation scheme

2019 year ,“Kentik A company survey shows that , Now 40% Of organizations consider themselves to be multi cloud users , Their organization has cloud services provided by two or more cloud service providers . One third of organizations have a hybrid cloud environment , At least one cloud computing service provider provides cloud services and internally deployed data center or third-party data center infrastructure .

 

Under this development trend ,IT Operation and maintenance management work has entered “ The best era of operation and maintenance ”, It's also “ The worst times of operation and maintenance ”. Enterprises are paying more and more attention to IT The value of operation and maintenance to business development , Find out IT The architecture has changed dramatically .

 

• The business deployment model is extremely flexible ： Public cloud 、 Private cloud 、 A hybrid cloud
• Business nodes are widely distributed ： It's hard to get to support the business XaaS The location of the instance
• The call hosting relationship is extremely complex ： The number of invocation dependencies between microservices is exponentially higher than before

Focus on operation and maintenance , The following difficulties need to be solved ：

• Production problems are not found in time ： Because the service invocation relationship between systems is not transparent , And tradition “ Total amount monitoring ” The pattern of , This results in... In the transaction link “ Problem service ” The impact of can not be quickly warned and notified , There is a certain lag in operation monitoring .
• The workload of troubleshooting is heavy ： Due to the limitation of monitoring means , And the operation data standards of each system are not unified , To solve the production problem, a large number of “ Development ” And “ Operation and maintenance ” resources , And the communication cost is high .
• Inefficient problem solving ： Because there is no unified serial identification for the operation data between systems , And different recording standards , It makes it impossible to locate quickly “ Problem service ”.

Observability

thus , It also triggered IT New exploration and practice in the field of operation and maintenance , There has been a different “SRE”、“ Observability ”、“AIOps” Equal concept . In order to solve the new generation IT The above problems under the framework , First of all, we should solve the problem of the data representation of the system running state , The industry put forward “ Observability (Observability)” Concept .

 

2017 year ,Peter Bourgon Proposed three pillars of observability —— indicators （Metrics）、 Call chain （Tracing） and journal （Logging）, These three dimensions cover almost all kinds of representation behavior of the application , Developers collect and view data from these three dimensions （Telemetry Data） Can do all kinds of things , Keep abreast of the running situation of the application . This has become the standard in the field of observability , It is also the foundation of system construction .

 

• indicators （Metrics）： A form of aggregated data , I often come into contact with QPS、TP99、TP999 And so on Metrics The category of , Generally, the design and implementation are based on statistical principles ;
• journal （Logging）： In a broad sense, logging is triggered by business requests or events , Record a snapshot of the application's state information . Unified collection of log data 、 Storage and parsing are affected by many factors , For example, structured and unstructured log processing , You often need a high-performance parser and cache ;
• Call chain （Tracing）： Come of SOA Technology era , The long call chain brought by servitization , It's hard to locate the problem by just relying on the log , Some measures are needed to compensate for complexity . So it's more expressive than Metrics More complicated .

The measure to implement observability is to produce at a small scale + collect + Handle + Consumption observations . Collecting observational data about the application environment is nothing new . However , From one application to another , Collection mechanisms and formats are rarely consistent .

 

For developers and SRE Come on , This inconsistency can be a nightmare . In providing observable solutions worldwide , The following agreements have to be mentioned / Tools , In support of observation data production 、 collect 、 Processing brings great convenience , Full link tracking is an important measure to practice observable consumption in the operation and maintenance field ：

 

• In the cloud native scene , Virtualization is more thorough 、 The environment is more dynamic . Make full use of observability to realize full link tracking , To achieve high business availability 、 Satisfy SLA Other requirements .
• Track the whole transaction link in a visual way , Realize rapid problem discovery 、 Location problem 、 Assist in problem solving ; To be more intuitive 、 Generate and use the observation data in a scientific way for real-time monitoring and analysis .
• introduce AI Technology for automated exception discovery 、 Location and repair .

Problems and challenges

From the link layer 、 The data layer 、 The application layer looks at , There are still many problems to be solved to complete full link tracking . The problems and challenges encountered in the link layer include cloud native / Convergence of internal and external traffic data in virtualized environment 、 Copy 、 shunt 、 The filtering technology is not complete , Cloud native / It is difficult to link traffic data between virtualized environment and physical switch environment ; Lack of performance monitoring analysis based on complete links .

 

The problems and challenges encountered by the data layer include ： Irregular logs （Logging）、 Call chain （Tracing） It is difficult for data to accurately describe the business end-to-end topology , The observation target data is incomplete ; An imprecise indicator system （Metrics） Information interference or analysis failure caused by .

 

The problems and challenges encountered by the application layer include ： The traditional concept of application performance monitoring technology is very important for cloud native / In a virtualized environment IT The observation capability requirements of the components do not match . for example ： Association and sorting of asynchronous interaction among multiple components 、 Call chain backtracking based on time slice, etc .

 

meanwhile , From the overall view of business scenarios, within each level 、 Lack of complete end-to-end penetration between levels , An observed system that cannot be completely described . Cloud wisdom passes through 10 Years of continuous practice 、 The accumulation of 、 Explore 、 innovation , A complete new generation oriented system has been built and implemented IT Full link tracing solution based on the architecture , Now we will systematically introduce the technical concept and implementation measures of the scheme .

Overall solution for full link tracking

Applicable scenario

No monitoring tools

Currently in manual operation and maintenance . As the business grows , Containerized applications , Manual operation and maintenance cannot meet the monitoring and service link tracking requirements of relatively complex services , Respond quickly to business , The operation and maintenance development cannot keep up with the business development .

There are a few monitoring tools

There are a few monitoring tools . Such as basic monitoring , Network management monitoring tool , Open source tools Zabbix etc. . However, the tool dispersion cannot be based on the service full link tracking system .

There are relatively complete tools

There are relatively complete monitoring tools , There are some tools that can form links , Such as commercial application performance monitoring , Open source ELK Log monitoring , Also exploring Pinpoint、SkyWalking、OpenTracing Standards, etc . But the application is more traditional , The modification log needs to be planned , There is no configuration management to assist in the formation of full link tracking .

Unified management and intelligence of operation and maintenance data

Plan unified operation and maintenance management , The goal is to make use of AIOps Ability to improve call chain tracing , With the help of intelligent operation and maintenance capability, we can reach the preliminary intelligent operation and maintenance level . The construction of the tool layer is relatively complete , Open source 、 commercial ,CMDB etc. . However, the call chain cannot be fully displayed and flexibly use the full link tracking data . The tools are scattered and cannot be based on the service full link tracking system . The link is incomplete , Links and monitoring tools , Monitoring data can not form a joint force , Configuration management cannot be applied to the full link tracking scenario .

Solution

The overall solution for full link service tracking relies on monitoring tools and intelligent operation and maintenance platform , It mainly includes the following four solutions , Log link tracing solution 、 Application link solutions 、 Network link solutions 、 Full link tracking solution based on fused data . The first three solutions rely on monitoring tools , The fourth solution based on fusion data , Integrate various operation and maintenance data , Based on intelligent operation and maintenance platform and algorithm capability . Achieve the ultimate goal of the whole link .

 

 

Basic objectives of full link service tracking , Significantly reduce mean time to recovery from failure . Guarantee in each key link of the operation and maintenance process

 

• Failure prevention phase ： Full link tracking index planning and observation , At the same time, the indicator is converted to the alarm threshold , If the fault occurs, predict and alarm in advance , The operation and maintenance issues can be handled at the first time ;
• Fault discovery stage ： The alarm shall be quickly notified to the operation and maintenance team ,
• Analysis and solution stage ： Fast fault analysis and processing based on full link service tracking , Quickly analyze and locate O & M problems through link tracking and visualization , Be measurable and observable .
• The second round and conclusion stage ： Historical data analysis , Full link optimization and supplement , Root cause location analysis , Business system optimization suggestions .

Through the above process , It is mainly about the efficiency improvement in the fault discovery stage, especially in the analysis and solution stage , Significantly reduce mean time to recovery from failure MTTR The role of intelligent algorithm in the whole process

 

summary

Review the problems and challenges in the field of full link tracking , Practice the contents of this plan by landing , Effectively solved .

The difficulty of using observational data

• adopt AI Algorithm ability combined with expert experience , The implementation is complex IT Fast fault detection in environment 、 Root cause location 、 performance optimization ;
• Identify the global performance of key call chains in business scenarios , Auxiliary business optimization ;
• Provide traceable performance data , Quantify the business value of the operation and maintenance department

It is difficult to build a chain of observation data

• Based on the processing capacity of the operation and maintenance data center , Collect rich observation data in real time / Handle / Storage / analysis , Build a fusion observation data system ;
• Conduct whole process display and upstream and downstream impact analysis through multi-dimensional topology .

The problem of difficult access to observation data

• Multiple sources ： Front and rear ends 、 Cross cloud deployment 、 Third party tools, etc ; Multiple data types ： journal 、 indicators 、 Call chain 、 The network traffic 、 Tripartite topology, etc ;
• Multilingual ：Java 、Go etc. ;
• multi-protocol ：OpenTracing、OpenTelemetry etc. ;

Open source benefits

Cloud intelligence has become an open source data visualization platform FlyFish . By configuring the data model, it provides users with hundreds of visual graphics components , Zero coding can achieve a cool visual large screen that meets your business needs . meanwhile , Flying fish also provides flexible expansion ability , Support component development 、 Customize the configuration of functions and global events , Facing complex demand scenarios can ensure efficient development and delivery .

Click the address link below , Welcome to  FlyFish  Like to send Star. Participate in component development , There are ten thousand yuan in cash waiting for you to get .

GitHub Address ： https://github.com/CloudWise-OpenSource/FlyFish

Gitee Address ：https://gitee.com/CloudWise/fly-fish

Ten thousand yuan cash activities ： http://bbs.aiops.cloudwise.com/t/Activity

Wechat scanning identifies the QR code below , remarks 【 Flying fish 】 Join in AIOps Community flying fish developer exchange group , And FlyFish project PMC Face to face communication ～