Efficient practice intelligent analysis of massive log data in aviation industry

For the massive real-time business data monitoring and analysis challenges faced by the current aviation industry , Cloud intelligence has launched a service system based on log data , Use real-time streaming data collection 、 Big data processing and artificial intelligence technology to achieve the solution of intelligent analysis of massive log data , help IT Operation and maintenance personnel in real time 、 Dynamic discovery of business and IT Faults and anomalies in the system , Quickly locate the root cause of the problem , Ensure the business continuity of the Aviation Division 、 Stable and efficient operation .

Challenges and needs

With the rapid development of aviation and upstream and downstream service industry chain , There are many systems 、 The architecture is complex 、 The old and new business systems coexist for a long time , The massive log data generated continuously lacks effective processing means , Yes IT Operation and maintenance bring great risks and challenges .

Take civil aviation for example , Passengers who choose to travel by air need to experience ticket booking 、 Check in 、 A series of processes such as boarding , Each kind of business processing must span the application system of multiple platforms , Each system will output log data in real time to record the current application operation , Any application failure or performance problem in the above process will affect passengers' travel .

How to understand the overall operation status of the business in real time , How to base on massive 、 Real time application log analysis can quickly identify abnormal business operations and potential risks , How to dynamically determine the business exception standard according to the change of business volume , How to timely alert against abnormal situations in the current business , How to realize the integration analysis and query of log data and other operation and maintenance monitoring data , Fast 、 Accurately locate business and system failures , These are all problems that need to be solved by the log intelligent analysis platform .

Cloud intelligent log intelligent analysis solution and platform architecture

Cloud intelligent massive log intelligent analysis solution includes real-time collection of system log data 、 transmission 、 Clustered message queues 、 Preprocessing components and real-time streaming big data processing , Complete the parsing of log data 、 transformation 、 desensitization 、 After business logic processing , Stored in distributed storage system with structured data , Then call one or more algorithm combinations in the algorithm set for intelligent analysis of multiple scenes 、 Alarm and visual display , At the same time, provide data API Meet the needs of other data in-depth mining and exploration .

The architecture of the massive log intelligent analysis platform is shown in the figure below ：

Program features and advantages

The cloud intelligent log intelligent analysis platform solution is the continuity of aviation business operation 、 The improvement of user experience and efficient operation and maintenance of business provide reliable technical support , This scheme has the following advantages ：

1. Rich log data real-time collection

   1. Support self owned high-performance collectors , Compatible with mainstream open source data collectors 、 Flexible and horizontally expanded data access interface , Fast access to streaming data .
   2. Ultra high throughput and minimal response time , Data access can be queried at the second level .

2. High performance 、 Dynamically expand the data processing platform . Built in dozens of processing components , Support visual processing pipeline And single step processing result verification 、 The processing results are stored in columns 、 Linear storage expansion . The platform has real-time streaming data processing capability , Meet the strict requirements of real-time processing and intelligent analysis of massive log data in the aviation industry .

   1. Support horizontal scaling according to the amount of data , Data processing power ：40T/ God ,EPS：130 ten thousand / second .
   2. Support multiple nodes to write at the same time , Write speed ： Single node ：60 Ten thousand / second . Under the same conditions , Write speed is faster than Elasticsearch fast 10 More than times .

3. Multi scene intelligent analysis platform capability

   1. The algorithm platform is based on the latest architecture and artificial intelligence algorithms , Have strong intelligent analysis ability , Provide algorithm support and expansion for upper layer services .
   2. Built in multiple intelligent analysis scenarios , Including log intelligent search and context analysis 、 Automatic log pattern recognition 、 Intelligent anomaly detection based on Algorithm 、 Intelligent alarm and indicator alarm docking 、 Log and application performance monitoring 、 Fusion and correlation analysis of basic monitoring data .

Program value

Cloud intelligent log analysis solution based on massive log data has been used in aviation 、 Finance 、 Successfully implemented in intelligent operation and maintenance projects in energy and other industries , Practice has proved that this platform can bring obvious improvement and promotion to industry customers ：

• Realize the unified collection of discrete logs 、 Handle 、 Storage 、 Archive and query , Greatly improve the convenience of log management and analysis .

• Intelligent analysis based on real-time log data helps business personnel control the business operation status in time , Quickly find out the abnormality of business operation and alarm in time , Reduce business interruption time .

• Based on business log data 、 Application performance monitoring data 、 Fusion analysis of basic resource monitoring data , Realize the rapid positioning of the causes of complex online business and system problems , Improve the efficiency of dealing with complex problems .

Customer case practice

The main business of an aviation enterprise is for airlines and upstream and downstream service providers , Provide business processing 、 Electronic distribution 、 Settlement and clearing services , When performance is slow or interrupted , The existing monitoring system cannot effectively locate faults and anomalies .

In response to this user demand , Cloud intelligence has built a business operation and maintenance intelligent analysis platform based on massive real-time business log data for the enterprise , Through the real-time collection of business data 、 Handle 、 Multidimensional intelligent analysis , Discover the abnormal situation of the system business level in time , Enhance the business warning of the airline customer 、 Fault diagnosis and intelligent analysis ability .

This platform has found many times in advance that the response time of business processing has become longer 、 The interaction error rate of business instructions increases and other fault precursors . Based on intelligent analysis algorithm, make prediction and timely alarm for upcoming anomalies , So as to effectively avoid the business risk of failure .

Summary and Outlook

The cloud intelligent log intelligent analysis platform is based on AIOps Strategy , Integrate basic resource monitoring 、 Application performance monitoring 、 User experience monitoring and other multi-dimensional operation and maintenance data and artificial intelligence analysis algorithm , Be able to discover potential risks in business operation in time , Assist managers to make accurate judgments and decisions , Achieve continuous insight into the health of business operations and future development trends , Enhance the core competitiveness of enterprises .

Open source benefits

Cloud intelligence has become an open source data visualization platform FlyFish . By configuring the data model, it provides users with hundreds of visual graphics components , Zero coding can achieve a cool visual large screen that meets your business needs . meanwhile , Flying fish also provides flexible expansion ability , Support component development 、 Customize the configuration of functions and global events , Facing complex demand scenarios can ensure efficient development and delivery .

Click the address link below , Welcome to FlyFish Like to send Star. Participate in component development , There are ten thousand yuan in cash waiting for you to get .

GitHub Address ： https://github.com/CloudWise-OpenSource/FlyFish

Gitee Address ：https://gitee.com/CloudWise/fly-fish

Super experience officer activities ：http://bbs.aiops.cloudwise.com/d/712-flyfish

Ten thousand yuan cash activities ：http://bbs.aiops.cloudwise.com/t/Activity

Wechat scanning identifies the QR code below , remarks 【 Flying fish 】 Join in AIOps Community flying fish developer exchange group , And FlyFish project PMC Face to face communication ～