Using elk to build a log analysis system (I) -- component introduction

One 、 Analysis of architecture ideas

General log analysis scenarios ： Directly in the log file grep、awk You can get the information you want .

In larger scenes , This method is inefficient , The problems include how to archive when the log volume is too large 、 What to do if the text search is too slow 、 How to multi-dimensional query .

Need centralized log management , Log collection summary on all servers . The common solution is to establish a centralized log collection system , Collect the logs on all nodes , management , visit .

Generally, a large system is a distributed deployment architecture , Different service modules are deployed on different servers , When problems arise , Most situations need to be based on the key information exposed by the problem , Go to specific servers and service modules , Building a centralized log system , It can improve the efficiency of location problem .

A complete centralized log system , It needs to include the following main features ：

• collect － Be able to collect log data from multiple sources
• transmission － It can stably transmit the log data to the central system
• Storage － How to store log data
• analysis － Can support UI analysis
• Warning － Able to provide error reports , Monitoring mechanism

ELK Provides a complete set of solutions , And it's all open source software , To use with each other , Perfect connection , Efficient to meet the application of many occasions . At present, a mainstream log system .

Two 、ELK brief introduction

ELK Three abbreviations of open source software , respectively ：Elasticsearch , Logstash, Kibana , They are all open source software . I added a new one FileBeat, It is a lightweight log collection and processing tool ,F