Logstash、Fluentd、Fluent Bit、Vector？ How to choose the appropriate open source log collector

In this article, we will discuss the most popular log collector , Include Logstash、Fluentd、Fluent Bit and Vector.

Whether you are already using the open source log collector , Or are you ready to choose one or more for your environment , Understanding the key requirements of the log collector is critical to your daily operations . These requirements include high data throughput 、 reliability 、 Extensibility 、 flexibility 、 Security and resource consumption （CPU、 Memory ）. In this blog , We will discuss the most popular log collectors , Include Logstash、Fluentd、Fluent Bit and Vector.

Key requirements to consider when evaluating log collectors

Before delving into a specific open source log collector implementation , Here are important requirements to consider when evaluating log collectors .

High data throughput

To successfully debug the problem , The engineering team needs a large number of logs per second and low latency log processing . To avoid business interruption or failure , Engineers need to quickly obtain key log data , This is the first choice for log collectors with high data throughput .

reliability

The log collector should ensure a high degree of integrity of the processed data . Even when the data throughput increases , Data integrity should also be maintained . The frequency and amount of data lost by the log collector should be limited in any case , And it's best to avoid .

Extensibility

There are several strategies that enable the log collector to process large amounts of data . Filter logs that do not belong to high priority 、 Parsing or compressing complex logs is only a small part of it . Besides , It is important to consider that such data processing may lead to CPU And memory resource utilization increases , This is the price paid for expanding capacity . Besides , As the data rate increases , There will be higher resource consumption and possible back pressure , This will also affect the scalability of the log collector .

Handle multiple data formats

Logs from various elements of cloud applications and infrastructure have many different formats . Containers and microservices are developed in different programming languages or frameworks , There are different log format methods . Avoiding the need to use multiple log collectors to handle multiple formats reduces the overall complexity .

Support various data sources and destinations

Log collectors get data from various cloud environments . Log data sources may also include message queues and streaming media platforms , for example Kafka、Redis and RabbitMQ. The log collector sends data to different destinations , For example, log management tools and storage archives . The ability of the log collector to handle various sources and destinations increases their flexibility and availability .

Security

When evaluating any log collector , The ability to handle sensitive information should be considered , For example, anonymous or exclude confidential fields and send logs to the storage backend in a secure way .

Now? , Let's discuss specific open source collectors and their main features .

Logstash

Logstash Is one of the most popular log collectors , It is ELK（Elasticsearch、Logstash、Kibana） Part of the stack .

Logstash Major advantages ：

● Dealing with structured and unstructured data .

● Support enhanced data security through anonymity or the ability to exclude sensitive fields .

● Support hundreds of plug-ins , Including the input 、 Filter and output plug-ins . The filter plug-in performs log processing , For example, aggregation and parsing .

Even though Logstash Is a reliable log collector , There are many options for processing log data , But if the memory consumption is very small, it is the key requirement , Then the other log collectors described in this article may be better . because Logstash Yes, it is Java Compiling , So we need to JVM Support . If you are going from embedded devices and IoT Collect logs in the application , This is not the best choice .

Fluentd

Fluentd It's a log collector , Memory usage is very small , It can handle various log sources and targets . Many supported plug-ins allow you to connect to multiple types of sources and targets . Like other log collectors ,Fluentd Typical sources of include applications 、 Infrastructure and message queuing platform , The usual destinations are log management tools and storage archives .

Fluentd Major advantages ：

• Support many log sources and targets
• Flexible and extensible parsing options , Supports multiple input formats
• Have a huge ecosystem , Including hundreds of plug-ins and Ruby Ability to write your own plug-ins
• Support Apache license , edition 2.0
• Supplier neutrality （CNCF project ）

If you are looking for supplier neutrality ,Fluentd It's a good choice . It's in Kubernetes And container environments .

Fluent Bit

Fluent Bit stay Kubernetes It works well in container environments such as clusters . Besides ,Fluent Bit It can be expanded and still save resources , Because it takes up very little space .Fluent Bit Not only in Kubernetes Often used in the environment , And it can be deployed on bare metal servers 、 Virtual machines and embedded devices .

Fluent Bit Major advantages ：

• Lightweight design with minimal memory footprint （ Usually less than 1MB）
• Easy to extend architecture
• With many inputs 、 Pluggable architecture of filters and output plug-ins
• Support metrics based and log based payloads
• It supports sending logs to the storage backend through a secure connection
• Support use SQL Stream processing
• Support Apache license , edition 2.0
• Supplier neutrality （CNCF project ）

Fluent Bit Collect logs and metrics from various sources , And send them to different destinations , Just like other log collectors .Fluent Bit What really shines is the embedded 、 Edge and other resource constrained environments , In these environments , Lean runtime with extensive input / Pairing output options is critical .Fluent Bit Not just a log collector , It can also be used as a stream processor and forward log data to Fluentd Transmitter for .

Vector

Vector It is designed as a high-performance log collector . Compared with other log collectors discussed in this article , It is a relatively new product .

Vector Key advantages ：

• Efficient memory /CPU Consumption and high data throughput
• Good reliability with correctness and delivery guarantee
• Includes customizations for dynamically transforming data in a secure and high-performance way DSL
• Support metrics based and log based payloads
• A lot of input and output integration
• It can be deployed as a proxy or aggregator

Due to its extensive deployment options 、 Support for metrics and logs and available digital integration ,Vector Is an excellent 、 Flexible choice . Besides , because Vector Yes, it is Rust Compiling , It provides memory security and efficiency assurance , Make it unique among other existing products . Whether you are working in a new environment or an old environment , It's worth taking a closer look .

Vector A unit testing framework is introduced , Complex log collector topologies can be maintained more easily . Besides ,Vector The software component of attempts to provide delivery assurance for logs and events delivered to the destination .Vector Users can also get Vector General code stability guarantee of components .

Generalization

It's hard to find a perfect log collector . Choosing the right log collector depends on your specific needs and requirements . for example , If you are working on a project that requires a small amount of resources IoT The application looks for a log collector , So it's best to use Vector or Fluent Bit instead of Logstash. If you are looking for supplier neutrality , that CNCF Supported projects （ for example Fluentd and Fluent Bit） It's a good choice .

Check the performance carefully 、 resource consumption 、 Flexibility to support various input and output formats 、 Extensibility 、 reliability 、 Supplier locking and safety requirements , It can help you find a suitable log collector .

Translated from

original text ：https://era.co/blog/choose-open-source-log-collector

Focus on

This article was first published on WeChat public 【 Attack the cloud 】, Scan the left code to pay attention to , Learn more about consulting , There are more free resources for you to learn